In some cases, especially found unfavorable when inspecting toroidal cavities, the borescope may be subject to gravity. This, in combination with limited steering and controlling abilities through the inner spaces, can make it challenging satisfying the high standards or inspecting these parts at all. Hence, a generic and more robust approach is needed to help overcome these limitations.

The novel solution proposed in this work aims for a defined and yet contactless probe guidance on a circular path, being adjustable towards certain sensor characteristics. The path is described by the distal end of a material compound tube, which is formed in a certain manner. The overall system is related to continuum robotics and is particularly suitable for inspecting toroidal cavities with hard to reach areas. In contrast to known high-precision solutions [1] this highly automatable and minimally invasive system requires no disassembly of parts.

Next, related work is reviewed followed by a description of the proposed solution and an experimental investigation of its capabilities. Certain characteristics of the system are then identified and possible limitations discussed. This paper is concluded with required future work.

#### 2 Related Work

In this section an overview of recent developments in the field of endoscopy is given followed by selected continuum robots that can be used for inspection purposes and tube forming as the means on which this work is based on.

#### 2.1 Endoscopy

An endoscope, as shown in Figure 1a, mainly consists of a flexible hose or a rigid tube. Devices being used in the technical sector are typically referred to as borescopes. The hose can hold different kinds of conductors, optical fibers or tethers and is very resistive against environmental impacts as well as mechanical stresses. Endoscopes are available in various diameters and lengths. Special optics for imaging as well as for lighting are located at the distal end of the hose. Conventionally, optical fibers are used for lighting and in the form of bundles for imaging also. A respective light source is then located inside of a control device which also offers a representation of the image on a screen. Typically, the probe directly contains multiple sensor types for imaging and other purposes. Hence, no optical fibers are needed anymore and several electrical conductors can be used. The distal end can mostly be actuated in order to manipulate the direction the head is facing. This can be controlled using tethers via the control device.

Recent devices offer intuitive user interfaces simplifying the navigation through the part of interest as well as offering numerous aids for measuring and documenting features of interest. High resolution imaging sensors, which can also be arranged in pairs to allow for stereo vision, provide the detail of information needed. Recent publications describe advanced localization and real time tracking techniques based on these. Examples include the use of optical models for improvements in accuracy of endoscopic measurements [8], photogrammetry for obtaining 3D maps on a larger scale [3], [4] or approaches for localizing the probe relatively against the inner spaces using triangulation and other optical solutions [11] or even using shape sensing of the hose itself [14]. Additionally, some authors have claimed applications in microscopy using endoscopes [9].

#### 2.2 Continuum Robotics

In contrast to conventional industrial robots respective systems in continuum robotics usually consist of joints in high quantities that are chained together (Figure 1b). Beyond the capability of industrial robots of moving the flange within any quantity of poses these systems additionally allow for keeping the robots structure itself close to a defined trajectory in space. This enables for reaching out to inaccessible areas while still being able of avoiding obstacles. Such robots exist in various magnitudes of scale while especially those from the fields of endoscopic robotics as well as surgical applications are of greatest relevance to the objective addressed in this work.

Endoscopic robots are designed to fit through small hollow spaces while being required to describe winding curves. This is typically achieved by involving numerous individual joints. These may be actuated using conventional tethers, compressed or decompressed fluids, electromagnetically by coils or local motors in combination with gear boxes in the joints or even using piezoelectric elements for instance [12]. Possible configurations may be divided into those whose shape evolves with them being fed into the inner spaces of interest and after that cannot be changed anymore and more complex systems that are capable of freely transforming their shape at any time. The latter are known as snake robots [5].

The field of soft robotics comprises mechanisms that don't rely on rigid joints but on flexible ones rather. These often attempt to imitate natural organisms and take advantage of various physical phenomena. These may be pressure, thermoelastic behavior, friction, electromagnetism or the use of tethers for instance. Advantages are adaptability to and conservation of their respective environment, enhanced safety of operation in the presence of humans or smaller magnitudes of scale of the mechanisms being possible.

Robots being combinations of both fields exists such as concentric tube robots

for instance [6]. These actually do not consist of distinct joints but allow for following certain curves due to pre-bent concentric tube segments. Their constitution is not rigid but flexible. These robots do not suffice the requirements within this application, therefore tube forming is introduced in the next subsection as a means for achieving the task.

#### 2.3 Tube Forming

Various tube bending forming techniques have been developed along with the arising respective demands for strong and lightweight bended tubes in different industries such as aviation, aerospace, automotive and more. Hence, certain manufacturing parameters have improved and further developed comprising possible bending radii, bending angles and shapes, material characteristics and manufacturing tolerances. Tube forming can be conducted either cold or warm and under different loading conditions like multipoint bending, compression bending, stretch bending, roll bending, rotary draw bending or laser bending. Different tube materials can be processed and different basic cross-sectional shapes of tubes be used such as circular, rectangular or even arbitrary profiles. Fabrication can occur seamlessly or welded for instance. Depending on the chosen process parameters and the wrought material highly diverse magnitudes of forces can be involved in the fabrication process. Tube forming introduces some challenges though like possible defects in form of wrinkling, over thinning, cross section distortion or spring back due to complex and uneven tension and stress distributions as well as special requirements to the tools. A robust prediction and control of the phenomena is a common topic of research by using experimental, physical, analytical or numerical approaches [13].

Most commonly used for obtaining complex multi dimensional geometries is roll push bending (Figure 1c), where continuously applied mechanisms like feed and torsion are used in combination with certain displacements of the rolls and furthermore investigations on process models exist [2], [7], [10]. This technology is also available fully computer controlled.

Each forming process constitutes of plastic and elastic deformation behavior of the material used. The latter occurs in the form of spring back of the material.

(a) Schematic endoscope (b) Continuum mechanism (c) Roll push bending

Fig. 1. Related work

#### 3 System Description

In the following section the novel inspection system as well as the path planning strategy are described and their potential investigated using an experimental setup. The system's characteristics are evaluated in more detail eventually.

#### 3.1 System Definition

This work aims at the generation of circular paths for the purpose of probe guidance. The underlying core mechanism is an adapted and extended kind of tube forming technology. By forming a special material compound tube the evolving trajectory described by its distal end is used to approximate the desired path and thus, to carry a probe. Further key components of the system include a guide tube (Figure 2a) and a possible actuation unit.

The forming process may be conducted manually under certain limitations regarding the achievable accuracy and repeatability of the path. Incorporating an optional actuation unit helps enhancing the results. But still, generating the desired path is not trivial, unwanted deviations can occur easily and in addition multiple curvatures are required to enter the cavity of interest successfully. Similar to conventional forming strategies, where both plastic and elastic deformational behavior occurs, this approach makes use of both of these effects. Here, especially the elastic component is evoked intentionally. Along with basic material feed an additional and distinct portion of torsion may be used to influence the evolving circular path's direction and tilt. Furthermore, distinct elastic pre-deformations of different magnitudes and orientations can be induced using forming mechanisms like bending the material forth and back.

(a) System schematic (b) Roll arrangement (c) Material compound

Fig. 2. Proposed solution

The entire forming strategy is organized into two stages. Abovementioned complex forming mechanisms are conducted outside of the toroidal cavity to be inspected. An additional, subsequent shaping of the material takes place using a guide tube where also the majority of the circularity is induced. Prior to the path generation, the guide is inserted and positioned through one of the possible inspection openings. The guide tube's shape follows certain geometric rules. Clearances between the material compound tube and the guide tube's inner surfaces allow the previously applied forming steps to influence the shape, unfolding at the guide tube's outlet inside of the cavity. The guide tube furthermore allows for gaining access to hollow spaces that can only hardly be reached, through small openings or bores, for instance.

The material compound tube used here (Figure 2c) consists of a thin aluminum strip wrapped longitudinally, thus forming the shape of a tube. The aluminum strip is laminated with a protective foil on either side and is adhesively encapsulated by a jacket of polyethylene. Even though the material was originally developed for shielding telephone cables and preventing those from corrosion due to moisture 1 , its layer structure leads to certain material characteristics that are beneficial to this work. These do not only include its capability of carrying cables or tethers inside but also its low specific weight per length, its high robustness, its highly repeatable plastic deformability and its high dimensional stability when bending curves. These especially allow for the contactless and self-supporting realization of the generated path while carrying a probe at its distal end.

By adjusting the forming parameters, the guide tube's geometry and the material's dimensions paths of different magnitudes of scale may be realized and thus the method be used for inspecting a variety of parts. Beyond the use of a basic material and process model, additional localization techniques may be used to further improve accuracies and compensate deviations like sag, for instance.

#### 3.2 Path Planning

The path planning strategy feeds the two-stage forming process while taking different boundary conditions into account regarding the part, the sensor, the tube material and the forming capabilities. Figure 3 illustrates the path planning strategy. As an initial step, a geometric representation of the desired circular

Fig. 3. Path Planning Process

path is generated with respect to the toroidal cavity's 3D model facilitating an optimization and equalization of the distances between the sensor and the part's inner surfaces. Subsequently, the geometric representation of the circular path is partitioned into segments. The size of these equally sized segments is calculated based on the sensor's characteristics such as its aperture angle or the desired measurement density, for instance. The forming will therefore be conducted intermittently. Next, the different forming mechanisms such as feed, torsion and

1 Jachimowicz, L.: Corrosion proof shielding tape for shielding telephone cables. US Patent 3,233,036 (1963).

bending needed for the first segment are quantified in order to achieve a certain shape unfolding at the guide tube's outlet with the distal end approximating the desired path as close as possible. The forming can immediately be carried out either manually or using a programmable actuation unit. The forming may be followed by several measurements using the probe at the approached location. After the completion of the measurements the path planning process returns to the computation of the next individual segment's forming parameters. By incorporating an optional localization technique as an additional part of the measurement sequence at each individual location a pose estimation may be implemented allowing for the closed loop control of deviation residues that may not be compensated using the open loop path generation. During the entire process the distal end's trajectory is crucial indeed, but not the evolving shape itself.

#### 3.3 Experimental Setup

An experimental setup is used to evaluate the proposed system. At first, the material compound tube is fed through an adjustable roll arrangement (Figure 2b) which is mounted on a rigid frame (Figure 4a). Like this, parameters can be tuned in order to reach the targeted shape and thus distal end trajectory. The roll arrangement is further used to derive and validate a suitable geometry of the guide tube and therefore miniaturization of the system. Possible future guide tubes are also tested in form of additively manufactured specimen of plastic or metal. An industrial robot serves as a programmable actuation unit and is able to handle the tube material using a gripper and allows to continuously monitor process forces using a load cell. The robot conducts the intermittent forming using the mechanisms described above.

#### (a) Setup with rolls and industrial robot (b) Lasertracker plot

Fig. 4. Experimental setup and circular trajectory plot

During the experiments it has been found that several influencing factors for the resulting curvatures exist. These do not only incorporate the plastic deformation due to the forming process but also effects in form of elastic spring back after exiting the guide tube, gravitation having an effect on the tube's own mass, conductors running inside or a connected probe, intended pre-deformations or inhomogeneity of the tube's layer structure.

Experiments have been conducted for gaining knowledge about realizable diameters of the circular path, control possibilities regarding diameter and trajectory deviations, load capabilities regarding probes or conductors, repeatability and reversibility of deformations, forces required for the forming work and the system's oscillation behavior.

#### 3.4 Results and System Characteristics

Using the tested material compound tube's diameter of 8 mm circular paths of up to 1 m in diameter could be formed, while an idealized circular shape could still be approximated. Accuracies can be reached down to a few centimeters. This was evaluated in detail using a laser tracker device. The plot (Figure 4b) shows the obtained path, which differs from the idealized circle with proceeding length, due to bending, caused by the tube's mass. In addition to its own weight the system is able to carry small loads of up to 50 g at its distal end as well as conductors inside of the tube. The forming mechanisms mentioned in the previous subsections can be used to control the circular path within some certain boundaries and to compensate for deviations, if identified. The tests revealed that the chosen material compound tube can be formed several dozens of times after which it tends to break. Even in case of fatigue no residues of the tube will remain since the polyethylene jacket still holds the compound together well. The system is prone to oscillations easily which originate from the forming process itself but also from external vibrations and from loads applied to the tube. Forming process loads were up to 80 N when pushing the material, but strongly depend on the roll arrangement, also. As it can be seen in Figure 4b, fast load application leads to significant oscillations (from 180 ° on), while smooth load application allows to avoid these. The optimized guide tubes derived from the roll arrangements allow for entering parts through small openings or bores far smaller than 20 mm.

#### 4 Discussion

Since the material compound only features a limited lifespan it needs to be a disposable component of the overall inspection system. The circular diameters that can be obtained from the forming process using the guide tube are only representative for the tested material diameter. Other magnitudes of scale are possible using different material sizes. Since the overall concept is self-supporting and thus aims at contactless path generation with respect to the part and furthermore no residues will remain inside in case of fatigue the system may also be used for inspecting sensitive areas. Making use of the forming mechanisms mentioned above with the intention of controlling the path would imply having information about the current pose available. While stationary operation is already proven, mobile operation at different sites may still be developed. Hereto, reducing the process forces would be crucial in order to be able to miniaturize possible mobile forming units. Using drive units the forming can be conducted synchronously with possible ongoing measurements, also. Loads like probes may be carried by the system but care must be taken of the oscillations the system is prone to. This may also be done with help of the forming strategy. On the other hand side this fact formulates certain requirements with respect to the sensors as well. The chances to miniaturize the roll setup in form of the guide tube enables for insertion of the system through tiny openings or bores. Special about the suggested solution is the fact that the forming is conducted bipartite. Like this, hard to reach inner spaces can be accessed and a circular path be unfolded.

## 5 Conclusion

#### 5.1 Summary

In this work a performant new approach for the contactless inspection of toroidal cavities using circular path generation is presented, especially characterized by its high automation capability. It fosters the separation of inspection and diagnosis tasks while eliminating several drawbacks of conventional inspections. The approach enables simplified access to inner spaces, that are hard to inspect. Intermittently generating distinct circular paths allows for systematic inspection while meeting the boundary conditions of possible probes as well as parts of interest, leading to more valuable and reliable results. The resulting probe guidance system is self-supporting and therefore clearances between sensors and surfaces to be inspected can be well adjusted for. The solution promises to be configurable for various parts of interest. A first evaluation of the novel solution has been conducted experimentally while its capabilities and limitations were pointed out. Using this systematic approach completeness and reproducibility of measurements can be significantly improved.

#### 5.2 Future Work

Certain actions need to be taken in order to further develop the proposed solution towards an applicable condition. These include:

Furthermore, an optimization of the forming process parameters needs to be conducted in order to reduce oscillations for instance. Using the advanced knowledge obtained by respective models of the system a design strategy for adapting the system for various inspection subjects is to be derived.

#### Acknowledgements

The work was performed in collaboration with Lufthansa Technik AG and funded by German Federal Ministry of Economic Affairs and Energy under grant number 20X1709.

#### References

**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Multi-agent path planning: comparison of different behaviors in the case of collisions**

Michaela Krä1 , Ludwig Vogt1 , Valentin Spannagl1 and Johannes Schilp1, 2

1 Chair of Digital Manufacturing, Faculty of Applied Computer Science, Augsburg University, Eichleitnerstr. 30, 86159 Augsburg, Germany

2 Fraunhofer Research Institution for Casting, Composite and Processing Technology – IGCV, Provinostr. 52, 86153 Augsburg, Germany

#### **Abstract.**

In the context of Industry 4.0 and cyber-physical production systems, the role of production logistics is perceived as more and more important in order to reach the overall manufacturing targets. One central aspect in organizing the flow of material consists in task allocation and path planning for transport resources disposing of growing autonomy. There are various approaches for multi-agent path planning as well as the way of dealing with collisions. Collisions are possible due to traffic volume and can either be treated on planning level or in a short-term way on control level.

The paper presents existing strategies for path finding before giving an overview of methods to deal with autonomous transport resources that meet in a manufacturing environment. Then, different existing behaviors and reactions in the case of collision detection based on several criteria are compared. This step allows classifying the strategies depending on the manufacturing environment and its organization.

**Keywords:** production logistics, autonomous transport resources, collision avoidance.

## **1 Introduction**

With the introduction of the concepts of Industry 4.0 like digitization and decentralization, various challenges arise in production environments. In addition, customer demand regarding increasing number of product variants as well as shorter product life cycles result in smaller lot sizes. [1, 2] Production logistics play an important role in the manufacturing process and companies understand more and more its influence on the throughput time [3].

The utilization of autonomous resources is supposed to turn logistics more efficient due to their possibility to organize themselves on a decentral level. This means that the operational level obtains decision-making autonomy within defined constraints. A challenge in this context is the balance between taking advantage of the autonomy and reaching central planning targets. One central aspect is the interaction of various resources with possibly different skill sets. [4–6]

Mobile autonomous resources, i.e. self-directed vehicles also called autonomous guided vehicles (AGV), take center stage in current research. There exist different concepts for multi-agent path finding (MAPF) presenting each advantages and disadvantages. [7, 8] Considering the range of concepts for MAPF and especially collision avoidance an evaluation of these approaches comparing their applicability in a specific manufacturing environment is missing. An approach is needed in order to identify a suitable strategy or set of strategies for avoiding collisions in production environments.

The scope of this paper is to give an overview of existing approaches for dealing with and avoiding collisions of autonomous transport resources. In addition, these ways are compared based on a choice of valuable criteria.

#### **2 State of the art**

Relevant literature concerning production logistics, autonomous resources and MAPF is summed up in this section to present applicable concepts. Before taking a deeper look on path planning and collision avoidance, a short introduction to logistics and mobile resources in general is given.

#### **2.1 Logistics and mobile resources**

As described in the introduction, AGVs are used for a variety of single and multi-agent applications but the main sector is logistic systems. The most common logistic scenarios are the transportation of dock containers in a terminal [9–11] and transportation services in warehouses as well as production lines [12]. While in the past, AGVs were only used for transportation services recent projects have also focused on expanding their skill set. Hoff and Sarker [13] have distinguished between six different major types of AGVs that include unit load, towing, parallel truck, forklift, light load and assembly line vehicles. The newest addition to these classic AGV systems have been drones which are tested in warehouses [14]. The vehicle design and fleet combination is just one of many central research topics in AGV Systems. Others involve the sensors, navigation, localization, vehicle control and the communication between the agents [15].

#### **2.2 Path planning for multiple resources**

The classical MAPF problem for a set of *n* agents receives an input tuple *(G,s,t)* where *G = (V,E)* is an undirected graph with *V* vertices denoting locations and *E* edges. *s : [1,…,n] - V* maps an agent to a start vertex , and *t : [1,…,k] - V* maps an agent to a target vertex. Often the graph is given with its corresponding adjacency matrix that displays the connected vertices and the weight of the edges (c.f. figure 1). For the solution of this problem, one can find methods, which take different constraints into consideration.

The Dijkstra´s algorithm is one of the most commonly known algorithms for the shortest path search [16]. It calculates the shortest path between two vertices in an undirected grid.

**Fig. 1.** Representation of an undirected graph with its adjacency matrix.

An evolution of Dijkstra´s algorithm is the A\*Algorithm, which estimates the distance, i.e. costs, from the start vertex to the target vertex to preselect appropriate vertices with a heuristic [17]. Warren [18] proposes a modified A\*Algorithm which is faster than the normal A\*Algorithm due to a further reduced solution quantity.. Both the Dijkstra and A\*Algorithm are not flexible and cannot react to a changing environment [19]. Therefor the D\*Algorithm (Dynamic A\*) was developed by Stentz [19] as an extension of the A\*Algorithm. Koenig and Likhachev [20] propose the D\*Lite Algorithm which combines the heuristic approach of the A\*Algorithm with the incremental one of the D\*Algorithm. Other extensions of the D\*Algorithm are the Focussed D\* and Field D\*Algorithm. The Focussed D\*Algorithm focusses on the repair of the cost map, thus reducing the computational costs [21]. While all of these approaches are limited by a discrete set of possible transitions in the grid the Field D\*Algorithm is interpolation based and calculates smooth paths for the agents [22].

These algorithms were developed for single agent path planning but can also be used for MAPF problems. The M\*Algorithm is specified for MAPF problems and minimizes the global cumulative cost function of all agents [23]. An extension of the M\*Algorithm is the M\*UM\*Algorithm which considers uncertainty in the position and pose of the agent [24, 25]. Nebel et al. [26] propose an M\*Algorithm for uncertainty in the destination of the agents path while Ma et al. [27] developed a MAPF algorithm with delay probabilities.

#### **2.3 Collision detection and avoidance**

One of the main topics within the MAPF problem is the collision detection and avoidance between the agents. There are two main approaches for the collision detection: one is a centralized and the other one a decentralized. For the centralized approach the information about the actual position, path and speed must be collected in a central location [28]. Felner et al. [29] are adding a heuristic to the Conflict-Based Search (CBS) by aggregating collisions among the agents. While this approach uses discretized time steps and assumes that all actions have the same time duration Andreychuk et al. [30] propose a Continuous Time Conflict-Based Search (CCBS).

The decentralized approach enables the agents to detect collisions by themselves and react accordingly. Wei et al. [31] present an approach where only the affected agents need to communicate with each other in case of a collision. Trodden and Richards [32] use a Model Predictive Control (MPC) where only one agent replans, while all others continue with their path. The agent that replans its path is identified via a bidding system, where the highest bidder receives the replanning token in each iteration. Desaraju and How [33] use a two-step approach where the first step is the individual component and the second step is the interaction component. While in the first step each agent minimizes the cost of its own paths, the interaction component minimizes the total cost across all agents.

If a collision is detected, researchers propose a variety of strategies for the behavior of the involved resources. Liu et al. [34] plan paths for a multi-agent pickup and delivery problem where no paths cross each other during the duration of the exercised task. A similar approach is used by Kiarostami et al. [35] where a Monte-Carlo Tree Search is used to find non-overlapping paths.

Sun et al. [36] developed a decoupled approach where every agent's path is planned with an A\*Algorithm. Every agent continuously checks its front area and if another agent is detected in a critical region, a behavior-based collision avoidance strategy is executed. The behaviors are specialized for an intralogistic problem description and selected by specified traffic rules. Another decentralized approach where possible collisions are detected by the agent itself is proposed by Gochev et al [37]. Instead of a rule based collision avoidance the agents switch in a collision avoidance mode. In this mode the agents create a roundabout along the original path which is faster than stepping back and changing directions. Chang et al. [38] are using a detection shell around each agent and according to the nearest object inside this detection shell breaking and gyroscopic forces are calculated.

A five layer neural network with three hidden layers which is able to generate crossing paths and the associated velocity vectors of the agents so that no collision occurs is presented by Chen et al [39]. This approach is based on the optimal reciprocal collision avoidance (ORCA) which is often used for the collision avoidance in free navigating agents [40].

#### **3 Concept**

Based on the approaches presented in chapter 2, criteria to be considered in manufacturing systems are identified before giving a comparison of the concepts for collision avoidance. For the use of autonomous transport resources in real manufacturing environments, this is a necessary step in order to choose appropriate strategies depending on individual production constraints.

#### **3.1 Criteria to compare collision avoidance strategies**

To evaluate the different collision avoidance strategies it is necessary to define the environmental conditions. The criteria to compare the different collision avoidance strategies are not to be confused with the environmental settings. Since there exists no past work on the ranking or comparison of these influences for the different algorithms we have identified the following environmental influences which were the most common research topics in chapter 2.2:

All of these aspects have to be considered not in an isolated test area, but in real manufacturing conditions, the previously mentioned environmental settings. These comprise the next parameters based on Adam [41]:

#### **3.2 Comparison of different collision avoidance strategies**

There exist many different approaches for AGVs to behave in the case a collision is detected. We have identified five basic strategies from the research done in chapter 2.3 and the collision avoidance behavior presented by Sun et al. [36]. Though they explain eight different behaviors, all of them can be reduced to two basic strategies. That is why we adapted the avoiding and waiting strategies of Sun et al. plus the variation of the start time, no crossing paths and a change of velocities for our evaluation. All strategies are further described in the following segment. To avoid repetitions it is assumed that the paths of the agents cross at vertex *n*, the velocities of the agents are *v* and the distance from the start vertex to vertex *n* is *d*.

**Fig. 2.** Two crossing paths where the agents have different priorities. agent 1 passes vertex *n* before agent 2 because of a higher priority.

### **4 Application and evaluation settings**

The environment for the test cases is a ROS based path planner and its simulation environment. A test scenario is set up to evaluate different criteria for the collision avoidance strategies presented in chapter 3.2. The parameters mentioned in chapter 3.1 are meant to be analyzed and structured regarding their influence on the choice of suitable collision avoidance strategies. They make the simulation better comparable to real life applications but also computational expensive.

Workers are seen as obstacles and these will cross and obstruct the planned paths at a certain percentage. The duration of the obstruction is determined with a Gaussian normal distribution. In addition, we assume that the workstations have different states,

so the vertices of the grid are changing. Vertices will have the capacity one, so only one agent can occupy each vertex at a specific time step. The edges also have a capacity of one. Despite this, it is possible that multiple agents are in the same cross or pass each other in a lane because the ROS path planner builds its own grid on the prescribed map.

To evaluate and compare the different strategies we analyze a set of key performance indicators in the simulation. Since most researchers try to minimize the sum of costs of all agents this is one of the identified parameters. By analyzing additional parameters, the influences of the different collision avoidance strategies can be shown. Furthermore, we analyze the traffic which is generated from the different strategies and also the grid capacity utilization. With these parameters, we can make a first assessment of how the applied strategies influence workers or how easily additional orders can be implemented in the plan:

The first tests in a simulation environment compare the average throughput time of four AGVs with each executing orders of two or three jobs for the strategies priority and path crossing (cf. chapter 3.2). The results for the mean value of the order duration show that avoiding collisions by circumnavigating already planned paths leads, as expected, to 10 up to 30 % longer throughput times. It is evident that these strategies depend highly on the considered manufacturing layout and the existing traffic. This needs to be regarded in a more detailed way.

### **5 Conclusion**

Current challenges linked to the concepts of Industry 4.0 affect in particular production logistics. This comprises among others the need of more flexible solutions for material transport in manufacturing environments. Thus, the use of autonomous resources increases as these means of transport present advantages compared to standard logistic solutions and as they allow relocating decisions on resource level.

This contribution presents an approach to compare different ways of dealing with collisions between AGVs based on valuable criteria. The treated types of collision avoidance were path crossing, priorities, start time, velocities and internal collision avoidance mode. Relevant criteria to evaluate these strategies are order duration, time constraints and prioritization.

The next step will be a more extensive evaluation, validation and verification of the concept presented in chapter 3 within a simulation scenario. These analyses will aim at verifying the applicability of the stated criteria and complete them if necessary. Additional varying speeds of the vehicles on their way through the production environment have to be implemented to make the application more realistic.

#### **Acknowledgements.**

The presented results have been developed within the research project FORobotics (www.forobotics.de). The Bavarian Research Foundation funds this project.

## **References**

on Automation Science and Engineering 13(4): 1433–1447. doi: 10.1109/TASE.2016.2603781

**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Trajectory Optimization Methods for Robotic Cells Considering Energy Efficiency and Collisions

Elias Knochelmann ¨ 1 , Dwayne Steinke1 , Joel Greenyer2 , Svenja Spindeldreier1 , and Tobias Ortmaier1

1 Institute of Mechatronic Systems, Leibniz Universitat Hannover, Hannover, Germany ¨<EMAIL_ADDRESS>
2 Software Engineering Group, Leibniz Universitat Hannover, Hannover, Germany ¨

Abstract. In production robots are moved at maximum speed whenever possible in order to achieve the shortest overall cycle time. This can lead to individual waiting times, especially in interlinked production processes. These waiting times offer opportunities for optimization. Due to high energy prices and political efforts, energy efficiency has become the focus of trajectory optimization in recent years. Robot cells with a common intermediate circuit offer the possibility of energy exchange across individual axes or robots. By adapting the robot trajectories, the total power consumption of a robotic cell on the grid side can be significantly reduced. This paper focuses on trajectory optimization, whereby a detailed collision detection of individual robots is included within the analysis. It is shown that with collision detection energy optimization for cramped robot cells becomes possible and the losses in efficiency compared to the optimization without it are minute.

### 1 Introduction

Reducing CO2 emissions through higher energy efficiency is one of the main goals of our time. Operating costs of factories can also be reduced by lowering the energy consumption of its components. As automation advances, the number of robots in production is constantly increasing [22]. This has also been recognised by the EU. An energy roadmap [7] has been issued in 2014 with the goal of increasing energy efficiency by 27 % until 2030. The aim of this work is to reduce the energy consumption of industrial robot cells by adapting robot movements. The method is evaluated using the robot cell shown in Fig. 1.

The reduction of the energy consumption of robots can either be hardware-

Fig. 1: Investigated robot cell packing a pallet, with several potential collisions.

based or software-based [5]. Examples for hardware changes are e.g. extension of the

*Handling and Industrial Robotics*, https://doi.org/10.1007/978-3-662-61755-7\_21

T. Schüppstuhl et al. (Hrsg.), *Annals of Scientific Society for Assembly,*

DC link to several robots [27] or the installation of energy storage devices [13]. With these two methods recuperated braking energy from the motors is stored or used for other robots. This means that recuperated energy for overvoltage protection does not have to be converted into heat [27]. However, these methods have the disadvantage that additional adaptation costs are incurred per robot and that systems for installation have longer downtimes.

In the case of software solutions, costs are only incurred during development, and the transfer to several systems is negligible. Software solutions are a geometrically or time sequence change to a robot trajectory [5].  In [30] a robot trajectory was decomposed into viapoints and the position of these were optimized. In [15] the energy absorption of a trajectory was optimized based on maximizing the energy exchange in the DC link. Another possibility is a different time sequence of robot trajectories for a given task. In most cases several robots are involved. One method in this field is, instead of waiting after fast movements, the robots movement may be slowed down. In [9] different robot trajectories were investigated and it was found that slowing down trajectories almost always saves energy. However, movements can also be shifted in time so that braking movements and acceleration, from different robots/axis, directly overlap over each other. Of course this only has advantages if robots can exchange energy.

In industry this is not possible at the moment, but by implementing a DC factory, an energy exchange at factory level becomes possible [27]. In this paper it is assumed that the robots have such a connection. The method of sliding over each other can also be combined with the slowing down of fast movements. In [14] such an optimization for a multi-axis system was presented. Losses from overvoltage protection are reduced for one cyclic trajectory per axis. A method for scheduled energy optimal trajectories was presented in [29] and great energy savings were shown for a single robot. When considering geometric path changes or when changing the time sequence of several robot movements, it is important that no collisions occur. [28] took collisions into account, but only for a restricted area and [24] used simple bounding boxes. In [2] an iterative collision detection for a robot was presented, which guarantees collision freedom for the next pose. However, to the best knowledge of the authors, for larger linked movements of several adjacent robots with a detailed collisions detection, as shown in Fig. 1, no uniform method for trajectory optimization was published.

An automated methodology to exploit braking energy recuperation though scheduling and slowing down trajectories was proposed in [12]. The proposed methodology accomplishes this in two levels. In the first level, the sequential control optimization (SCO), the overall discrete control of the system is optimized to maximize the times in which the acceleration and deceleration phases of individual robots beneficially overlap. Usually, the order of robot movements can be arranged in multiple ways, and some control strategies offer more potential for braking energy recuperation than others. Then, in the second level, the detailed trajectory optimization (DTO), for each set of braking and acceleration phases that an optimized control strategy brought to overlap, the trajectories of the involved axes must be aligned in detail. For example, the braking phase of one axis can be shifted, stretched, or compressed to match the energy demand for accelerating another axis. In this paper, we focus on the DTO, a method for the overlying SCO was already published in [12].

The SCO calculates the acceleration and deceleration phase of robots that should overlap to maximize the energy efficiency gain. These time points and related robot paths are given to the DTO. Here the robot trajectories are improved in an model-based offline optimization. The two internal robot control variables (*start delay* and *compression factor*) are used as optimization parameters. This guarantees an intuitive transferability to industrial plants. The entire process is model-based and offline, therefore only the newly calculated parameters must be changed in the controller. In addition, a collision detection is presented, which can be used as a nonlinear constraint within the optimization. The collision detection method was only applied to DTO, because collision in robot cells, as the one sketched in Fig. 1, are a matter of trajectory timing. This collision detection routine is a two-layered process [4] to guarantee a fast and accurate detection. The method will be tested in the simulated robot cell shown in Fig. 1.

In Sect. 2 the energy model used to simulate the overall energy consumption of a production process is described. Sect. 3 builds on the model of Sect. 2 to optimize the overall energy consumption. The optimization approach and the applied constrains are outlined in Sect. 3. To prevent collisions, which can occur during optimization from Sect. 3, an collision detection is described in Sect. 4. Results are evaluated in Sect. 5 and the paper concludes in Sect. 6.

#### 2 Modeling of Industrial Robots

In this section the robot model used in optimization will be presented. Since the model is often invoked within the optimization, one of the main requirements, besides the accuracy, is a short computing time. The complete model was already presented in [12] but it will be described roughly for the sake of completeness. The general structure of the energy flow is shown in Fig. 2.

The power flows from the main grid Pgrid to the constant consumers Pl,grid. These are normal losses caused by fans and controllers. Besides that, the main part of Pgrid flows thought the rectifier TDC,1 into the DC intermediate circuit. The DC intermediate circuit mainly supplies the inverters TAC1−n , which in turn supply the motors M1−n, whereby n represents the numbers of inverters/motors of each robot. Considering the gear, the motors generate a joint torque τ and a joint speed ϕ˙. This drives the segments of the robot to follow a given path

Fig. 2: Detailed energy flow in the investigated robots.

q(t), q˙(t), q¨(t) with an additional

weight fext. When the robot slows

down, the motors turn into generators and recuperate energy. This energy is converted to DC and can supply other robots via the DC link. If too much power flows into the DC bus, the DC bus voltage increases. As overvoltage protection, a braking resistor Rchp converts the excess energy into heat. The complete energy model of the robot can be written as:

$$E = \int\_0^{t\_{\rm end}} P\_{\rm grid}(t) \mathrm{d}t = \mathbf{f}\_{\rm m}(\mathbf{q}(t), \dot{\mathbf{q}}(t), \ddot{\mathbf{q}}(t), \mathbf{f}\_{\rm ext}). \tag{1}$$

For a known robot, a robot path q,q˙ ,q¨ and an extra weight fext can be used to calculate the required energy. The model is nonlinear and therefore no gradient based methods can be used for optimization.

The energy consumption model is validated for multiple different KUKA industrial robots [17] and has a remaining model deviation of approx. 5 % over the whole spectrum of operating temperatures [8].

### 3 Optimization Strategy

Common industry robots have six joints, but the axes movements are all synchronized to the slowest joint. The output of a production, therefore, depends on the slowest axis movement. An unnecessarily strong acceleration of the other axes has no advantage with regard to the quantity produced. The synchronization can, therefore, lead to energy savings, but there is no recuperation from decelerating, because all the joints are slowing down at the same time. The main goal of the optimization is to make all robot movements equally fast/slow and prevent unnecessarily strong acceleration. The optimization problem thus consists of optimization variables for compression of the trajectory δov,j and a start delay τs,j for a robot j to synchronize accelerating and decelerating from different robots. The effect of these variables on a single joint robot axis can be seen in Fig. 3.

Fig. 3: Visualisation of the used optimization variables.

The start delay τs,j for a robot j shifts the trajectory n - time steps into the future. The lower limit is zero and the upper limit is the travel time for that trajectory of the slowest robot τtr,krit subtracted by the travel time τtr,j of the robot j. By optimizing multiple robots with only one trajectory each, the travel time of the slowest robot τtr,krit equals the total travel time τtot, otherwise it is the sum of the the slowest trajectory τtrl,krit of the respective l multi-robot movement. The compression variable δov,j reduces the maximal velocity q˙j and acceleration q¨j of the trajectory:

$$
\dot{q}\_{\text{new},j} = \delta\_{\text{ov},j} \cdot \dot{q}\_{\text{org},j} \quad \wedge \quad \ddot{q}\_{\text{new},j} = \delta\_{\text{ov},j} \cdot \ddot{q}\_{\text{org},j}.\tag{2}
$$

The travel time of the robot j changes to τtr,j δov,j . The lower limit is 0 and the upper limit is 1. The optimization problem finally can be described by:

$$\min\_{\boldsymbol{\pi}\_{\text{v}}, \boldsymbol{\delta}\_{\text{ov}} \in \mathbb{R}} E\_{\text{grid}}(\boldsymbol{\pi}\_{\text{s}}, \boldsymbol{\delta}\_{\text{ov}}) \text{ s.t. } 0 \le \boldsymbol{\delta}\_{\text{ov}, j} \le 1 \tag{3}$$

$$0 \le \tau\_{\mathbf{s},j} \le \tau\_{\mathbf{tr},\mathbf{k}\text{rit}} - \tau\_{\mathbf{tr},j} \tag{4}$$

$$t(\tau\_{\sf s}, \delta\_{\rm ov}) \le \tau\_{\rm tot} \tag{5}$$

$$c(\tau\_\ast, \delta\_\text{ov}) < 1,\tag{6}$$

where two new nonlinear constraints t(·) and c(·) are introduced. The first constraint makes sure that the total travel is at least not slower than before the optimization. c(·) is a nonlinear function that returns one if a collision is detected. The collision detection will be described in the following section and is skipped here. With this the optimization problem for multiple robots and multiple trajectories is fully defined.

The non-linear optimization problem and especially the nonlinear constraints result in gaps in the solution area. For that reason a particle swarm optimizer (PSO [26]) was chosen. The PSO does not guarantee that the global minimum will be found, but returns an acceptable solution in reasonable time. A more detailed view on constraints handling and the gaps in the solution area will follow in the next section.

For runtime reasons it makes sense to split the optimization into several sub-problems and solve them individually. Sub-problems are single trajectories of each of the participating robots, so τtr,krit equals τtot. The described optimization is only the secondary level, so the SCO matches the trajectories with the most energy saving potential. Since it is already known from the SCO which trajectories of the different robots have to be superimposed, there is no need to optimize ntr trajectories of m robots. It is sufficient if only one trajectory per robot, selected by SCO, is optimized in one sub problem.

The drawback of this method is that the success of the underlying optimization depends on the SCO. By dividing the optimization problem, a satisfactory solution in a satisfactory time is obtained, but if the SCO does not find the best matches, the solution is not the best possible. However, without dividing an exemplary test case with 3 robots and 10 trajectories each can no longer be solved on a PC in a satisfactory time.

The fact that only one trajectory per robot is considered simplifies the constraints of optimization. The lower limit for the compression factor δov,j of robot j is now limited by the travel time of the slowest robot in the considered subproblem. Before there were multiple trajectories of the same robot in a row, so there was no defined end of a single trajectory, because trajectories could overlap. The start and end time of the critical robots are known by the SCO and therefore the trajectories can be separated. The upper limit for the start delay now also depends on the compression factor. In contrast to the standard problem, again without overlap, the formulation of the constraints is easier. This simplifies the constraints to:

$$\frac{\tau\_{\text{tr},j}}{\tau\_{\text{tr},\text{krit}}} \le \delta\_{\text{ov},j} \le 1,\tag{7}$$

$$0 \le \tau\_{\rm s,j} \le \tau\_{\rm tr,krit} - (\tau\_{\rm tr,i} \cdot \delta\_{\rm ov,j}),\tag{8}$$

$$c(\tau\_s, \delta\_{\rm av}) < 1. \tag{9}$$

So there is no need for a nonlinear time constraint, because only through linear constraints it can be ensured that the total travel time τtot remains the same. The collision detection c(·) stays the same and will be described in the next section.

#### 4 Collision Detection

Using the model from Sec. 2 the optimization strategy from Sec. 3 works well if the robots do not share a common workspace. However, if two or more robots share a workspace there is the possibility that optimized trajectories may lead to collisions even if non-optimized trajectories do not collide. For example, there are two robots (A and B) standing across each other. Robot A is moving vertically and robot B is moving horizontally, so there is a possibility that these two are colliding in the middle. Assuming that robot B is the critical/ slowest robot, then there are two optimization parameters: The compression factor (override) and trajectory start delay of robot A. The collision detection leads now to a divided solution space as shown in Fig. 4. This (red) barrier depends on the compression factor and the start delay and marks a collision. Because of this every generated trajectory of the optimization needs to be checked for collision to only consider valid solutions.

Fig. 4: Solution space divided by collisions of a two robot cell optimization problem

The collision model uses CAD data of the robots as well as other objects in the cell and imports it as a polygon mesh model. For every sample time step the positions of the objects are recalculated from the robot joint angles of the current trajectory. If no collision is found for all sample time steps, the current parameter vector is valid and can be used in optimization.

The repeating collision detection routine is, as shown in [4, 18, 6], divided into a broad phase that approximates the objects with bounding boxes and a narrow phase which determines exactly if two objects collide. The broad phase calculates candidates that may collide and need further investigation in the narrow phase. In the past lots of algorithms and solutions for use in broad phase [20, 6, 11, 19, 16] and narrow phase [21, 23, 10, 3, 1, 25] with advantages and disadvantages depending on the number of objects, complexity of the meshes or compactness of the objects have been developed.

Therefore for the collision detection routine within the trajectory optimization three different methods for the broad phase (axis-aligned bounding box pair test (AABB) [6], the Sweep and Prune algorithm (S&P) [6] and an oriented bounding box pair test (OBB)[11]) and two for the narrow phase (V-Clip [23] and the Gilbert–Johnson–Keerthi separating-axis algorithm (GJK) [1]) were implemented and evaluated. The selection was made on the basis of speed, implementation effort and availability of literature.

To compare the selected algorithms three runtime tests were performed. This and all other test were conducted in MATLAB on an i5-6400 desktop computer. In order to guarantee comparability with the robot cell application, three tests with one trajectory each time were performed for the set-up of the test cell (Fig. 1) . Table 1 shows the average calculation times in ms for the collision check for each case. Each test was repeated 200 times and the average value was obtained. Due to the small number of tests, the deviation is very high (up to 10%), but a generalized statements can still be derived: Sweep-and-Prune shows its advantages in this application-related test, because the system is made up of many objects that move only partially, which is the strong point of this algorithm. Further detailed tests have shown that GJK is faster than V-Clip for the narrow phase, therefore GJK and S&P were chosen as the best solutions.

To speed up the optimization it is also possible to only examine those objects that can collide. With the proposed optimization strategy the robots change their trajectory but not their path. Two objects can only collide if the spatial paths, for a given optimization parameter set, cross. The collision detection approach is therefore expanded by an extra precomputation phase. There, in advance the objects that have the potential to collide will be determined. The precomputation phase is not exact but an approximation, because the reconstruction of a polygonal mesh from multiple meshes is too complex for a preexamination. This generates from any parameter set a point cloud for each object consisting of all vertices from all collision sample time steps. These point clouds represent the paths with all time dependencies eliminated. If two points from different

Fig. 5: Power demand and relative robot speeds for the same trajectory

point clouds are within a detection radius r they are marked as possibly colliding and relevant for the collision detection routine. All other objects in this robot cell e. g. robot bases, fences, conveyor or similar are irrelevant, as they don't change their position and can never collide with the robots.

To integrate the collision detection into the PSO three types for restraining the nonlinear constraints are available: absorbing walls, reflecting walls and invisible walls [26]. Fig. 4 shows that a divided solution space can result from the layout of the cell. Since the particles of the chosen PSO algorithm must be able to diverge through these boundaries, only the invisible walls [26] are suitable for constraints handling.

#### 5 Results

Fig. 5 shows the simulation results of an example system consisting of three robots as displayed in Fig. 1. In Fig 5 the overall grid power demand and the DC bus power flow between each individual robot and the DC bus in comparison to the relative robot speed Vref is shown. The relative robot speed is defined as the absolute sum of all robot joint velocities standardised to the maximum value of the slowest axis of the slowest robot. The travel time of robot 2 is the longest. Therefore, robot 2 is the critical robot of this example system. Its trajectory parameters are not optimized and the movement does not change between the portrayed optimization types. Fig. 5(a) shows the non-optimized

Fig. 6: Comparison of the energy demand with different optimization strategies

case. Each robot moves from its start point to its endpoint as fast as possible. This results in wait times for robots 1 and 3 and high grid power demands while acceleration.

Fig. 5(b) shows the case of optimization without considering collisions. The optimization leads to slower movements of robots 1 and 3 in such a way that all robot trajectories are synchronized. However these trajectories lead between 1.68 s and 1.78 s to a collision between robot 1 and 2 and are therefore not feasible. The time of the collision is marked in red. Using the nonlinear collision detection constraint as shown in Fig. 5(c) this problem is solved because robot 1 moves faster and avoids the collision.

Fig. 6(a) shows a comparison between the energy demands of the different optimization strategies. Without any kind of optimization the example system consumes 4563 J. An optimization without collision detection leads to an energy demand of 3983 J which is a saving of approximate 12.7 %. Changes to the energy demand of the example system through collision detection are minimal and lead to 3984 J. The optimisation results in Fig. 5(b) and (c) indicate that it is not as efficient to match acceleration and deceleration as it is to just compress the trajectories, because there the trajectories were simply stretched. This has two possible causes: First, the DC link capacity consists of three times the amount of a single robot. This means that more energy can be stored for a short time. Since this capacity is not used as an energy storage but mainly as a smoothing capacity, it will probably be smaller for industrial applications, because smaller means less investment costs. The second reason is, that by slowing down the trajectory the acceleration and deceleration was also reduced and thus also less braking energy is recuperated. These observations can be confirmed for a larger system consisting of 3 robots with 39 trajectories each, as shown in Fig. 6(b). By the sub problem optimization the overall energy saving is approximately 9 % compared to trajectories in which each robots moves as fast as possible and needs to wait for the critical robot. The overall results can be seen in Table 2.

Table 2: Simulation results for different types of trajectory optimisation for the test cell in Fig. 1 consisting of 3 robots with 39 trajectories each.

## 6 Conclusion

In this paper the second part of a novel engineering methodology for the development of energy-efficient production system controllers is presented. The functional principle and results of the first optimization level (SCO) have already been published in [12]. This publication presents the underlying optimization strategy (DTO) together with a collisions detection. The complete optimization problem was set up, with two optimization parameters i. e. start delay and compression factor. Based on the already presented SCO the optimization problem could be divided into multiple independent sub problems. Simplification for the optimization constraints were also be derived from this division, so that no non-linear constraints are required except for collision detection. Only by these simplifications it is possible to solve systems with more than 10 consecutive trajectories.

The disadvantage of this division is, however, that the success of this procedure depends heavily on the SCO. If the SCO does not find a good solution, the result of the DTO is also not optimal. For small cells with few trajectories (< 10) it may make sense to solve the general optimization problem.

The implemented two-step collision detection was presented, as well as a short runtime comparison of the different methods. In addition, a method was presented which will remove robots from the collision detection if they can never collide with other robots. All methods presented were tested on a virtual test cell ( Fig. 1). The DTO reduced the energy consumption when loading the pallet by 9 %. Especially due to the large DC-link capacity, the optimization did not match acceleration and deceleration phases, but slowed down the trajectories as much as possible. The collision detection has only marginally reduced the energy saving. With this method it is possible to optimize DC-powered robots even though the space in the cell is limited.

## Acknowledgment

The presented research is funded by German Research Foundation (Deutsche Forschungsgemeinschaft/DFG).

### References

**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Localization and grasping of small load carriers with autonomous unmanned aerial vehicles

Markus Lieret1 , Michael Lallinger1 , Markus Tauscher1 , and J¨org Franke1

Institute for Factory Automation and Production Systems (FAPS), Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg (FAU), Germany<EMAIL_ADDRESS>{michael.lallinger<EMAIL_ADDRESS>
Abstract. The application of unmanned aerial vehicles (UAV) in the area of inspection, survey or urban logistics has become a rapidly developing research domain. While the feasibility of material transports with UAVs has already been shown in the scope of different projects, the payload is thereby usually transferred manually into the UAV's load handling device. A decisive factor for the economic usability of UAVs for aerial transportation, however, is a fully automated system including the autonomous recognition and pick-up of the cargo. We therefore present a solution for the automated detection, localization and grasping of small load carriers with UAVs. The system includes a specialized load handing device, a camera-based real-time tracking solution for small load carriers and a fusion of the global and relative position measurements to achieve the in-flight positioning accuracy required for the autonomous cargo pick-up.

Keywords: autonomous aerial vehicles, aerial grasping, object localization

## 1 Introduction and related work

In recent years, UAVs have found their way into various industrial applications. In addition to tasks in the area of inspection or surveying, there are also various use cases in the field of material transport, as UAVs enable fast, highly flexible and direct material transport in the air. In addition to the transport of parcels in retail or the delivery of urgent medical products or devices, UAVs are also suitable for the transport of spare parts or urgently needed parts on the company premises or within a production hall. Due to their three-dimensional scope of movement UAVs are able to operate in the unused space above the production systems, extend the flow of materials by the third dimension and offer a highly flexible and fast transportation. [1]

While different publications have already shown that autonomous UAVs can provide an efficient addition to existing logistic systems there are still only few solutions or publications describing fully functional systems for the automated aerial in-house transportation [2]. Existing research and pilot projects on the aerial transportation with autonomous UAVs commonly do not focus on the autonomous cargo pick-up. [3] [4]

To enable autonomous aerial gripping of objects, different operating principles including suction-gripping, interlocking gripping or magnetic gripping have been developed during the recent years. Among the most common design are mechanic finger or jaw grippers as they are easy to produce, robust and can handle objects independently of their material and shape. However, they require a precise positioning of the UAV or an underactuated and flexible design to be able to grasp objects properly.

To compensate the displacement between the UAV and the target object caused by inaccuracies during manual flight Pounds et al. utilize an underactuated four finger gripper for the reliable in-flight grasping of different objects. Each finger features two elastic joints to compensate the contact forces between gripper and object, resulting from the positional fluctuation of the UAV. [5] [6]

Optical tracking systems such as motion capturing systems allow a precise tracking of UAVs with an accuracy below 1 mm. The usage of such systems allows the application of fully actuated grippers in combination with autonomous drones. Qi et. al. utilize a basic single-DOF finger gripper for the successful aerial gripping of a cuboid [7].

The usage of interlocking connections for aerial grasping has been shown in [8] and [9]. Thereby pins or clamps are actuated by servo motors to penetrate the cargo objects and to attain a secure fastening of the object.

While the automated cargo pick-up is considered a key requirement for the cost-efficient in-house aerial transportation, the state of the art lacks of working solutions in this area. We thus present a system architecture for the autonomous detection and grasping of small load carriers with autonomous UAVs. The system includes an autonomous hexarotor system equipped with a camera system to detect and locate the cargo objects and an appropriate load handling device to realize an autonomous grasping.

#### 2 System architecture

To ensure an economic usage of the transportation of small load carriers with autonomous UAVs both the cargo pick-up and the drop-off need to be fully automated. Therefore, we want to achieve the following transportation scenario: The UAV receives a transportation task including an approximate position of the load carrier (e.g. the position of the stockyard) and the drop point. After the approach to the transmitted pick-up position, the UAV detects and locates the load carrier and positions itself precisely above the object to allow a reliable pick-up. After the pick-up using an automated load-handling device the cargo is transported to the transmitted destination. Subsequent to the aerial transport the load carrier is either precisely set down on a landing area defined by optical markers or simply at the transmitted drop-off coordinates.

As we focus on the indoor transportation we use an ultra-wide band (UWB) tracking system for global positioning. The system allows a tracking with 10 cm accuracy and can cover areas of several ten-thousand square meters. When flying in outdoor areas this localization system can be easily replaced with tracking data provided by satellite-based radio-navigation systems without modifying the remaining system components.

#### 2.1 Mechatronic cargo handling device

To ensure an automated pick-up and drop-off a mechatronic load handling device is developed and attached to a modified version of a commercially available hexarotor system. The load handling device is designed to be set down vertically on the load carrier. To compensate inaccuracies and fluctuations in the positioning of the UAV and to center the load carrier during the landing phase the system provides four lead-in chamfers (cf. Figure 4).

To fasten the load carrier after the landing, retaining bolts are used to create an interlocking connection with the indentations on the outer contour (1). As depicted in Figure 1, the individual bolts (2) are guided by linear bearings (3) and actuated by servo motors (4). A servo lever with a slotted hole (5) is used to convert the rotary motion into a linear one. As the servos are only actuated while the UAV is resting on the load carrier, no additional force or momentum other than the friction force needs to be overcome to move the bolts. During the flight phases, the weight force of the load carrier presses the bolts onto the bearing thus creating an efficient load restraint additional to the counter-momentum of the motor.

Fig. 1: Fixation of the load carrier by an interlocking connection between the undercut of the load carrier (1) and the retaining bolts (2). The bolts are guided by linear bearings (3) and actuated by a servo motor (4) and a servo lever (5)

#### 2.2 Object localization and tracking

Machine learning approaches are often considered as state of the art for object segmentation tasks. There are various neural networks available for object segmentation such as Mask R-CNN, DeepLab or FuseNet that allow an instance segmentation based on colour or combined colour and depth information [10– 12] as well as advanced pose estimation pipelines for segmenting and grasping various objects [13]. Light-weight models and neural network architectures also allow a near real-time image analysis low-cost hardware such as the NVidia Jetson Nano Board or different single board computers equipped with the Intel Neural Compute Stick.

However, as machine learning approaches still require large amounts of suitable and annotated training data especially when the load carriers is filled with different objects, we favour a solution based on color and shape segmentation.

Our approach for the detection, segmentation and localization of the individual load carriers is based on their known colour (blue) and shape (rectangular with known size). The detailed algorithm to segment and locate the objects is presented in detail below.

To extract the load carrier from the image, in a first step a basic colour segmentation algorithm is used. Based on a tolerance range centred on the actual colour value of the load carrier all blue pixels are segmented and converted into a binary mask. As depicted in the left image in Figure 2 the resulting mask contains most of the load carrier's pixels but is still affected by noise and falsely identified areas.

Afterwards Gaussian smoothing is applied to the binary image to reduce the effect of the remaining noise that can especially occur at the edges of the segmented objects. The Gaussian filtering prevents the incorrect detection of the noise as the edge of an object during the subsequent edge detection using the Canny-algorithm (cf. Figure 2, centre).

By applying topological structural analysis the contours of the remaining objects can be extracted. As only rectangular shapes are required, we apply the Douglas-Peucker-algorithm to approximate the remaining contours with the minimal required number of points. As an rectangle is fully described by its four corner points all contours with more or less points are discarded. To extract the rectangles actually representing a load carrier their enclosed area is analysed. As the intrinsic camera parameters, the size of the load carrier and the current flight altitude is known, the expected size of the load carrier in the camera image can be estimated and all rectangles that do not fulfill this criterion can be discarded.

Next, the scalar product of the vectors defining the rectangle's borders is calculated to get an estimate of the angle between them. This is used to only store rectangular shaped contours with corner angles of approximately 90 degrees. Based on this information, it can be ensured that e.g. for trapezoidal shapes, which also consist of only four corner points, the related contours are no longer stored for further processing.

Finally, the contours are sorted by their enclosed area and only the contour with the biggest area is kept. This is to avoid calculating the position of smaller rectangles within the box resulting from the possible influence of shading or overexposure (cf. Figure 2, right). After the segmentation the transform between the camera and the computed centroid of the load carrier is determined using the camera's inverted intrinsic matrix.

Fig. 2: Binary mask resulting from the color segmentation process (left), output image of the contour detection (center), detected load carrier with bounding box and centroid (right)

#### 2.3 Pose estimation and software architecture

As the optical tracking system features a significantly lower measurement noise than the UWB tracking system, the provided transform is not only used as setpoint but also used to increase the in-flight positioning accuracy of the UAV. This is necessary as the positioning accuracy that can be achieved by the data of the UWB tracking system and the UAV's intertial measurement unit is oftentimes not sufficient to pick up the load carriers. To improve the estimation of the current flight position we apply the covariance intersection algorithm presented by Niehsen [14].

Thereby the relative pose between the UAV and the load carrier is transformed into global coordinates and fused with the pose provided by the UWB tracking system. The inverse transformation is based on the UAVs global position at the time of the initial localization of the load carrier.

Using the covariance matrix Kxx of the global UWB tracking data and the covariance matrix Kyy of the pose provided by the optical tracking a combined covariance matrix Kzz can be calculated as

$$\mathbf{K\_{xz}} = \left(w \ast \mathbf{K\_{xx}}^{-1} + (1 - w) \ast \mathbf{K\_{yy}}^{-1}\right)^{-1}.\tag{1}$$

To find the optimal solution for Kzz the value of w is computed in such a way that the determinant of Kzz is minimized. Using the available closed-loop formulations a fused position estimation

$$P\_F = \mathbf{K\_{xz}} \ast (w \ast \mathbf{K\_{xx}}^{-1} \ast P\_C + (1 - w) \ast \mathbf{K\_{yy}}^{-1} \ast P\_G) \tag{2}$$

is calculated and forwarded to the internal kalman filter of the flight control unit (FCU). Thereby the position values provided by the UWB tracking system and the box detection are denoted as PG and PC .

The implemented process for cargo pick-up, transportation and drop-off is depicted in Figure 3. Initially the overlying task management sends an approximate position setpoint to the FCU, indicating where the cargo object is located

Fig. 3: Flowchart of the implemented software system used for object segmentation, pose estimation and controlling of the load-handling device

(not shown in the chart). As soon as the UAV reaches the provided position the process for locating and grasping the load carrier becomes active.

Based on the provided RGB-data the load carrier is segmented and localized using the algorithm presented in section 2.2. The resulting transform and the known transforms between the camera and the FCU and the load handling device and the FCU are then used to calculate an updated position setpoint that is forwarded to the FCU.

Using the covariance intersection algorithm an improved position estimation is calculated whenever valid relative tracking data is available and forwarded to the internal kalman filter of the FCU to improve the in-flight positioning accuracy.

The internal land detector of the FCU provides information whether the UAV is already landed or still in-flight. Based on this feedback, the load handling device controller drives the individual servo motors to fasten or release the cargo.

After a successful pick-up (which is determined by the increment in the required take-off thrust) the UAV heads for the provided drop-off position. When the drop-off position is referenced by additional optical markers, they are tracked using the algorithms presented in [15] and the resulting relative position is used as input to the covariance intersection algorithm, otherwise the positioning during the landing phase only relies on the UWB tracking data.

#### 3 Evaluation

The resulting system is depicted in Figure 4. The developed load-handling device is attached to a DJI F550 airframe with the corresponding electronic speed controllers and motors. As FCU we use the open-hardware system Pixracer running the 1.9.2 stable release of the PX4 autopilot [16] and a NVidia Jetson Nano on-board companion computer (OBC) for object detection and data fusion.

The implemented software utilizes the Robot Operating System (ROS) and the provided ROS communication layer. To allow a precise tracking of both

Fig. 4: Hexarotor system equipped with the developed load handling device

the load carrier and the landing area two down-facing cameras are attached to the system. The first is located in the center of the mounting plate of the load handling device and the second one is mounted to the outer frame of the UAV. The individual servo motors are connected to a suitable driver and controlled by the OBC.

While the FCU runs the real-time modules such as attitude and position controller the OBC runs additional software modules required for general flight processes and navigation. Core element of our previously developed software framework is the mission controller, a software module that is responsible for basic navigation tasks, path-following, system diagnosis and failure detection.

Figure 5 compares the achieved in-flight accuracy based on the pose estimation of the internal kalman filter when hovering around 1 m above the load carrier. To simplify the evaluation and depiction the corresponding static setpoint is subtracted from the position values. All flights are conducted with the same closed-loop controller but it's effect on the positioning accuracy and potential improvements will not be discussed in detail. When only the UWB tracking information is used, the resulting in-flight position deviates around ±5 cm from the setpoint in x- and y-direction. However, the deviations in z-direction reach values of up to 20 cm. This can be traced back to the fact that the noise in the measurement values of the UWB tracking system in z-direction is significantly higher then the noise in x- and y-direction.

When fusing additional optical tracking information, the deviations decrease to maximal values of ±4 cm in every direction. Compared to the UWB-only

Fig. 5: Comparison of the achievable positioning accuracy when only relying on UWB tracking information and when fusing UWB and optical tracking data using the covariance intersection algorithm

dataset, especially the accuracy in z-direction improves significantly enabling a consistent and precise descending. Compared to the z-direction, the positioning accuracy in x- and y-direction increases less which can be traced back to inferior measurement inaccuracies of the UWB tracking system.

While the optical tracking of the load carrier improves the flight accuracy and improves the success rate of the cargo pick-up further improvements are required. The lead-in chamfers of the current load handling device can compensate movement of up to ±5 cm, however, the automated pick-up currently only succeeds approximately every second time (evaluated on the basis of 20 test flights). Besides the in-flight position oscillations also faulty tracking information, friction on the lead-in chamfers and oscillations caused by the ground effect are accountable for the poor success rate.

#### 4 Conclusion and Outlook

Within this paper we have presented an approach to the autonomous aerial grasping of small load carriers with UAVs. Using a camera-based solution for segmenting and locating the cargo objects, a newly developed load-handling device and the covariance intersection algorithm for optimized pose estimation, the automated pick-up of small load carriers was realized. While it was shown that the autonomous pick-up of small load carriers is possible with the present system, the reliability and rate of successful pick-ups still needs to be improved.

To optimize the flight and grasping performance within future research, the influence of optimized and model-based controllers will be examined. Also improved cameras with optimized field of views and additional sensors to measure the current movement speed will be integrated and the influence of vibrations caused by the spinning rotors will be examined.

### Acknowledgment

The project receives funding from the EU ECSEL Joint Undertaking under grant agreement n737459 (project Productive4.0) and the German federal ministry of education and research under grant agreement 16SE0198.

#### References

**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Determination of Gripping Forces for Handling Surfacesensitive Objects**

Michael Krampe[0000-0003-3996-0255] , Michael Miro[0000-0003-2347-7496] , Miguel Angel Villanueva Portela[0000-0002-3769-9068] and Bernd Kuhlenkötter[0000-0002-5015-7490]

> Lehrstuhl für Produktionssysteme, Ruhr-Universität Bochum, Universitätsstr. 150, 44801 Bochum, Germany<EMAIL_ADDRESS>
**Abstract.** Even for modern robot attached gripping solutions, surface-sensitive object handling remains a challenge. A minimal gripping force is required to grip and handle a surface-sensitive object safely. At the same time, this gripping force should not exceed a certain level to avoid damaging the sensitive surface of the object. Because of this, it is necessary to determine the mostly small range of optimal gripping forces to ensure a safe handling. Therefore, this paper describes an approach to evaluate this range of gripping forces for surface-sensitive objects using by means of chocolate covered marshmallows. For this purpose, a test bench with the same characteristics as a planned gripper is developed. Throughout the project, this planned gripper will be controlled with the help of the detected range of gripping forces. In addition to these static measurements, it is planned, that the gripper will calculate the handling speeds, accomplished by a six-axis industrial robot, with its acceleration forces. The achieved results of this paper will be the starting point for the upcoming measurements with the robot. Furthermore, the influence of the temperature on the range of gripping forces was evaluated, to show the relevance of consistent environmental conditions for the handling of surface-sensitive objects.

**Keywords:** Gripper, Handling, Force control.

#### **1 Introduction**

In areas like the food industry, robots are often used for heavy load handling, e.g. packaging and palletising. In contrast, the handling of sensitive objects is still a major challenge for industrial robots. Recent articles [1, 2] illustrate the need of a sophisticated control strategy for sensitive object handling in various industries. Furthermore, the acquired sensor data is often used for the handling process alone and not broadcasted towards a MES or CPPS for further process analysis and decision making [3]. Available gripping solutions usually focus on the process of gripping and grasping without taking translational movements during transit into account. Especially sensitive objects are prone to be damaged by dynamic and abrupt changes of acceleration. This can be explained by the corresponding inertia that adds up to the applied static gripping force. As soon as the sum of dynamic and static forces exceeds the object's limits, it is damaged and may no longer be suitable for its designed purpose. In order to provide a suitable solution, the project *Sensitive Robot Based Gripping* (SenRobGrip) aims to develop a cyberphysical gripping system (CPS), which is capable of handling sensitive objects whilst improving the cycle time. To handle surface-sensitive objects, it is required to know the range of optimal gripping forces to handle it safely without damaging it. In some cases, this range can be so small, that even little changes of the environment can affect and lead to a shift of this range. The gripper should include this range of gripping force and the acceleration of the moving robot to optimise the pick-andplace task while transit. The base design of the gripper and the motion planning approach for the robot movements is described by the authors in former publications within the SenRobGrip project [4, 5]. Subsequently, this paper determinates the range of optimal gripping forces using the example of chocolate covered marshmallows and the influence of temperature. These forces are necessary to adjust a later process control with the real gripper. For that, a test bench is developed which covers the same gripping characteristics as the planned cyberphysical gripping system [6].

#### **2 Use Case**

The food industry provides many different practical examples for the demand of sensitive object handling. The range of products is varying in consistency, geometry and is often covered by pressure-sensitive surfaces, which break or show signs of maltreatment if not handled correctly. This maltreatment is directly linked to customer satisfaction and may lead to decreasing sales. In order to automate pick-and-place processes for products like these, it is necessary to consider static and dynamic forces within the handling processes. To provide a realistic use case, the developed gripper is designed to be able to handle chocolate covered marshmallows at high velocities and accelerations without damaging the surface. The proposed scenario makes it necessary to develop a handling device for these size variable and extremely fragile products in order to provide a solution for a broader product spectrum. Furthermore, environmental circumstances like temperature and humidity have a distinct influence on the product and therefore have to be monitored throughout the production.

## **3 Gripper Hardware**

The first step is the development of a new and versatile gripper as shown in Figure 1. The design aims to provide high stiffness for an accurate gripping force measurement as well as for being flexible enough to handle various object geometries. The gripper consists of a hollow cylinder to which matching flange adapters for different robot models can be attached. For communication and control purposes, a Raspberry and Arduino are placed inside the cylinder and mounted to a 3D-printed inlay. The finger drivetrains connect to the bottom of the cylinder. To improve stiffness and reduce friction, the fingers are attached to linear motion guides and are propelled by a belt drive.

**Fig. 1.** Gripper hardware design.

Four 24 V DC motors actuate the motion and generate a finger movement speed of up to 0.4 m/s which allows the gripper to close in less than 0.15 seconds.1 The device is equipped with four exchangeable piezo-resistive force sensors – one in each finger – which provide precise measurements on a scale of 0 to 10 N and an accuracy of up to 0.01 N at 860 samples per second.2 In addition to this, a 6 degree of freedom (DoF) inertial measurement unit (IMU) MPU-6050 provides the needed information about the robot's movements. Thus, the accuracy of up to 0.598 mm/s2 for translational motions and 0.0153°/s for rotational movements at 1 kHz allows robot-autarkic control. This enables the manufacturer-independent usage, as no proprietary interface is needed. To monitor environmental influences, a dedicated sensor gathers data on environmental influences like temperature and humidity in order to provide a feedback to the production management system to maintain stable process parameters.

1 Faulhaber 2250S024BX4 CSD with 1:14 227 gearhead

2 Honeywell FSG010WNPB with HX711 analog-digital-converters

## **4 Gripping Force Test Bench**

The aim of the gripping force test bench is to simplify the testing using chocolate covered marshmallows and to define the object specific range of gripping force in this example. The general construction is shown in Figure 2. The chocolate covered marshmallow lies on a small podium in the centre of the device and is surrounded by four single grippers. Each of these grippers have its own carriage combined with a rail for the linear movement powered by a servo drive with the help of a belt drive. The gripper jaw is a 30° segment of a circle with a radius of 30 mm and a height of 10 mm made of Polylactide. The podium in the center can be lifted up and down by a seesaw to see if a chocolate covered marshmallow is held safely by the grippers immediately.

**Fig. 2:** CAD model of the gripping force test bench

The sensors for force measurement are built into these gripper jaws. To read out the values of the sensors and implement a controller, an analog digital converter is used and connected to an Arduino Due microcontroller by an I2C Bus System.

## **5 Experimental Design**

The main objective of this experiment is to evaluate the range of the gripping force to handle chocolate covered marshmallows without damaging them. This range is limited by the minimal force which is needed to hold the object and the maximum force to hold the object without damaging it. First, the minimal force can be evaluated with the podium and the seesaw in the centre of the test bench. After the gripper is closed around the object, the podium can be lowered to see if the chocolate covered marshmallow stays in position. In a second step, the maximum forces can be evaluated by searching for damages of the sensible chocolate surface. Within the following experiments, the surface damages are divided into five stages. Stage zero represents a not damaged surface and the other four stages describe the level of damaging from a visible light up pressure spot (stage 1), over one crack (stage 2) and many cracks (stage 3) to a complete break of the chocolate (stage 4). These four stages of damage are shown in Figure 3.

**Fig. 3:** The four stages of damage

Before explaining the evaluation and showing the first test rounds a technical problem needs to be explained. We identified a random effect of shaking between two opposite grippers. After some adjustments of the controlling and the Arduino software it was possible to nearly eliminate the shaking after some test runs. Afterwards it occured for a few cases only, but it mostly effects the minimum gripper force which is needed to handle the chocolate covered marshmallow. These cases will be marked in the upcoming results. The shaking results from the combination of fast high-end sensors with very high resolution for the measurements and simple motors providing limited step sizes to change the gripper jaws positions. The final gripper with the more expensive motors was already in production so we were not able to use these motors for the gripping force test bench. This leads to problems in the force controlling which is shown by the shaking sometimes. For the final gripper we will not expect these problems thanks to the high-quality motors that will be implemented.

To evaluate the minimum gripping force the gripper is adjusted to test in specific ranges.

The forces are measured when the gripping process is finished, which means that the motors stopped moving and the object is gripped without the podium underneath. After some iterations the range for the minimum gripping force at room temperature (RT) is detected between 0.15 – 0.35 N based on each single gripper. In order to later include the movements of the robot into the calculation, each gripper finger is individually controlled. For the following 30 tests, a three-point controller with the determined range was tested without losing grip at any test. The results are shown in Figure 4. The four values of the individual grippers are combined in the graph to form an overall gripping force for simplification. The developed lower limit for the range of gripping force lies at around 0.88 N.

**Fig. 4:** Measurements for the minimal gripping force at room temperature

To define the upper limit for the range of gripping force every tested chocolate covered marshmallow is gripped twice to ensure that no damage is missed out in the first attempt. After some iterations the range for the maximum gripping force at room temperature is detected to range between 0.8 – 1.2 N based on each single gripper. Again, random force values within this previously determined range were tested. 26 chocolate covered marshmallows are tested twice, which results in 52 measurements. The results are shown in Figure 5.

Two objects are damaged after the first grasp and four objects are damaged after the second grasp. This leads to eight negative measurements compared to 44 positive ones. The measurements marked with "\* Stage 2" are from a marshmallow with a very rough and bumpy chocolate cover, which results in higher pressure peaks. This shows the importance of a good production quality for future handling tasks.

The objects 20 and 25 were damaged at the second grasp. The single gripper forces from the four measurements are shown in Table 1. For object 20 the crack is between gripper 2 and 3. The cause of this crack could be the rise of the gripping force from 1.03 to 1.08 at gripper 2 or 0.88 N to 0.97 N at gripper 3. For object 25 the crack is between gripper 1 and 4. In this case the rise from 1.11 N to 1.15 N at gripper 1 or 0.99 to 1.09 at gripper 4 could be the reason for the crack. This shows the small range of tenths and hundredth of Newton gripping force that can lead to a crack in the chocolate. Because of these small fluctuations the upper limit for the overall range of gripping force is set to 3.5 N as shown in Figure 5.

**Tab. 1:** Detailed look on two tested objects

Furthermore, the effect of temperature on the range of gripping forces is evaluated. The chocolate covered marshmallows are cooled down to 8°C and the procedure for finding the range of gripping forces is the same. As shown in Figure 6, the lower limit for the range of gripping forces at 8°C is higher compared to the equivalent limit at room temperature. While testing the maximum gripping forces for the upper limit of the range, the previously mentioned shaking of the test bench is a problem again. This shaking results in some gripping force peaks in Figure 7. Furthermore, some chocolate covered marshmallows got a crack from the temperature differences. Overall, the results shown in Figure 7 lead to an upper limit at around 5 N at 8°C. The different results between the measurements at 8°C and room temperature were expected due to the adhesive and melting characteristic of chocolate.

**Fig. 6:** Measurements for the minimal gripping force at 8°C

Linear (upper limit)

**Fig. 7:** Measurements for the maximum gripping force at 8°C

## **6 Results**

First of all, the results show that it is possible to determinate the range of gripping forces for surface-sensitive objects like chocolate covered marshmallows. Also, the problems with the shaking grippers will not be expected when using the higher quality motors in the final gripper. Even with the applied cheaper motors for these experiments have shown that the required sensitivity is possible. The evaluated range of gripping force for the example of chocolate covered marshmallows is found between the limits of 0,88 N to 3,5 N at room temperature and 1,2 N to 5 N at 8°C. These results show the limits for the controlling of the surface-sensible gripper developed in this project. The force sensitive object handling needs to determine and actuate the controlling in these defined limits to allow a safe handling of surface-sensitive objects. It should be emphasised that these tests are made on a specific test bench without the acceleration forces of the robot. This will reduce the range of motion and emphasize the importance of controlling the individual fingers. We are currently trying to transfer these results to the real gripper (shown in Figure 8).

**Fig. 8:** The real gripper

Furthermore, the results show the importance of the temperature controlling of the product environment. The handling of surface-sensitive objects in winter or summer could make a difference in product quality if the season affects the ambient temperature, yet this paper shows that with some information about the effects of the temperature – and by integrating the measurement into the cyberphysical gripper control - it is even possible to enable a safe handling.

#### **Acknowledgements**

The research project *Sensitive Robot based Gripping* (SenRobGrip) is funded with kind support of the European Union's EFRE program *Investments in growth and employment*. Within the time span of March 2017 to February 2020, IBG Automation GmbH, IBG Robotronic GmbH, and the Chair of Production Systems of Ruhr-Universität Bochum are participating in the SenRobGrip project.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Disturbance Force Estimation for a Low Pressure Suction Gripper Based on Differential Pressure Analysis.**

Benjamin Wirth1 , Sven Coutandin1 and Jürgen Fleischer1

1 Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
**Abstract.** In times of ever increasing performance requirements imposed on newly developed products, fiber reinforced plastics have become essential for lightweight applications. While mass related stiffness and strength are huge advantages, automatic handling of the raw materials, such as fabrics, can be very challenging. Low-pressure suction grippers have been proven to be capable of handling fragile air-permeable materials in a non-destructive manner, giving them a clear advantage over potentially damaging competing gripping technologies such as needle grippers. However, they also consume large amounts of energy to ensure successful handling operations. This research aims to estimate disturbance forces applied to the manipulated workpiece by monitoring its internal suction pressure, which will in turn allow for running the grippers at a bare minimum power while still being able to react to disturbances enabling safe handling operations.

**Keywords:** Process Monitoring, Gripping Technology, Textile Handling.

## **1 Introduction**

While weight related stiffness and strength of continuous-fiber reinforced plastics (FRP) are superior to most classic metallic materials [1]. There is a lack of automation solutions in the handling of "*limp and air-permeable semi-finished textile products*" [2]. Low pressure suction grippers have been identified as a technology with great potential for automated handling of semi-finished textile materials [3, 4].

They can maintain a moderate suction pressure at high volumetric flow rates [5], which allows for sufficient holding forces even when air is able to pass into the suction cavity of the gripper either by leaking airstreams passing the material or by penetrating the porous material directly.

For automated handling of air-permeable sheet materials, low pressure suction grippers have been used in a variety of research applications [6–9].

However, due to a lack of compensation mechanisms for '*disturbances during the handling process'* they are often run at maximum suction power to ensure reliable material transportation even when exterior forces may interact with the handled textiles. This mode of operation is very energy-inefficient and thus not desirable. [10]

This work focuses on estimation of disturbing forces applied during handling of airpermeable blanks by monitoring the interior pressure inside the grippers' central cavity.

## **2 Low Pressure Suction Grippers**

Grippers using partial vacuums at moderate volumetric flow rates have been identified as one of the most promising technologies for automated textile handling [11].

Fig. 1 shows a sketch for a simple suction gripper.

**Fig. 1.** Sketch of a simple suction gripper and Forces acting on a gripped Fabric. Based on [10]

These systems usually feature central cavities, which are partially evacuated to a low pressure throughout this paper the differential pressure between suction cavity and surrounding atmosphere will be depicted by.

A suction plate, which may contain small holes preventing major material deformations and restricting airflow in non-covered areas, is usually attached to the gripper. This avoids "short circuits", that would lead to a rapid decline in suction pressure] 12[.

In a one dimensional lifting operation the handled materials are generally subjected to four different forces [10]:

$$F\_H = A\_{eff} \cdot \Delta p \tag{l}$$

The contact force with which any gripped material is pressed against the suction plate can be described by [10]:

$$F\_{CP} = F\_H + F\_G + F\_N \tag{2}$$

### **3 Evaluation of the State of the Art**

#### **3.1 Pressure Monitoring for Suction Grippers**

Götz measured differential pressures inside a low pressure handling system for leather blanks during operation for validation purposes, however, he did not relate them to significant process properties. [9]

The gauge pressure inside a suction Gripper has been used for gripper control by Ehinger. The interior pressure was controlled by a PI-controller to match predefined values found by experimental analysis. Allowing for secure draping of a textile material even with some non-covered holes on the suction plate. [6] While controlling the interior pressure will lead to a response on disturbing forces acting on the handled material, these forces are not quantified. For that an additional observation of the supplied suction energy would be necessary.

Fleischer et al. have shown a dependence of the internal negative pressure inside a coanda-effect based grippers' suction cavity on the number of manipulated layers for air-permeable materials. In experiments placing up to 2 layers of textile materials beneath a suction plate and gradually decreasing the input power-stream to the gripper it was found, that for any given input power the interior pressure approaches a higher vacuum with additional layers adhering to the manipulator. This allows the system to directly estimate the number of sheets separated from a stack-magazine. [8]

#### **3.2 Gripping Force Estimation for Low Pressure Suction Grippers**

Fleischer et al were able to measure holding forces between a suction gripper and carbon fiber materials by attaching brass electrodes to the suction plate. The measured electrical resistance between the electrodes has been shown to directly depend on brasscarbon contact resistances, which in turn decrease with higher holding forces applied to the handled textile. By designing control loops for energy efficient operation, they were able to create a dynamic system capable of quickly reacting to disturbances, while also reducing energy consumption by nearly 90% compared to a conventional static energy supply. Furthermore, they were able to control the holding force, such that a reliable separation of singular sheets off of a stack-magazine was possible. [8]

This approach, however, relies on the electrical conductivity of the handled textile and is therefore not applicable for a wide variety of common materials.

Brink et al. developed a "*Force Controlled Variable Fan Gripping Module*" incorporating an electrically powered fan as a vacuum generator. Strain gauges were integrated into the fan grill, which serves as a suction plate in this system. With increasing handling forces, increasing deformations in the grill can be observed, which were in turn registered as a change in resistance for the strain gauges. Using this principle, Brink et al. were able to monitor process forces occurring during the handling of non-crimp fabrics. They furthermore noted, that a dependence of the gripping force on the number of attached layers is observable. [7]

Retrofitting sensor integrated suction plates introduces undesired complex alterations to commercial gripping-systems. A less invasive approach for measuring disturbances acting on the gripper is required. Monitoring the pressure inside the suction cavity promises to be an efficient solution to this problem. This research shows an approach for generation of a process model relating negative pressure to disturbance forces based on an offline calibration. This system can be used for online monitoring of handling operations. It may also be used as part of a state estimation algorithm allowing the gripper system to control contact forces at the gripper-material interface.

## **4 System Description**

The SCG 1xE100 gripper used in this research (see Fig. 2) is manufactured by J. Schmalz GmbH.

**Fig. 2.** Left: SCG 1xE100 gripper with suction plate Right: cross sectional sketch of a coanda-effect based gripper based on [8]

The gripper is activated by supplying air at a positive operating pressure ே to port 1 this high pressure air fills a primary cavity, which it can escape through a thin slit leading to a central secondary cavity. The inwards moving air will tend to adhere to the slightly curved top surface. This primary flow ܳே facing upwards will have a suction effect on the air at port 2 creating a negative pressure and a secondary airflowܳ. During operation an additional suction plate is screwed onto the gripper at this interface. The combined airflow will leave the suction cavity by port 3. Another connection from the outside directly to the suction cavity is provided by port 1/2, which is usually used for a blow off functionality [5].

Initial tests probing the internal suction pressure at different positions by laterally inserting hypodermic needles into the suction cavity have shown, that even small translational variations of the probing-position can have small (varying by about ͲǤͲͷ݇ܲܽat suction pressures of ൌ ͲǤͳ݇ܲܽ) but significant effects on the resulting pressure readings. These differences are presumably caused by the non-homogenous suction plate geometry and turbulent flow regimes directly behind the it. Therefore, an optimal probing position should be situated as far away from the plate as possible. For transferability of measurements amongst different SCG- 1xE100 grippers the blow-off port 1/2 has been chosen as an ideal probing position.

#### **5 Procedure**

An adapting gripping system capable of monitoring and reacting to changing disturbance forces may be realized by using the controller architecture shown in Fig. 8.

In such a system a desired contact force ܨǡௗ௦ can be set and controlled, which will directly implement dynamical reactions to any disturbance forces acting on the handled material (see equation (2)). A PID-controller can be used to dynamically change the operating pressure of a suction gripper depending on an error measurement based on desired and estimated contact forces between the gripper and the handled textiles.

**Fig. 3.** Proposed control loop for the final gripper system. Parts analyzed in this research are highlighted

This research aims to show how a disturbance force estimation in the shape of a model ܨሺܳேǡ ሻ can be obtained by taking the following steps:

1. Recording of example pull-of-experiment data

#### **6 Experimental Setup**

In order to characterize the correlation of disturbance forces ܨே and the suction pressure an experiment as shown below is set up.

The experimental setup sketched in Fig. 3 consists of an SCG 1xE100 gripper that can be moved up and down by a distance ݏ with a lead screw driven by an electric stepper motor (not depicted in the figure). The forceܨ, at which the gripper is pulled towards the base is measured by a force sensor. The material specimen is clamped between two metal plates with a central rectangular cutout (100x100mm seen in Fig. 3.). The operating pressure ே is supplied by a proportioning valve, while the flow rate ܳேis measured by a flow meter and suction pressure being monitored by a differential pressure sensor.

**Fig. 4.** Sketch and photograph of the used test stand.

**Fig. 5.** Leakage occurring after deformation of the gripped fabric

Pull-off tests have been performed by supplying a specific operating pressure ே and moving the gripper upwards by an increasing distanceݏ, while recording ܳேǡ ܨ and. This movement leads to a gradual separation of the material from the suction plate allowing for additional air to bypass the manipulated sheet, avoiding the pressure drop across the textile, leaking into the suction cavity and thereby decreasing the measured value of. This leakage happens in the test stand as well as when gripping freely hanging sheets as depicted in Fig. 4.

## **7 Experimental Results**

#### **7.1 Model Construction**

This section discusses the aforementioned offline calibration for the carbon fiber material shown in Fig. 3.

Exemplary pull-off tests yield the results presented in Fig. 5, with pull-off forces ܨand differential pressures plotted against the traveled distance s (Fig. 5 a). and ܨ are plotted against one another from test start up until the maximum force ܨ௫ is reached (Fig. 5 b). The same plot is repeated on the right for measurements afterܨ௫ was reached (Fig. 5 c). The clusters of lines shown in Fig. 5 correlate to supply pressures of 0.5; 1; 1.5; 2; 2.5 and 3 Bar.

**Fig. 6.** Measured forces and negative pressures for an exemplary pull-off test

At the beginning of any given pull-off-test the pressure stays nearly constant, with no noticeable separation of the gripped material from the suction plate. Recorded forces are here mainly dominated by the material deformation. After 0.5-1.5 mm of travel slight separation can be observed with a decline in. The recorded forces keep rising until a maximum force ܨ௫ is reached close to an inflection point seen in the ൈ ݏplot between 1 to 4 mm of travel. At s = ~5 mm full separation can be observed with forces quickly approaching 0 N and settling at a non-zero value generated by the flow resistance of the installed suction plate.

Since the supply pressure at the gripper is set and not directly measured, the operating air flow measurementܳே, which behaves almost proportional to ே has proven to be more reliable in dynamic situations. As seen in Fig. 5 the negative pressure increases practically linearly with increasing operating pressure, while decreasing almost quadratically with increasing disturbance forces. Based on this observation a simple polynomial modeling approach ሺܨǡܳேሻ ൌ ଵܨଵܳே ଵଵܨܳே ଶܨଶ is chosen. A parameter optimization for the specific results shown in Fig. 5, gives a polynomial numerical model, relating pull off forces and operating pressure to the measured differential pressure (beforeܨ௫ was reached):

$$p\_{U,a}(F, Q\_N) = -1826Pa - 279.8 \, F \, \frac{Pa}{N} + \\\ 174.02 \, Q\_N \frac{Pa}{l \, ml \, n^{-1}} - 56.72 \, F^2 \frac{Pa}{N^2} - 3.168 \, F \, Q\_N \frac{Pa}{N \, l \, ml \, n^{-1}} \tag{3}$$

Here,ܳே is given in norm liters per minute.

The model represented by equation (3) reaches an adjusted ܴതଶ = 0.9930 over 224 sample points collected in 18 individual experiments.

Analogously, another model can be obtained for the part of the tests after the maximum force has been reached.

$$p\_{U,b}(F, Q\_N) = -301.3Pa - 161.5\ F\frac{Pa}{N} + \\\\ 1.24\ \,\mu\_N\frac{Pa}{l\,\,mol\,\,^{-1}} + 15.64\ \,F^2\frac{Pa}{N^2} - 0.8202\ F\,Q\_N\frac{Pa}{N\,\,l\,\,m\,\,^{-1}}\tag{4}$$

Using the same approach, similarly close approximations have been found for a variety of different materials. The two models described above can be combined using a linear separator at ൌ ଵ ଵ ொಿ షభ ڄ ͶͲͲͲܲܽ െ ͲͲܲܽ.

$$p\_U(F, Q\_N) = \begin{cases} p\_{U,a}\{F, Q\_N\} \ , \text{if } \frac{1}{160} \frac{Q\_N}{l \text{ min}^{-1}} \cdot 4000Pa - 600Pa \ < p\_U \\ p\_{U,b}\{F, Q\_N\} \ , \text{else} \end{cases} \tag{5}$$

For any combination of Ͳ and ܳே Ͳ a corresponding estimated disturbance force ܨ can be found by calculating the force, which minimizes the absolute difference between the predicted and measured negative pressure:

$$F\_{\mathbf{e}}(\mathbf{p}\_U, Q\_N) = \arg\min\_{\mathbf{F}}(|\mathbf{p}\_U\{\mathbf{F}, Q\_N\} - \mathbf{p}\_U|)\tag{6}$$

#### **7.2 Model Application**

Manually moving the gripper up and down on the test-stand allows for a comparison between measured forces and estimations made by the previously described models, solely dependent on and ܳே. Results of these simple validation tests at an operating pressure of ே ൌ ʹܾܽݎcan be seen in Fig. 6.

It is clearly visible, that significant disturbance forces and in turn separations are directly seen by the estimation. While both shown curves overlap for most parts of the diagram slight deviations can be seen, which may be explained by non-elastic behavior of the tested fabric.

**Fig. 7.** Model evaluation by manual movement of the gripper

In another test manually pulling the fabric down similar observations were made as shown in Fig. 7.

**Fig. 8.** Model evaluation by manual movement of the fabric

Measured and estimated forces closely match one another while forces are applied evenly around the gripper. Under this load case the model estimation shows an average error of about ͳܰ.

With punctual loads (as seen at ݐ ൌ ʹͲݏ in Fig. 7) forces will not be estimated as precisely. Here, the estimated force is off by almost 100%. Nevertheless, it can be concluded, that the estimation model qualitatively shows good correlation with the measured data.

#### **8 Conclusion and Outlook**

This work has introduced a differential pressure-based estimation method for disturbance forces acting on fabric being manipulated by a low-pressure suction gripper. After obtaining experimental data, polynomial models can be formulated which serve as a look-up point for the desired force estimation. It has been shown, that such estimation model can be used to deduce qualitative information about forces acting on the gripped material. While these estimates do not show perfect resemblance of the forces, especially for different load cases, the two show a strong correlation.

In future research these results will be generalized to allow for model estimation based on material properties and gripper characteristics. Which will then be used to define a control structure capable of adjusting the systems. As shown by the works of Förster such a control structure can furthermore be used for robust separation of singular sheets of air-permeable material from a stack magazine [3], paving the way for highly automated production lines in both classical and technical textile industries.

## **9 Acknowledgement**

This research was financially supported by the German Research Foundation (DFG) (funding no. 397485737)

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Integration of Collaborative Robotics in Vocational Training Using Mixed Reality**

Torsten Sebastian Sievers1 , Bianca Schmitt2 , Maren Petersen2 and Kirsten Tracht1[0000-0001-9740-3962]

1 University of Bremen, Bremen Institute for Mechanical Engineering (bime), 28334 Bremen, Germany 2 University of Bremen, Institute Technology and Education (ITB), 28334 Bremen, Germany<EMAIL_ADDRESS>
**Abstract.** The change in production towards smaller batch sizes demands a high degree of flexibility, especially in assembly. In this context, human-robot collaboration (HRC) is becoming increasingly important. The goal is not to displace the human being by the robot, but instead receiving support from it. Nevertheless, employees at all levels have to be prepared for the HRC. Trainings offered from robot manufacturers focus on robot specific operations and are not sufficient for the development of competencies for innovative and creative process planning and optimisation. Therefore, a mixed reality learning environment can be used for the implementation of such training, since real production facilities are only available to a limited extent for training purposes. In the context of the implementation of a mixed reality learning environment, the question arises as to the necessity of haptic feedback. This article presents a concept for integrating HRC in German initial and continuing vocational education and training using mixed reality.

**Keywords:** Human-Robot Collaboration, Learning Factory, Assembly.

## **1 Introduction**

The importance of flexible assembly is growing in particular due to the increased production of small series through customer-specific productions [1]. The stepwise automation of the production process enables a cost-effective production of customer-specific products. In particular, the design of human-robot collaboration (HRC) plays a central role with regard to future assembly processes, whereby the form of cooperation between human and machine is decisive. The implementation of HRC enables a high overall productivity and a better product quality. [2] However, the safety of the employees showed as a difficult problem [5]. Therefore, the planning of collaborative processes can be supported and validated in a virtual environment [6]. The virtual environment is based on a digital twin using data from new product variants and resource configuration. It allows a simulation of the collaborative processes including the human. The results show, that in order to ensure a safe use of collaborative robots and increase their potential, qualified employees are required. [7] Therefore, employees at all job levels must be qualified, because it is unlikely that HRC will result in production without employees and it does not represent a desired goal. Rather, the human being remains the most important control and decision-making authority. [3] The aim should be to place the human being at the centre of the process in the sense of designing the work processes [4].

In most cases, the system integrator trains the employees in the use of a specific robot. This training is limited to the teaching of programming knowledge and productspecific basic qualifications. Initial research, as part of the KoRA third-party funding project indicates that these training courses do not meet the expectations from employees in the company. In this context virtual environments offer a safe and reconfigurable learning environment. GAMMIERI ET AL. already published a virtual environment for HRC enhancement [8]. They integrated a forward and inverse kinematic of a KUKA collaborative robot and detected collisions. Furthermore, the digital model can be connected with a real robot. Due to the integration of several collaborative assembly scenarios, the virtual environment can be used for safety considerations as well as for training operators. Another approach form MATSAS AND VOSNIAKOS introduced a virtual reality training system for HRC [9]. The System visualises safety issues like contacts or collisions and can be used to investigate acceptability of HRC. Moreover, OYEKAN ET AL. investigate the reactions from human to movements of the robot [10]. The goal is to test collaborative processes in advance to ensure a safe process. However, none of these approaches provided neither a differentiated learning environment based on the individual qualification level, nor an appropriate teaching concept.

#### **2 Mixed Reality Learning Environment - KoRA**

In September 2019, the University of Bremen started a research project named KoRA with the aim to prepare employees at all levels for HRC. The teaching and learning concept enables employees to view complex assembly systems holistically in order to recognise the potential of HRC and use it in a targeted manner. The approach uses a learning software made in Unity in combination with learn and work assignments (see section 3) that rely on the visualisation techniques of Mixed Reality. Within the assignments, the dissolving of rigid borders between manual and automatic assembly is focused by the use of collaborative assembly.

The possibility of participation of the employees offers the potential for dismantling obstacles to collaboration with robots [9]. This is of particular important because the employees conviction about the advantages of HRC is a crucial component for the successful introduction into the factory [11]. The complex assembly scenarios are prepared within the framework of learning and work assignments. These take into account operational reality and real work processes and thus enable project-based learning in realistic scenarios. In particular, scenarios on the topics safety, collaboration suitability and technical feasibility are intended (see Fig. 1). The concept addresses trainees, skilled employees and supervisors, as well as trainers and teachers in relation to the German dual vocational training.

**Fig. 1.** Concept of the KoRA mixed reality training

#### **3 Didactical concept**

The didactical concept is based on learn and work assignments (see Fig. 2). They are particularly suitable for dual vocational training due to their occupational world relevance, the link to problematic situations and the project-based learning organisation [12]. Especially, learners benefit from the problems they face in their daily work because the relevance, i. e. the present and future significance (according to KLAFKI), becomes apparent to them [13].

The learn and work assignments are also characterised by their project form, which promotes the independent development of a problem solution, both as an individual and as a team effort. The acquired knowledge is applied in the development of solutions, the planning of the work steps as well as the implementation and adaption to the individual starting position.