The introduced hardware in combination with the abovementioned processes offer a solution to increase the degree of automation of the current processes. The classification and identification of different groups of parts, with respect to their automation capability contributes to process standardization. High-volume processes such as insert placement and gluing can be fully automated with complete flexibility regarding type, number and position of inserts. Commonly used hardware e.g. gantry kinematics, screwing and adhesive dosing units or grippers can be used for this purpose. However, because of the varying geometry of attachment parts, manual assembly cannot be neglected completely. For assembly tasks which require simultaneous handling of multiple parts in different joining directions, manual assembly is preferred. Nevertheless, the laser projection system or collaborative robot offer flexible worker assistance for material handling, joining or highlighting. This introduced combination of fully automated and semi-automated processes contributes to an increase in productivity during the pre-assembly of aircraft interior components and defines a baseline for future research in the field of cabin interior production.

The analysis of the required process information for the automated assembly and assistance systems is a first step to a digital process description. These information complements the previously developed digital manufacturing process chain. Part and assembly specific parameters can be derived from existing construction data or are defined once during production planning. Process specific parameters can be easily adapted and executed by the proposed hardware. This contributes to a paperless documentation, distribution of information on the shopfloor and allows quick process reconfiguration for new assemblies. However, production management systems and databases have to be implemented to ensure reliability.

The hybrid approach on pre-assembly shows potential for high process flexibility although manual labor is reduced, due to expandable hardware, such as additional assistance systems like industrial cameras, sensors or a moving head spot.

#### 5 Summary and Future Work

In this work, a semi-automated pre-assembly system has been introduced to increase the degree of automation and productivity to enable suppliers to remain competitive with low wage countries. The range of components and assembly processes have been analyzed and classified with respect to their automation capability. Standardized groups of processes have been identified and hardware has been proposed. For further research, a partial demonstrator of a hybrid-assembly station was presented.

In future work, the fully automated solutions for insert placement have to be implemented and the automation of adhesive application has to be investigated. Furthermore, a holistic concept for continuous digital cabin production must be developed which allows to implement further automation solutions in a manual production environment. This requires research regarding efficient assembly planning methodologies for semi-automated assembly systems focusing on flow of information, digitalization and development of data-oriented design criteria to reduce commissioning effort of flexible automation technology. In future work, sensors can be integrated in the hybrid-assembly station to allow e.g. presence or assembly progress monitoring, action recognition or quality ensuring inspection tasks.

Acknowledgements The work presented in this paper is carried out in a project funded by the Federal Ministry of Economic Affairs and Energy of German (BMWi).

#### References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

An Approach for Direct Offline Programming of High Precision Assembly Tasks on 3D Scans Using Tactile Control and Automatic Program Adaption

Maximilian Metzner, Dominik Reisinger, Jan-Niklas Ortmann, Lukas Grünhöfer, Andreas Handwerker, Andreas Blank and Jörg Franke

#### Abstract

This contribution defines a methodology for the direct offline programming of robotic high-precision assembly tasks without the need for real-world teach-in, even for less-accurate lightweight robots. Using 3D scanning technologies, the relevant geometrical relations of the offline programming environment are adjusted to the real application. To bridge remaining accuracy gaps, tactile insertion algorithms are provided. As repetitive inaccuracy compensation through tactile search is considered wasteful, a method to automatically adapt the robot program to continuously increase precision over time, taking into account multiple influence sets is derived. The presented methodology is validated on a real-world use case from electronics production.

#### Keywords

Lightweight robots Peg in hole Photogrammetry Machine learning

D. Reisinger Technical University of Applied Sciences Nuremberg, 90489 Nuremberg, Germany

A. Handwerker Digital Industries Division, Siemens AG, 91058 Erlangen, Germany

M. Metzner (&) J.-N. Ortmann L. Grünhöfer A. Blank J. Franke FAU Erlangen-Nürnberg, 91056 Erlangen, Germany e-mail<EMAIL_ADDRESS>
#### 1 Motivation

Many manual assembly tasks use the sensory abilities of the human worker to achieve high precision. Automation of such tasks usually involves special high-accuracy manipulators and tedious calibration of the application. Such systems are usually expensive and cannot be included into the material flow of existing manual production lines, in the sense of a hybrid system. To allow a safe robot operation in a manual system without the need for complete encapsulation of the automated process, lightweight robots (LWR) are often chosen. The lightweight construction of these manipulators however, further decreases their inherent accuracy [19]. The manual teach-in of such tasks is also time-consuming, which is why most programs are prepared in simulation tools by offline programming (OLP) and are then adapted to the actual application via re-teaching of key interaction points. This approach however greatly decreases system flexibility, as new product or variant introduction thus requires a manual touch-up on the real system resulting in production downtimes. Furthermore, the teach-in of such processes is often itself not accurate enough by itself, resulting in iterative teach-point optimization.

To address this, we introduce a methodology for the direct OLP of high-precision assembly tasks that still allows for the use of low-accuracy LWRs. The system is also capable of automatic program adaption for increasing process accuracy over time.

First, we introduce the relevant state of the art concerning high-precision robotic assembly, often referenced as peg-in-hole, as well as 3D scanning for production system planning. Based on this, we derive a need for action and requirements for such a method. In the third chapter, the new approach is described, going into detail on main features. We then validate the approach on a real scenario from power electronic production. The results are discussed and the contribution is concluded on an outlook of further planned research activities.

#### 2 State of the Art

As our approach combines strategies of robotic high-precision assembly and 3D scanning, relevant research approaches in both fields are presented. From this, we derive a research gap to allow direct OLP of high precision assembly tasks and detail our contribution.

#### 2.1 Robotic Peg-In-Hole Assembly

Robotic peg-in-hole assembly is categorized in five strategies by Li and Qiao: Sensing-information based, compliant mechanism-based, environment constraint-based, sensing constraint-based and human-inspired [16]. Xu et al. use a differentiation between contact-model-based and contact-model-free strategies [25]. The latter involve learning-by-demonstration techniques and reinforcement-learning approaches, which are custom to every application and thus hardly reusable. Contact-model-based approaches involve environment-constraint-based strategies and sensing-based compliant control. As shown in Metzner et al. [19], sensing-information-based methods offer great potential for economic automation of high-precision assembly even without highly accurate manipulators. Another advantage of such systems is the ability to reuse compensation routines and thus to increase engineering efficiency. In this contribution, we focus on force/torque-sensing (FTS) based systems, as in contrary to vision-based systems, high precision can be achieved without the need for online setup of the vision system and without the need for markers.

FTS-based precision assembly has been demonstrated by Jasim et al. A spiral-shaped search pattern with continuous surface contact is introduced [11]. This concept is abstracted in multiple contributions, e.g. to ambidextrous robots [26]. More general strategies have been implemented through the LIAA project, leading to the PiTaSC library [20]. Lately, commercial software systems, like Drag&Bot or Artiminds RPS have been introduced, that also offer FTS- and vision-based compensation modules [6]. However, all of these approaches are based on development, combination and parametrization on the real application, leading to longer system setup times and an inherent invariability to variant changes, new product introduction or processual changes, as a touch-up of the real system is required, leading to lengthy down-times. Metzner et al. present an approach for offline program synthesis to allow an offline preparation and parametrization of the program [19]. However, the application still needs to be calibrated with the real robot, as the simulation scene for OLP frequently deviates from the real application more drastically than can be compensated with the search strategies. Furthermore, all of the above approaches are static, meaning that a data-driven internal optimization of the robot frames is not performed, leading to wasteful repeated search times under the presence of poorly taught points or changes in the system.

#### 2.2 3D-Scanning of Industrial Settings for Production Planning

3D-Scanning is used mostly for planning of new production systems in brown-field scenarios [18]. Scans are often taken using LiDAR-Scanners, also used for surveying, archaeology and building process documentation. This procedure allows for the capturing of very large areas on a factory scale, are however only partially suited for fine adjustment of small individual components due to resolution restrictions. Such scans are most suited for global planning of layouts, media in- and outlets and clearances.

Precise local scanning is done using either simultaneous localization and mapping (SLAM) approaches or structure-from-motion (SfM) routines. In SLAM approaches, an initial frame from an RGBD-sensor is iteratively complemented with succeeding frames that are fitted into the generated map [3]. These maps are mostly created using handheld RGBD-sensors and offer a time-efficient way to capture 3D scenes. However, such sensors suffer from significant sensor noise, which, complemented with fitting error propagation from the SLAM routine leads to subpar scanning accuracy.

SfM routines base on feature matching on numerous RGB images with identical optical parameters, which are then used to calculate the relative capturing poses for each frame based on a sparse reconstruction on the feature level. In a next step, this information is used to perform a dense reconstruction of the scene using multi-view stereo approaches [22, 23]. Such approaches have shown to offer high accuracy, especially on texture-rich settings, whilst only requiring a simple RGB camera for capturing. However, industrial application and validation for fine adjustment of virtual planning approaches has not been shown.

#### 2.3 Research Gap and Need for Action

This contribution aims to enable a direct offline programming of high-precision assembly by providing a methodological model for deviations influencing accuracy in assembly. A compensation of the different deviations is conducted using CAD calibration from 3D scans and FTS-based tactile insertion strategies. Furthermore, an approach for data-driven optimization of the robot interaction frames is presented. We extend the state of the art by providing a systematic classification of accuracy influences in high-precision assembly tasks and corresponding strategies for their compensation. We furthermore quantitatively evaluate the usage of SfM approaches for the detailed 3D capturing of production systems for precise OLP. We extend existing strategies for FTS-based deviation compensation by data-driven optimization and validate the functionality on a realistic test scenario. We also demonstrate the integrated applicability of the methodology by an offline implementation of a robust sub-millimeter assembly process for an electronics device without any manual touchups.

#### 3 Methodology

The methodology focuses on compensating three general types of accuracy influences: global independent deviations, systemic dependent deviations and random deviations. The global independent deviations are static biases for the entire system and independent of other operational parameters or time. They can be distinguished in macro-deviations and micro-deviations. Macro-deviations stem mostly from differences in geometric dependencies between the virtual ideal planning scene and the real system. Micro global deviations are caused by absolute accuracy errors of the manipulator as well as manufacturing and assembly tolerances of system components.

Dependent deviations occur due to manufacturing and assembly tolerances of the parts, e.g. varying by supplier, the work piece carriers or the material supply channels. They are thus dependent on at least one other parameter. Random deviations include the repeatability precision of the manipulator and other random external influences.

To compensate for global independent deviations, a one-time adaption of the robot interaction points is sufficient. For macro-deviations, this is achieved by integrating the real system geometry into the virtual planning scene through 3D scanning, as detailed in Sect. 3.1. Independent micro-deviations, dependent deviations and random influences are compensated by using tactile assembly strategies, see Sect. 3.2. As the tactile compensation of deviations is as such a wasteful, time-consuming process, a data-driven approach to further differentiate the independent micro-deviations and dependent deviations, as well as their dependent parameter, is shown in Sect. 4.2. The goal is to eliminate all systemic deviations, leaving only the random deviations, which however occur on a much smaller scale than the other influences.

#### 3.1 Precision 3D-Scanning and Surface Reconstruction

To allow a low-cost 3D scanning of the real system, we use a structure-from-motion (SfM)-based approach for photogrammetric reconstruction from high-resolution RGB images. The real system is photographed with static camera parameters from different viewpoints ensuring a high overlap between pictures. Through inter-picture feature matching, the geometric dependency of the picture viewpoints, and thus the 3D structure of the system is derived. The resulting point cloud is then reconstructed into a surface model to allow fine adjusting with the virtual system model in the simulation/OLP tool, in this case Tecnomatix Process Simulate. Most relevant for robot OLP is the robot base, the work piece and the material supply 6D pose, as those are used to define the interaction points in the robot world coordinate system. Fine-tuning can be done manually or aided by e.g. ICP routines [2]. To implement ICP-based approaches for known parts, the CAD model is subsampled into a point cloud. Unknown parts, for which no CAD model is available, are categorized based on their geometry. Elements compounded from a few primitive geometries, such as the floor, tables and pillars, are estimated using a RANSAC approach. Here, the individual elements are fitted with the most likely primitive model based on fitting error minimization [7, 21]. More complex elements of the 3D scan, such as freeform surfaces and cables, are meshed using implicit function approaches, allowing for an efficient processing of large point sets [1, 4, 5, 12−15]. All other reconstructed geometry from the scan, which is not used for interaction point determination is used for collision-free path planning.

#### 3.2 Tactile Assembly Strategies and Data-Driven Program Adaption

The general approach for tactile assembly used in this contribution is detailed in Metzner et al. [19]. The assembly process is divided into three phases: approach (A), search (S) and insertion (I). Dependent on part and assembly-specific parameters, the different variants for each phase are selected and parameterized. For compensation of micro-deviations, the search phase is most important, as the total deviation is determined here.

The results of each individual search phase in every dimension is recorded together with relevant process parameters, such as carrier ID, base part manufacturer ID or material supply ID. The global deviation for each point is determined after a small sample of test runs as the mean of the found positions in each dimension.

The dependent variable compensation requires more data, depending on the magnitude of the impact of each variable. Through a statistical hypothesis test, significant deviations between the actual goal value and the distribution of recent points of one class of one parameter are identified. As variance homogeneity cannot be assumed, the dependent variable may also influence the distribution, and sample size will also differ, a two-sided test statistic for independent samples according to Welch is used [24]. In case a significant deviation is found, the difference of the old and new arithmetic mean is saved as an additional compensation factor linked to the investigated parameter value. A new target position for each parameter set is calculated on the robot controller by adding up all such parameter-linked deviations.

#### 4 Validation

First, each functional part of the methodology is validated individually on a common setup. A fully integrated execution of the entire methodology is also demonstrated. We validate our system on a scenario from electronics production, the assembly of through-hole-devices (THD) into printed circuit boards (PCB). This process is very challenging to automate, as assembly tolerances are usually only tenths of millimeters, to allow the robust soldering, and the parts are inherently fragile. We use a UR10 LWR equipped with an ATI FT AXIA 80 force-torque sensor (FTS) and a Weiss CRG 200–085 gripper for our setup. The system also consists of a workpiece carrier system for the PCBs and a material supply in the form of a tray.

#### 4.1 Evaluation of Scanning Accuracy

The photogrammetric 3D scanning is validated by direct comparison of a point cloud and surface model generated by our SfM pipeline using a Sony Alpha 6000 camera (6000 4000 pixel), see Fig. 1, with those of a FARO Quantum V2/Blue LLP scanner. The Faro Scanner is based on a calibrated coordinate measuring arm with a laser line sensor and has a nominal measuring accuracy below 25 lm. It is thus chosen as a baseline

Fig. 1 3D reconstruction of the robot base and PCB carrier system from images using SfM

ground truth for comparison with our scan. Directly using the Faro scanner itself is not feasible in most applications due to long setup and calibration times, reachability issues and very high initial hardware cost in comparison to a simple handheld camera as used by our approach.

The distance measurements, trimmed at 3 mm maximum distance and limited to the robot base and work piece carrier with PCB to account for differences in scanned portions of the system, is shown in Fig. 2. The distance measurement is done in CloudCompare [9], using quadratic function distance calculation (k = 6) to account for cloud resolution differences, see also Girardea-Montaut [8]. Most points fall well under an absolute distance of 1 mm (green) (median distance 0.63 mm). Especially feature-rich areas, such as the edges of PCB, carrier and robot base are very accurately captured. These distinct features are also the most suitable for fine-adjustment of CAD geometries. The same observation is made comparing the surface models.

Fig. 2 Cloud to cloud distances of the SfM scan to the FARO scan (reference)

#### 4.2 Performance of Compensation Strategies

As the functionality of our search and insertion strategies has already been shown in Metzner et al. [19], we focus on the validation of the scan-to-execution approach and our data-driven compensation strategies. The scan-based OLP methodology is validated by using the 3D scan to OLP a program implementing our methods. Both material supply and assembly position are derived from the scan and optimized via tactile strategies. The program is transferred to the real setup and executed without further adaption or calibration. As shown in Fig. 3b, the first execution run results in a visible offset (1) which is compensated by the tactile search (2). In the second run (3), this independent deviation is compensated to a degree that an insertion is possible without a search.

To validate our data-driven approach, we conduct a baseline analysis, repeatedly measuring the same assembly position of a mechanically fixed PCB by a four-point tactile approach in positive and negative x-y to quantify the robot's inherent measurement uncertainty. Then, we enhance a robot program for insertion of a THD into a circular hole by specified offsets in the perpendicular plane to the insertion vector. The reference goal location is determined as precisely as possible using the average hole center from the baseline test. The reached compensation values for each input parameter set, determined through a spiral-search approach, are saved. We then compare the resulting calculated offset values from our data-driven approach with the specified ones to evaluate the functionality of our compensation approach.

We recreate both independent as well as dependent deviations. For the dependent deviation, we also log the dependent parameter value set for our analysis. Asymmetries are deliberately added to the induced dependent deviations to simulate a biased parameter population in the initial compensation phase, which is likely in reality.

We implement three parameters, labeled manufacturer (M), carrier ID (C) and material feed (F). M and F each have two parameter values (A/B, 1/2), C has three (1/2/3). The magnitude of influence from each parameter, see Table 1, is derived from actual tolerance

Fig. 3 OLP of a precision assembly process; a OLP scene with integrated 3D scan; b Program execution; (1) First approach with noticeable deviation; (2) Final position after spiral search; (3) Optimized second approach; (4) Direct assembly without search

Table 1 Magnitude of parameter influence for test series

values of a similar application in electronics production. For each parameter in each test run, a random number from a normal distribution using the absolute deviation and its standard deviation s is drawn using the polar method [17]. For each M and F, we conduct 450 test runs, for each C 300, resulting in 900 test cycles plus 75 baseline test for both standard and optimized insertion.

In 1950 insertion tests, no failure is recorded. The distribution of specified and found deviations is shown in Fig. 4. We optimize the search points using the mean deviation calculated for each M, C and F parameter value as well as the asymmetry in global distribution (Dxmean = 0.14 mm; Dymean = 0.49 mm). The mean search time without optimization is 1.23 s, with 71% of insertions requiring a search. With our approach, on the same data points, no search is ever required.

Fig. 4 Insertion test series results colored by parameter combination. Negative specified position (•) and found position (+)

#### 5 Discussion

The findings on the point cloud and surface model accuracy show that our 3D scanning method is capable of reducing deviations of the virtual OLP scene to the real application into the submillimeter area. Even though further influences for deviations, such as the robot absolute accuracy, exist, a successful execution of a high-precision THD assembly process without touchups is demonstrated. We furthermore validate our data-driven approach for compensation of dependent deviations and find a total elimination of search times. We thus conclude that even deviation influences as low as 0.1 mm can be determined and permanently compensated after a reasonable data gathering period, even when an LWR is used.

#### 6 Conclusion and Outlook

In this contribution, we propose a methodology for direct OLP of high-precision assembly tasks through a combination of 3D scanning and tactile assembly routines including data-driven program adaption approaches based on defined deviation influences for continuous, automatic process improvement. We quantitatively evaluate the individual methods as well as the overall methodology. It is found that our approach is capable of robust OLP of processes with high accuracy requirements. Further research topics include the applicability on further use-cases and the validation on real-life industrial production data.

#### References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Industry 4.0

### Implementation of Innovative Manufacturing Technologies in Foundries for Large-Volume Components

Christian Klötzer, Martin-Christoph Wanner, Wilko Flügge and Lars Greitsch

#### Abstract

The development of new manufacturing technologies opens up new perspectives for the production of propellers (diameter < 5 m), especially since the use of the established sand casting process as a technology is only partially competitive in today's market. Therefore, different applications of generative manufacturing methods for the implementation into the production process were investigated. One approach is the mould production using additive manufacturing processes. Investigations showed that especially for large components with high wall thicknesses available systems and processes for sand casting mould production are cost-intensive and conditionally suitable. With our development of a large-format FDM printer, however, the direct production of large-format positive moulds for, for example, yacht propellers up to 4 m

W. Flügge e-mail<EMAIL_ADDRESS>
W. Flügge Chair of Production Technology, University of Rostock, Albert-Einstein-Straße 2, 18059 Rostock, Germany

L. Greitsch Mecklenburger Metallguss GmbH—MMG, Teterower Straße 1, 17192 Waren (Müritz), Germany e-mail<EMAIL_ADDRESS>
C. Klötzer (&) M.-C. Wanner W. Flügge

Fraunhofer Research Institution of Large Structures in Production Engineering IGP, Albert-Einstein-Straße 30, 18059 Rostock, Germany e-mail<EMAIL_ADDRESS>
M.-C. Wanner e-mail<EMAIL_ADDRESS>
in diameter is possible. Due to the comparatively low accuracy requirements for the mould, the focus is on the durability of the drive system and the rigidity of this FDM printer. Equipped with simple linear technology in portal design and cubic design of the frame structure with rigid heated print bed, the aim is to achieve maximum material extrusion via the print head. The production of plastic models not only facilitates handling during the moulding process, but also allows considerable time and cost savings to be made during the running process. A further step in our development is the direct production of the components using WAAM. A possible concept for robot-supported build-up welding for the production of new innovative propeller geometries is presented using the example of a hollow turbine blade for a tidal power plant.

#### Keywords

Additive manufacturing Propeller Fused deposition modeling Wire arc additive manufacturing

#### 1 Introduction

Additive manufacturing technologies (AM) have not only developed rapidly due to their ability to produce near-net-shape components with complex geometry, but also offer various advantages over conventional processes in the area of individual component production. In addition to the geometric and design freedom, production times, material consumption and, as a result, costs can be reduced enormously for small batch sizes. Thus, the development of new manufacturing technologies opens up new perspectives for the production of components of maritime systems [3].

Due to the complexity of the components, the sand casting process has been established especially for propellers and components of the propulsion train. Figure 1 shows the conventional production process of maritime components using the example of a fixed pitch propeller with focus on the casting process.

Based on the technical casting adaptation of the CAD model through additional machining allowance (consideration of shrinkage during the casting process), an external production of a wooden model takes place. The positive molds, each consisting of a wing and hub segment, have very long production and delivery times. Provided with an additional coating is the manual molding of the wooden model with cement molding compound separately for top and bottom. The repeated rotation of the model around the hub axis creates the shape of the ship's propeller. After each rotation, alignment of the model on the propeller rotation axis is required. The exact positioning and alignment of the wing models during molding requires a great deal of effort. Handling the solid wood models weighing more than several 100 kg is only possible using crane systems.

Fig. 1 Conventional production process using the example of a fixed pitch propeller

Afterwards, the mold must dry out depending on the surfaces present in the mould cavity. Casting and cooling are followed by measurement, on which the generation of the functional surfaces of the conical transverse press-fit and the finishing of propeller blade geometry by grinding to final dimensions are based. With a total duration of approx. 3 months, the production process is characterized by long service lives (drying and cooling times) and frequent machine changes.

Particularly in the production of small diameter propellers (D < 5 m), the established sand casting process can only partially be used as a competitive technology by European industry in today's market. In addition, the production of small propellers can be more complex than that of large propellers. Between the blades, there is a limited space for the mould construction and the later final contour machining. Therefore, the use of partially high-grade molding material and more complex molds is currently necessary. This leads to a further increase in manufacturing costs.

#### 2 State of the Art and Related Work

Additive manufacturing is in use along the entire value chain, starting with the production of sample parts (rapid prototyping) and the manufacture of tools (rapid tooling) through to the production of fully functional components (rapid manufacturing). With the focus on a traditional application area, the shipbuilding industry, where most actions are based on experience and tradition, selected studies and applications are presented below. Taşdemir and Nohut [8] lists a variety of applications and Ludwig et al. [4] presents the need for AM processes and steps for implementation in the maritime industry.

Investigations of additively manufactured molding sand models of a propeller cap show the disadvantages for a cost-effective and timesaving alternative to traditional model making, especially for large-format castings such as propeller molds with wall thicknesses of up to 500 mm. To comply with strict environmental and worker protection regulations, inorganic binder systems such as water-based alkali silicate binders are increasingly in use. Inorganic binder systems require additional post-curing at high temperatures. The preferred microwave drying is possible for sand molds with a maximum wall thickness of approx. 40 mm. This leads in particular to a low storability of the moulds, a strong tendency to cracking and brittleness of the moulding material. Additional process fluctuations in the additive production of large, thick-walled molds and its post-treatment increased the scrap rate.

Another possibility is positive mold production from thermoplastics by fused deposition modeling. For example, patterns made of polylactide (PLA) and acrylonitrile butadiene styrene (ABS) for mold making serve for the production of miniature propellers or rudder segments [6]. The production of samples of newly developed customer-specific components within 24 h significantly shortens the overall product development time and ensures the fastest possible product changeover. This process is currently only used to produce sample components for mold making or to validate individual product properties of small series and is limited to small component dimensions due to the available systems.

For final products, the suitability of buildup welding for additive manufacturing of nickel-aluminum-bronze components is significant in the maritime sector. In Ding et al. [2], the authors investigated buildup welding of thin-walled structures with a focus on material behavior. In another joint research project [7] the optimization of the Wire-Arc-Additive-Manufacturing (WAAM) process for the production of large-format components for maritime applications was investigated. In Damen Shipyards Gorinchem [1], an additively manufactured ship's propeller was produced in 298 layers and is a welded solid propeller with subsequent complex finishing by manual grinding. Another research project [5] shows a demonstrator for hollow stainless steel propeller blades weighing 300 kg. An evaluation of the demonstrator in terms of fatigue, corrosion, residual stresses, material properties, and geometry shape accuracy is not available in Damen Shipyards Gorinchem [1] and NAVAL Group [5].

#### 3 3D-Printed Mouldings for Sand Casting

A further approach for the integration of generative manufacturing processes into the existing production process is direct positive mould manufacturing by means of Fused Deposition Modeling (FDM). The first step is to develop an FDM printer for the additive manufacturing of large-format components. Subsequently, the existing production process must be adapted to implement the new manufacturing option. Finally, validation of the process and the printer is required using selected components.

#### 3.1 Development of a Large Format FDM-Printing System

Based on the positively evaluated results from preliminary investigations of scaled propeller blades, Fraunhofer IGP developed a large-volume 3D printer with a working space of eight m<sup>3</sup> in cooperation with Mecklenburger Metallguss GmbH, shown in Fig. 2. The FDM printer additively manufactures ship propeller models with a diameter of up to four meters in addition to propeller hoods and special designs.

The casting production process achieves accuracies of between 100 and 1000 µm in the manufacture of ship propellers. In comparison, the achievable accuracies for milling are 10–100 µm and for sintering even 2–10 µm. The comparatively low accuracy requirements justify the elimination of cost-intensive precision linear technology in the development of the large-format FDM printer. The focus is on the durability of the drive system as well as the rigidity and dimensional accuracy of the entire system. Accuracy requirements limit standard linear guides in terms of maximum length. Due to the long travel of the print head within one axis, the deflection of the linear guides is a major challenge. By means of FEM simulations and tolerance analyses of the individual components and its adjacent construction, the permissible deformations and natural frequencies of the overall system were optimized. The frame structure of the printer is cube-shaped and is equipped with highly rigid aluminum profiles to withstand the stress of sudden changes in extruder direction. Additional elements in the corner points of the frame structure support the rigidity of the overall system. In the center of the system, there is a non-moving printing bed. Individual setscrews align the four precision-milled aluminum plates on the frame substructure to within tenths of a millimeter. Silicone heating mats installed on the underside heat the printing bed to a maximum of 110 °C. An external control for the heating elements monitors the even heating of the printing bed. The X and Y-axes in the Cartesian gantry system move the filament extruder above the print bed. Due to the very rigid basic structure and the desired parallelism of the axes, it is possible to operate the X and Y-axes with only one motor each. Closed belts transmit forces and motion from the stepper motors to the roller carriages on rails. Despite the

Fig. 2 Development of a large format 3D printer

travel distances of 2000 mm in X and Y direction, the deviations of the nominal to the actual contour of the component to be printed are only a few tenths of a millimeter. The movement of the X/Y portal and the extruder in the build-up direction is implemented by ball screws. Stepper motors control four ball screws on two sides of the frame structure. Additional guide rails in the frame structure support the dimensionally accurate movement of the extruder. Due to the minimal movement in the direction of assembly, motors are additionally equipped with holding brakes. Filament spools wound on the top frame ensure easy unwinding and insertion into the extruder. The system is equipped with a typhoon® filament extruder for the processing of thermoplastic material. The heating power of 400 watts and maximum printing temperature of 500 °C implement maximum material extrusion of 200 mm<sup>3</sup> /s (0.9 kg/h), one of the highest flow rates for 2.85 mm filament. Usable with different die sizes up to max. 2.5 mm, the extruder can melt many times more material than standard 3D printers can with standard 0.4–1.2 mm dies. This is an important basic requirement for the additive production of large-format components. In order to determine the optimum process parameters, a defined test specimen was additively manufactured from different geometric elements (cuboid, sphere, inclined walls). In the statistical experimental design, the printing speed, layer height, printing temperature and infill are the factors to be investigated. The final visual inspection and verification of quality characteristics (dimensions, defects, overhangs) of the test specimens resulted in the following parameters in Table 1.

In order to check the positioning accuracy of the extruder and thus the relative dimensional accuracy of the printed object, the comparison of nominal and actual position is one way to evaluate. A large selection (n = 50) of randomly determined positions in the workspace of the 3D printer are used to check pose and repeat accuracy. A laser tracker Leica AT960LR measures the current position of the extruder and compares it to the specified mapped position in three-dimensional space. The evaluations result in an accuracy of ±1.5 mm. Causes for the deviations can be step angle errors and deviations of the step errors of the motors due to friction of the running rails or due to the belt drives. With regard to the component and process tolerances, this accuracy is sufficient.

#### 3.2 Implementation in the Production Process

A typical application for the additive production of a positive casting model at MMG is the MMG-escap®. This is a propeller cap to protect the shaft lock nut from seawater behind the propeller. The fin design, specifically adapted to the propeller, untwists the hub angle and reduces torque loss. The optimized flow behavior reduces cavitation at the rudder and hull, reduces wear and tear and increases both overall efficiency and operating time.

Figure 3 shows the adapted production process for a hybrid mould production using the example of a selected propeller cap. Fixed basic bodies exist for certain cap sizes, but for each propeller there are fins with different geometries. In the combined model setup, the basic bodies and cores are made of wood and the fin inserts of plastic. Starting with the CAD data from the propeller design, the geometries are adapted to production requirements in the pre-process. On the one hand, casting technology adjustments are necessary, mostly an additional machining allowance to take into account shrinkage during the casting process. On the other hand, transport holes must be defined and lifters must be specified for handling the mould. The user defines essential parameters for the fin bodies in the FDM process while the external pattern maker manufactures the base body. The Simplify3D software simulates the printing process for each component and generates the machine-specific code to start the fully automatic printing process. Once all printed parts are finished, the post process starts. In the first step, the operator removes any support structures, deburrs the component edges and fills any defects with filler. Alternating sanding and coating with a 2-component coating creates a smooth over surface. Prior to assembly, the employee assembles lifters and drawing belts. A final optical measurement of the model or later castings verifies compliance with the tolerance specifications.

#### 3.3 Validation and Results

The model production of fixed pitch propellers and propeller caps primarily uses the process. Process recordings, calculations and cost estimates document important parameters in the production process. Based on the collected production data, a comparison between conventional model making and mold making with additively manufactured components is possible.

The Fig. 4 shows the detailed costs and expenses for fin inserts of a propeller cap. Quotation prices and delivery times represent conventional model production by an external model maker. Printing time and post-processing processes are the main times in additive manufacturing. Additional non-productive times, e.g. for loading and unloading the jigs and fixtures, filament changes, transport times are included in the time recordings. The costs for additive model production include material and manufacturing costs. The manufacturing costs include energy costs, room costs, imputed depreciation and interest as well as production wages.

In addition to cost reduction (from 1500 € to 1120 €) and production time reduction (67%), the focus is on weight reduction of up to 60%. Plastic models of 20–25 kg are much easier to handle than comparable wooden models with a weight of approx. 60 kg. Above all, the assembly of the propeller canopy is more user-friendly. The data refer to the production of one fin insert. Since the propeller caps have several fin inserts, the cost, manufacturing time and weight savings for the entire model are many times greater. Thin-walled plastic models with optimized infill for fixed-pitch propellers reduce not only costs but also above all weight compared with solid wood models. The weight reduction makes it easier to handle the models during the moulding process.

The GOM Atos Triple Scan fringe light projection system then measures the components, see Fig. 5 the measurement of propeller cap PH00459 after casting. With a measuring field of 2000 2000 mm<sup>2</sup> , the measuring accuracy is a few tenths of a

Fig. 4 Comparison of costs and process times between conventional and additive manufacturing for fin inserts of the propeller cap PH00459

millimeter. The best-fit process uses a CAD model to virtually align the resulting model in a defined configuration. Especially in the area of the fins, good dimensional accuracy and small deviations from the target geometry are given. Allowances of up to max. 5 mm allow an economical mechanical finishing of the freeform surfaces.

#### 4 Robot-Supported Machining for the Additive Manufacturing of Maritime Components with Hollow Structure

In order to continue the path towards an effective manufacturing of small series of smaller propellers the direct component production using additive manufacturing processes has been investigated as an alternative to the casting process. Due to the relatively high deposition rate, small installation space restrictions as well as low investment and operating costs, arc-welding processes, especially WAAM, are predestined for the additive manufacturing of large structures. However, the main challenges are the large component dimensions in conjunction with the high accuracy requirements and the guarantee of freedom from defects as well as homogeneous mechanical-technological properties over the entire component. Figure 6 shows an experimental setup for conducting preliminary tests. An additional rotating and positioning device extends the working range of the 6-axis jointed-arm robot. The turning device can fix prefabricated cast work pieces so that buildup welding onto the intended base material is possible. The system also includes Fronius welding equipment, which is used in the standard arc process to build up the component layer by layer. The welding torch is located on the TCP of the robot, which connects wire feed hoses and connection hose packages with the TPSi 600 welding power source and the wire feed unit.

After the successful execution of initial joining tests and the production of test specimens for subsequent material tests, a model of a tidal power plant turbine blade was produced by build-up welding as a hollow blade. Current production possibilities for

Propeller cap PH00459

Fig. 5 Measurement results to illustrate the deviations of the casting surface from the planned one

Fig. 6 Left: test setup for the build-up welding, right: welded demonstrator blade with mechanical finishing

marine tidal turbines and hollow propellers are limited to casting processes or the construction of fiber composites. The use of heavy and stiff turbine blades in cast design requires high flow velocities and reduces the number of possible application locations. The use of fiber-reinforced plastics such as carbon fiber reinforced polymer (CFRP) has the disadvantage that they are not resistant to cavitation and are not yet fully recyclable. To reduce these disadvantages, the additive manufactured demonstrator is made of a copper–aluminum alloy, as good corrosion resistance to seawater, good weldability and high mechanical parameters characterizes it. The wing is part of a 3-blade turbine with a diameter of 4 m. With the aid of topology optimization and consideration of the manufacturing restrictions, the hollow demonstrator is 50% lighter than a solid casting (150 kg) with the same external shape.

The following investigations focus on hybrid machining strategies. A tool changing system on the robot provides the appropriate tools for the alternating additive and subtractive manufacturing steps. For hollow structures in particular, it is important to machine the inner surfaces in order to ensure that the components are also free of internal notches. Additional peripherals such as a line-cut sensor or arc camera monitor the production process. Together with the documentation of the welding parameters, the system detects weld seam irregularities. Continuous temperature measurements monitor the interpass temperature of the buildup welds. Due to the high thermal loads resulting from the layer-by-layer buildup, optimized welding strategies are essential for distortion-optimized path planning.

#### 5 Conclusion and Future Work

The investigations and developments made so far show the possibilities of introducing additive manufacturing methods in the casting process. Especially the application of complex geometries of ship propellers and energy saving caps with their fin systems show a high potential for reducing process costs as well as processing time. Beside these quantitative advantages, the resulting weight reduction in case of 3D-printed moulding patterns gives further benefit in terms of assembly and handling these components.

The use of direct printing methods for sand moulding does so far not satisfy the requirements for large-scale moulds. Here further work must be established in enhancing the process towards increased wall thickness and moulding size. Furthermore, the cost side of this possible process needs reduction measures to ensure an applicability under commercial considerations.

However, the direct metal printing approach based on the WAAM process appears more promising especially to be applied for medium size propellers and similar components. The main advantage here is in the complete skipping of any model and mould making process steps. Based on this omission of related costs an overall cost benefit can be gained. A clear antagonist for this possible overall cost reduction is the cost factor for the material. The difference between raw material costs as used for the casting process and the wire costs for the WAAM process strongly triggers the economical maximum size of components. Here further works is needed to push the limits towards larger component sizes as e.g. the development of manufacturing hollow components. Beside the positive influence on the material usage, additional functionality can be realised with benefit for the product.

#### References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### The Digital Twin as a Mediator for the Digitalization and Conservation of Expert Knowledge

Dominik Hüsener, Michael Schluse, Dorit Kaufmann and Jürgen Roßmann

#### Abstract

A Digital Twin is a virtual representation of a physical asset. It reflects the current state of that machine through a model and the data as observed by sensors in the real machine; and enables effective and efficient interaction with the machine, i.e. for monitoring and control purposes. The Digital Twin facilitates the collection of data, as well as its analysis and visualization through its user interfaces, i.e. GUIs such as screens or Mixed Reality that provide intuitive access to the data and facilitates its manipulation. Embedded in Virtual Testbeds the Digital Twin becomes an "Experimentable Digital Twin" (EDT), in which experiments can be performed and the different outcomes can be compared or evaluated. The intuitive representation of the assets allows the experts to interact with the twin, without highly detailed knowledge in computer science. The digital twin observes, records, and benchmarks experiments performed by the operator. This way the operator's knowledge becomes digitized and thus preserved as an abstract representation of data, formulas, and models inside the digital twin. By introducing the Digital Twin into the processes carried out by different operators (not only the initially observed expert), formerly intuitive decision-making processes of the operators are enhanced based on empirical data. As a result, the Digital Twin serves as an assistance system that can guide future operators and the outcomes

D. Hüsener (&) M. Schluse D. Kaufmann J. Roßmann

Institute for Man-Machine Interaction, RWTH Aachen University, Ahornstr. 55, 52074 Aachen, Germany

e-mail<EMAIL_ADDRESS>
T. Schüppstuhl et al. (eds.), Annals of Scientific Society for Assembly, Handling and Industrial Robotics 2021, https://doi.org/10.1007/978-3-030-74032-0\_20

of the experiments become reproducible. The specific representations of interactions and outcomes also facilitate collaboration between the machine operators and other stakeholders by providing different operators a common "perspective".

#### Keywords

Digital twin Decision making Knowledge digitalization

#### 1 Introduction

With the advance of Industry 4.0, there is a shift from mass production to innovative, specialized products with small batch sizes. Therefore, machine parameters need to be changed more frequently. In a traditional production setting, this leads to higher personnel, time, and material requirements and consequently higher production costs [1]. In some cases, the new settings need to be tested before the actual production can begin further increasing time and cost requirements. Even if the test pieces are within tolerances, the final product can be faulty. On older machines, which are still in use in some industry sectors, machine parameters need to be adjusted manually for each product, thus limiting the possibility of batch size 1 and driving costs up for small batch sizes.

Especially in some disciplines belonging to the traditional mechanical engineering sector (as textile mechanical engineering), the success of the outcome is depending on the skills of the operator and his experience in setting special machine parameters. Thus, effective production is only possible with experienced, skilled personnel. This knowledge is not measurable and thus not transformable. This problem is enhanced by a lack of young employees, trainees, and skilled workers, and the changes in demographics [10].

The knowledge however is not conserved but is in the heads of the employees and depends on word of mouth to be passed along. This is because knowledge is empirical and not quantized. When an employee leaves the company, the knowledge is gone. Due to the limited use of digitalization in some industry sectors, the training takes long, and even skilled workers need time to learn to use an individual machine independently.

The rest of the paper is organized as follows. In Sect. 2 the Digital Twin concept is presented with the integration of simulation in a so-called Experimentable Digital Twin and mentioning its role as a mediator. In Sect. 3, a brief overview of knowledge management is given and combined with the Digital Twin approach as a means of knowledge management. In Sect. 4, this concept is transferred to a use case of the textile industry.

#### 2 What is a Digital Twin?

The Digital Twin (DT) was first presented by Grieves [3] as a virtual model of a physical asset that receives data from the asset and virtually represents the state of the asset. In 2003 the "Digital Twin" concept was first presented. In 2014 Grieves published a white paper on the Digital Twin. A Digital Twin is a virtual representation of an asset that is fed with data from the real machine and also sends data to the real twin. The DT can be obtained by combining the data of the machine (the Digital Shadow) with a model of the machine (see Fig. 1).

The combination of the physical asset and the Digital Twin with its corresponding communication infrastructure is called a Cyber-Physical System (CPS). The physical asset can itself be a CPS. In Industry 4.0 CPS become increasingly important. Different machines have their own twin, while the twins can communicate with each other. The Digital Twin is used in health monitoring, also production planning (PLM), and the design of new products. In comparison to conventional planning software, what is new is that all data related to one product is stored together in one place (the Digital Twin) whereas before machines were able to store data but it was not combined into one unified presentation. This presentation however has brought a massive increase in value. Although it is only a simplified representation, it tries to model the behavior of the real twin accurately.

In Kritzinger et al. [5] literature on the Digital Twin in the manufacturing field is reviewed and classified as either Digital Model, Digital Shadow, or Digital Twin.

Fig. 1 The context of the Experimentable Digital Twin (EDT) and the Cyber-Physical System (CPS) [7, 8]

According to their definition, the difference with these terms is the communication between the digital and physical assets. Whereas the Digital Model is independent of the real asset, the Digital Shadow only receives data from the real asset and only the Digital Twin allows bi-directional communication between digital and physical assets. While the term "Digital Twin" is widely used in literature, it is often used for Digital Models and Shadows as well. Therefore, the need for a common definition of the Digital Twin is emphasized. The literature on actual Digital Twins according to their proposed definition however is scarce. Especially there is a lack of case studies on a higher level of integration.

A more recent literature review was performed in Tao et al. [11] with a focus on applications of the Digital Twin in industry. After the white paper of Grieves, the number of papers published is increasing steadily. It is also mentioned, that the definition of the Digital Twin is varying in different papers. The most relevant theoretical foundation of the DT is listed as DT modeling, simulation, verification, validation, and accreditation (VV&A); data fusion; interaction and collaboration; and service. The main applications currently where the DT is used are production and health management.

#### 2.1 What is an Experimentable Digital Twin?

The experimentable Digital Twin brings together modern simulation technologies with the Digital Twin concept [9]. Whereas before several simulations (FEM, fluid dynamics, multi-body dynamics…) were performed individually, this can lead to wrong results if there is no exchange between the results. The Digital Twin enabled by co-simulation brings together different simulation techniques and stores the relevant data in the Digital Twin such that is exchanged between the various simulations (see Fig. 2).

With simulation, it is not only possible that a Digital Twin supervises a system and has a virtual representation but also a view into the future becomes feasible. The Digital Twin is not only able to collect the machine data but also to generate new data and through this improve the functionality of the physical asset. In manual production processes, it is difficult to use artificial intelligence to optimize the process since data of the machine and the produced object are not stored together. The Digital Twin however also opens the door to artificial intelligence techniques that can optimize the behavior.

#### 2.2 Digital Twin as a Mediator

The Digital Twin displays its current status and can suggest actions to be taken through the human–machine interface. The machine sends its data which is processed by the Digital Twin to display valuable information to the user in an understandable way. The operator sees the data, can understand its meaning and can act accordingly (see Fig. 3).

Fig. 2 "Experimentable Digital Twin approach". The EDT combines several simulation algorithms [9]

Fig. 3 Schematic concept of the digital twin as a mediator between the operator and the real twin

In his whitepaper Grieves [3] describes, how the virtual representation is beneficial to conceptualize, compare and collaborate in production processes since they provide a more intuitive perspective compared to 2D sketches or data in tables or regular graphs. The realistic visual view is valuable since sight is the most important human sense. Different users can see a representation that easily allows them to classify the current state of the machine which is also consistent for different users [4].

The user can simulate the behavior of the machine and is supported to find suitable settings for a given task. The simulation can be displayed in 3D and information that is otherwise not visible to the operator can be visualized through the Digital Twin. Hence, the Digital Twin can also be regarded as a mediator between the machine operator and the machine itself. In Cichon and Rossmann [2] a concept to facilitate the interaction of humans and machines is presented. Among others, joysticks, screens and AR/VR allow direct interaction with the digital twin. The user is able to interact with the virtual machine similar to the real machine, and the (real) machine informs the user of its current status through the visualization of the Digital Twin.

#### 3 What is Knowledge Management?

Knowledge management is about conserving the (procedural) knowledge that is needed to perform a certain task. As mentioned earlier, the aging workforce requires that their knowledge is somehow stored in order not to lose the knowledge which is an essential asset for the company to remain productive. This is traditionally performed manually through observational studies and questionnaires [6]. Questionnaires however require the person conducting the study to have some basic knowledge of the process to be considered in order to ask the relevant questions. Otherwise, some necessary or useful information might be left out. Also, questionnaires require additional work to provide a benefit. They are only useful if the results are presented in a manner such that they are comprehensible to trainees. A textual description for example can be useful for experienced workers that need to learn how to operate a new machine but might be difficult to grasp for students that are not yet familiar with the functioning of the machine.

Observational studies are better in this way; however, it becomes costly very quickly if there is a high amount of different work tasks to do and infeasible to record tasks that only occur seldomly. Also, this requires a setup and is not simply integrated into the regular working process.

In Mohammad and Al Saiyd [6] a process for manually acquiring expert data in the context of artificial intelligence is presented. The required activities are the identification of the domain knowledge, finding experts that can provide that knowledge, finding a representation for modeling of that knowledge, and the construction of a knowledge base that can then be used in an inference engine (see Fig. 4).

#### 3.1 How to Conserve Knowledge with a Digital Twin

If the knowledge of the workforce is to be conserved, it can be digitized in some way. Digitization has many advantages and can increase the value of the captured information when it is combined with other information. Considering the activities in Mohammad and Al Saiyd [6], we propose to use the Digital Twin as a means of knowledge representation and to store the knowledge base. The Digital Twin also receives real-time data from the machine—as a result, there is one central data storage for all information related to that machine ("Single Source of Truth"). The Digital Twin can store data like a conventional database, but it also has the advantage of being able to visualize that data more intuitively. A further advantage of the Digital Twin is that at the end of the process a virtual model is generated that can be verified by the experts. Also, the virtual representation can help trainees as processes can be shown on a 3D model that—other than a video—can also be manipulated and is easier to grasp than a textual description. Through the use of augmented reality, it could even be possible to see a virtual representation of the machine along with the real machine that can guide the worker through the process of adjusting the machine parameters or maintenance with the advantage of having their hands free.

However, the data of the interaction is stored by the machine and transferred into the database of the Digital Twin, it becomes feasible to record processes on the go and understandably conserve them. Also, not very frequent tasks are conserved this way. The digital twin itself is subject to an ageing process, hence it needs to adapt to new production environments in order to sustain its relevance.

#### 4 Use Case

The project 'Development of an experimentable digital twin for the analysis and automated adaptation of textile manufacturing processes using the example of tufting technology (T-EXDIZ)' aims to show in the example setting of a textile machine how the Digital Twin can bring benefit to manual tasks that are difficult to automize. The machines are usually kept for decades before they are replaced, and older machines cannot be automized easily. It is however possible to include sensors into the machine, such that various parameters (e.g. angle, length) of machine parts or settings can be read. To improve automation, the Digital Twin can record these settings and thus create a virtual representation of the process. The operator can experiment with the Digital Twin, allowing him to experience the consequences of his activities. These steps can be visualized on a screen through a virtual representation of the Digital Twin (i.e. rendering) which is usually a simplified representation of the real twin. Hence, through utilization, the settings are kept for further use. It is then possible to test these new settings using the simulation methodology without having to waste material. If this product has to be produced again the exact values are stored and can be compared to the current values such that reproducibility is conserved. Besides, this also makes it possible to teach the process to new trainees. The Digital Twin can also simulate its behavior to provide training opportunities. Also, there is an opportunity to test new configurations without the costly procedure of testing each setting since the simulation can perform this testing in a fracture of the time. It can thus help trainees gain confidence and reduce the amount of training needed to operate the machine.

If a worker leaves the company, his procedural knowledge can be stored in an understandable form such that it is available to new trainees. Also, for experienced workers, it becomes simpler to check whether they have completed steps correctly. The human–machine interface (HMI)—a window in the virtual world where the digital twin lives—can help to visualize what steps must be taken. The spread by word-and-mouth also relies on a common understanding, the Digital Twin however has one unified representation for everyone. In training, it will be more easily possible to test multiple configurations.

This procedure is shown in the use case of a textile machine. In the textile industry automation is low and batch size one production is often not feasible. The computer has a model of the machine that can be used to simulate the machine. It receives data from the operation of the machine. The real data can be used to update simulation parameters and it

can also be used to detect irregularities and advise the worker to repair machine parts or to adjust settings. Instead of waiting for the finished product and noticing that it does not fulfill its specifications, the Digital Twin can simulate further behavior based on the current state. Thus, faults can be detected early and waste products, and thus cost is decreased.

The flow of data and information between the different components is shown in Fig. 5.

Currently, a DT model of the kinematics of a tufting machine is created. In the next step, this model can be refined with the input of experts. Later an automated mode will be integrated that can optimize machine parameters through simulation transforming the DT to an experimentable Digital Twin.

#### 5 Conclusion

Due to an aging workforce, some personnel is leaving the production sites in the coming years. However, the knowledge of the workers is an important asset for the company to remain competitive, especially in sectors where manual labor is still widely present.

The Digital Twin is a virtual model of an asset that can be used to store and visualize machine data. As was shown in this paper, the Digital Twin can also be used to conserve the knowledge of the workforce and make it measurable in the first place. This helps in reducing training times for new employees and knowledge loss. The knowledge that comes from questionnaires, as well as observations, cannot only increase the value of the Digital Twin itself but the DT is also a useful tool to have a meaningful representation of that knowledge.

Acknowledgements The writing of this chapter was enabled within the context of the IGF project 'Development of an experimentable digital twin for the analysis and automated adaptation of textile manufacturing processes using the example of tufting technology (T-EXDIZ)' 21166 N/2 of the Forschungsvereinigung Forschungskuratorium Textil e.V.. It is funded via the AiF within the framework of the program for the promotion of joint industrial research and development (IGF) by the Federal Ministry of Economics and Energy based on a resolution of the German Bundestag.

#### References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### Web Service for Point Cloud Supported Robot Programming Using Machine Learning

#### Kristina Enes

#### Abstract

In industrial automation, the use of robots is already standard. But there is still a lot of room for further automation. One such place where improvements can be made is in the adjustment of a production system to new and unknown products. Currently, this task includes the reprogramming of the robot and a readjustment of the image processing algorithms if sensors are involved. This takes time, effort, and a specialist, something especially small and middle-sized companies shy away from. We propose to represent a physical production line with a digital twin, using the simulated production system to generate labeled data to be used for training in a deep learning component. An artificial neural network will be trained to both recognize and localize the observed products. This allows the production line to handle both known and unknown products more flexible. The deep learning component itself is located in a cloud and can be accessed through a web service, allowing any member of the staff to initiate the training, regardless of their programming skills. In summary, our approach addresses not only further automation in manufacturing but also the use of synthesized data for deep learning.

#### Keywords

Machine learning Digital twin Structured-light 3D scanner

e-mail<EMAIL_ADDRESS>
https://doi.org/10.1007/978-3-030-74032-0\_21

K. Enes (&)

RIF Institut Für Forschung Und Transfer E.V, Joseph-von-Fraunhofer Str. 20, 44227 Dortmund, Germany

T. Schüppstuhl et al. (eds.), Annals of Scientific Society for Assembly, Handling and Industrial Robotics 2021,

#### 1 Introduction

This paper addresses three issues regarding machine learning in the industrial field.

Firstly, many problems can arise when applying machine learning in a production process. Artificial neural networks need a large amount of data to train on and while there is a lot available containing day to day situations, like indoor and outdoor scenes, this is not the case for industrial environments. Suitable training data need to be collected and labeled manually, which can be difficult and tedious. Another problem is that the environment can influence the quality of the data, like distortions due to vibrations or due to changes in temperature.

Secondly, while the production of large lot sizes is mostly automated, the demand for small lot sizes increases, as well as the variety of the products. Also, the number of times the production process needs to be adjusted to new products increases, too. Since both the reprogramming of the robots, that are included in the process, as well as the image recognition algorithms, are tailored to the environment and the products, the incorporation of a new product to the production process is costly. An expert with programming skills is required. But such experts are not always available.

And thirdly, the acquisition of suitable training data for a specific task. Though there are many datasets with large amounts of training data they do not cover every situation. There are many environments in which training data is nearly impossible to obtain, e.g. the deep sea or space. It is also hard to gather data on situations that rarely occur naturally. Creating your own dataset of real-world data will take time and effort, since besides gathering the data, they need to be labeled manually, too.

Our approach relates to all three challenges. We propose creating a digital twin mirroring a physical production line in the industry to generate synthesized training data to train an artificial neural network, accessible through a web service, to learn to recognize and locate products given observed point clouds of the scene.

In a simulation, we can generate an unlimited amount of training data and the labeling of the data is automated. Also, a simulation can easily show extreme situations that rarely occur. With this we can generate a wide range of training data, giving a possible solution to the problem of insufficient real-world training data.

If an artificial neural network can successfully be trained with synthetic data with or without a smaller amount of real-world data to recognize and locate products, the flexibility of the production process increases, and the costs decrease. A new product can easily be incorporated by every member of the staff without the need of any programming skills and without the need to consult a specialist. With this, producing small lot sizes of a large variety will become more efficient.

Our solution is also one step in the direction of introducing machine learning into the industry, showing that the issues with insufficient data and error-prone sensor data can be approached by using simulations.

The following parts of this paper are structured as follows: in Sect. 2 we describe some previous work on the above topics. In Sect. 3 we introduce some works which will be used as a basis for our solution, VEROSIM, and Rüstflex. The architecture and components of our system are described in Sect. 4. We will discuss further plans for our project in Sect. 5.

#### 2 Related Work

Rossano et al. [12] describe how specialized knowledge is needed to program industrial robots. They give an overview of approaches to help with the program structure and the creation of new motion paths. The solutions suggested are either graphic-based, CADbased, or include manually moving the robot arm. But mostly those solutions have drawbacks. A current approach used by Drag & Bot [5] or ArtiMinds [3] includes the use of function blocks in the form of graphic program modules. In IntellAct [16] a robot learns by observing a human manually demonstrating certain tasks.

Sahbani et al. [15] describe two different basic research approaches for a robot to grip unknown objects. Firstly, the analytical approach, which is based on a mathematical model. Secondly, the empirical approach, which includes the imitation of human motions or planning a motion based on observations of an object. Bohg et al. [4] give an overview of the data-based planning of gripping an object. However, the approach of using deep learning in this context is relatively new.

Digital twins are researched since approximately 2010. Negri et al. [9] give a survey of current research regarding digital twins. They define digital twins, in the context of production systems, as a virtual representation of a production process that can be used for different types of simulations. Rossmann and Schluse [13] define experimental digital twins, which can be combined into complex simulation models, namely virtual testbeds. They represent all important parts of an application close to reality and can be used for interactive experiments.

Georgakis et al. [7] synthesize existing real-world training data of indoor scenes by superimposing images. Textured object models are placed into different background scenes at different locations and with different sizes. This is done by either image blending or by using depth and semantic information to make smart positioning. An object detector was trained using these images in addition to an existing dataset with good results. This approach allows the expansion of existing datasets, but in case of applications with no suitable dataset, a different solution is needed.

Tsai et al. [17] create a large set of computer-generated 3D hand images to train a convolutional neural network to identify different hand gestures. They discovered that adding about 0.09% of real-world images to the training process increases the accuracy from 37.5 to 77.08%. Lindgren et al. [8] generate a dataset of synthetic hand gestures by using modern gaming engines. The synthetic hand gestures are created by making variations to the kinematics. They train a classifier purely on those generated data. The results are accurate and can be used on real-world data.

Richter et al. [11] use modern computer games to generate labeled training data for semantic segmentation tasks. They add different amounts of synthetic data to two different semantic segmentation datasets and compare the results of the trained networks. They have shown that by including synthetic data to real-world data the performance of the network increases, reducing the amount of hand-labeled data needed to train a successful neural network. But in this case, the generated data is completely dependent on the game. Using this approach for a specific application might not be possible, since the influence, the user has on the resulting training data, is restricted and there might not be a game suitable for your application.

#### 3 Groundwork

#### 3.1 Rüstflex

Rüstflex [14] is a web application created by Vathos GmbH [18] which can efficiently retool industry robots. It can adjust a robot's movements, mostly those that can be executed without sensory aid. The application runs either in a cloud or in a local computer center and can be accessed through all mobile end devices. It provides a formula where all relevant information for the setup of an article is stored. We will use Rüstflex in our project for the parametrization of the demonstrator (see Sect. 4.5) to reprogram the robot.

#### 3.2 VEROSIM

With our simulation framework VEROSIM [19] we can create digital twins and virtual testbeds. A virtual testbed provides us with a completely virtual environment that can be used for experimentation. It can be integrated into real systems and can provide intelligent sensors, actors, and robots. We have already created a wide range of applications ranging from those in the industry to natural and urban environments as well as space. Three example applications are shown in Fig. 1. Our framework also provides several sensor simulations like ToF cameras, laser scanners, or radars, which can be built upon. One of the goals of this project is to upgrade our sensor simulations.

We will use VEROSIM in our research project to create digital twins of a production line and all its components, as well as to generate synthetic and automatically labeled data to be used as training data for an artificial neural network.

Fig. 1 Digital twins and virtual testbeds in industry, forestry, and space

#### 4 System Components

Our goal is to enable a robot to handle unknown objects using machine learning methods trained mainly on synthetic data. Our overall system consists of several parts. First, there is a physical production line and its digital twin. A digital twin simulates all parts of its real-world counterpart. It can consist of several other digital twins and simulates the communication streams between different components. A deep learning component located in a cloud can be accessed by the physical production system and the digital twin as well as a user through a web service. All these components will be combined in a demonstrator. The overall architecture of our system is shown in Fig. 2. Further descriptions of each component are given in the following chapters.

Fig. 2 System architecture

#### 4.1 Production Line

We view a physical production process as the basis of our system. This process consists of a pick and place robot, a sensor, a gripper, a robot controller, an edge controller, and products of the production line. The robot controller dictates the robot's movements, opens, and closes the gripper, and triggers the sensor. The edge controller contains a copy of an artificial neural network trained in a cloud. The deep learning component is further described in Sect. 4.3.

The edge controller receives the data generated by the sensor and gives them as input to the copy of the neural network. In this case the edge controller returns the positions of the products to the robot controller.

The specific production line we chose is situated at aha! Albert Haag GmbH [1]. It is a deep drawing process where products are redrawn one or several times. Between each redrawing a robot from type Fanuc [6] takes and places the processed product from a machine to a palette with a piece of cardboard between each layer of products. Then a new product is taken and placed from a palette to the machine to be redrawn again.

There are many different variations of products that can be processed in this production line. They differ in size, material, geometry, and the number of required deep drawings. Also, each product is oiled. The sensor we use should be able to make accurate recordings of these products independent of their material or shininess. The observed data is than used to recognize and locate the products in the topmost layer of the current palette. The sensor we choose is a structured-light 3D scanner. It can produce both images and point clouds. The resulting point clouds are forwarded to the edge controller.

#### 4.2 Digital Twin

Given the physical production line we build its digital twin using our simulation framework VEROSIM. But to what extend does a digital twin resemble its counterpart. Here are some, but not all, examples of what exactly is contained in a digital twin and what it can do:

We create a digital twin of all components of the production process while keeping the above points and more in mind. For the robot we define the kinematics and inverse kinematics. The robot controller can move the robot by following a predefined path and the sensor generates synthesized point clouds of the observed scene. The difficulty regarding the simulation of the sensor are the sensor's internal and external errors that occur during the recording. Our sensor simulation framework should be able to simulate

Fig. 3 Production process from aha! Albert Haag GmbH [1] and its digital twin

those errors, too, since our goal is to generate a simulation that is as close to reality as possible. But since the simulation contains all important information it is easy to automatically label all generated data.

In Fig. 3 you can see a picture of the physical production line on the left and its digital twin on the right.

#### 4.3 Deep Learning

Both the physical production line as well as the simulated production line are capable of exchanging data through the physical and simulated edge controller with a deep learning component located in a cloud. The physical edge controller can load a copy of a trained neural network and provides a method for local inference. On the other hand, the simulated edge controller can send generated data to the cloud as training data or it can send unlabeled data for inference. In the letter case predicted parameters describing the class and the locations of the currently observed products are returned.

The deep learning component uses the training data, generated by the digital twin and the physical production system, to train a neural network to both recognize and locate new products. It detects objects in point cloud recordings of a scene. The training data we need to generate consists of three parts. Firstly, the point cloud of a scene. Secondly all visible bounding boxes in this scene and thirdly a mapping from each point in the point cloud to the center of the corresponding bounding box. The deep learning component and the webservice will be provided by Vathos GmbH.

#### 4.4 Web Service

For easy access to the cloud and the deep learning component a web service is used. The web service allows a user in the production staff to upload the CAD data of new and unknown products to the cloud. He should also be able to initiate the start of the training process since this is the most expensive part of our system. The physical edge controller can synchronize with the cloud, loading a copy of the current neural network for a local usage. In this case the production system will keep working even during disturbances in the internet connection. Besides, the time needed for inference is shorter by using the edge controller in contrast to the time needed to access the cloud for inference. The simulated edge controller has a different duty than the physical one. Through the web service it can download the CAD data of the new products and replace old products with the new one in our simulation. After generating labeled training data, the edge controller can use the web service to upload them to the cloud where a neural net will use them to train.

Since Rüstflex also uses a web service for easy usage, it can be used as groundwork for the web service used in our project, which is currently in progress.

#### 4.5 Demonstrator

The demonstrator is based on the production process from aha! Albert Haag GmbH. It combines all previously mentioned components. A simplified version of the described production line will be built by Arthur Bräuer GmbH & Co. KG [2], an integrator of robot arrangements. The goal of the demonstrator is to show that it is possible to easily adjust an automated process to new and unknown products using only data driven algorithms. We will use it to test our system in the applications from both aha! Albert Haag GmbH as well es Rewe Digital [10]. While the process from aha! Albert Haag GmbH is used to build our system on, a different process from Rewe Digital in logistics is considered to ensure the generalization of our system. For the parametrization of the robot, we will use Rüstflex. The results of the demonstrator will then be used to optimize the other components of our system.

#### 5 Conclusions and Future Work

We are currently in the beginning stages of this project. Until now we have worked out the finer details of our system, specified the system architecture, which is partly shown in Fig. 2. We have specified the workflow of our chosen production line and built a first simulation model of said process. Vathos GmbH is developing the web service and the deep learning component. Currently we work on a digital twin of our chosen sensor and a component for labeling the generated data.

Further plans regarding this project contain finalizing our simulated sensor and training a network using the resulting data. At the end of our project, we hope to show that it is possible to improve a production process using our system and applying machine learning in industry using only synthetic data. Since a lot of clients wish for their production lines to be capable of dealing with small lot sizes, our approach would enable Bräuer GmbH & Co. KG to offer their clients more flexible production systems. In addition, small and middle-sized companies like Rewe Digital and aha! Albert Haag GmbH could improve their current and future production systems. We hope for our results to help many other such companies to further automate their manufacturing processes.

And lastly, the success of our project will lead to the capability of industry robots to solve certain tasks more autonomously, making it possible to abstract the instructions given to the robot.

#### References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### Usage of Augmented Reality for Improved Human-Machine Interaction and Real-Time Error Correction of Laboratory Units

Vadym Bilous, J. Philipp Städter, Marc Gebauer and Ulrich Berger

#### Abstract

For future innovations, complex Industry 4.0-technologies need to improve the interaction of humans and technology. Augmented Reality (AR) has a significant potential for this task by introducing more interactivity into modern technical assistance systems. However, AR systems are usually very expensive and thus unsuitable for small and medium-sized enterprises (SMEs). Furthermore, the machine's reliable data transfer to the AR applications and the user activity indication appear to be problematic. This work proposes a solution to these problems. A simple and scalable data transfer from industrial systems to Android applications has been developed.The suggested prototype demonstrates an AR application for troubleshooting and error correction in real-time, even on mobile or wearable devices, while working in a laboratory unit to simulate and solve various errors. The unit components (small garage doors) are equipped with sensors. The information about the state of the system is available in real-time at any given moment and is transmitted to a mobile or wearable

Chair of Automation Technology, Brandenburg University of Technology Cottbus—Senftenberg, Siemens-Halske-Ring 14, 03036 Cottbus, Germany e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
J. P. Städter e-mail<EMAIL_ADDRESS>
M. Gebauer e-mail<EMAIL_ADDRESS>
U. Berger e-mail<EMAIL_ADDRESS>
263

V. Bilous (&) J. P. Städter M. Gebauer U. Berger

T. Schüppstuhl et al. (eds.), Annals of Scientific Society for Assembly, Handling and Industrial Robotics 2021, https://doi.org/10.1007/978-3-030-74032-0\_22

device (tablet or smart glass) equipped with AR application. The operator is enabled to preview the required information in a graphical form (marks and cursors). Potential errors are shown and solved with an interactive manual. The system can be used for training purposes to achieve more efficient error correction and faster repairing.

#### Keywords

Augmented reality Error correction Human-machine interaction Industry 4.0 Technical assistance systems Data transfer

#### 1 Introduction

Augmented Reality (AR) is the computer-based extension of reality using smart glasses or other hardware devices. AR technology nowadays has a wide range of application areas [2, 15, 16]. By using AR, the reduction of error search and error correction is achieved. AR technologies have a great potential in commissioning [21] and maintenance [17], for assistance in training areas [10], assembly, repair, or automation [9]. AR applications (AR apps) serve as a useful tool in various technical and production fields (e.g., mechanical engineering and manufacturing automation).<sup>1</sup>

In the past years, AR technologies have been increasingly exploited in technical assistance systems, namely the visual ones [22]. AR is being used for the visualization of assembly tasks, machine operation, or repair processes [1, 6]. With the support of cameras (e.g., integrated into tablet or smartphone), the system automatically recognizes a malfunction or broken components, allocates the relevant information, and displays the exact assembly instructions directly in a real image. Thus, the automatic fade-in of the circuit diagram while working on an electrical system component is conceivable. For a technician, this simplifies recording a very complex system and the analysis of relevant measured values. The assistance via "instructions and feedback" [4] for technicians and other workers can even be originated from remote experts and executed in real-time [8].

Considering the current development stage of hardware and software in the field of AR technology support, writing user applications has become relatively simple and less time-consuming. Developers provide a vast amount of free software, which can be used for complex applications like error correction in mechanical problems [3].

<sup>1</sup> Usage of the created AR application is demonstrated here: https://www.b-tu.de/owncloud/s/ HwS8ZGFtd8ddToS.

#### 2 State-Of-The-Art Applications and Motivation for the Study

As it was mentioned in the introduction, the implementation of AR in the fields of mechanical engineering and manufacturing promises significant perspectives to reduce the complexity and increase interactivity of the human-machine communications.

Despite numerous research and development projects in the field of AR technologies [5], it has not yet been widely used in mechanical engineering. The main reason is the hardware's high pricing and heavy weight. From the first experiments from 2003 to the increasing interest since 2008, the hardware available for AR apps shared the abovementioned characteristics [4].

The recognition of real objects, which is based on artificial intelligence (AI), is essential for the further positioning of virtual objects in AR. However, the detection of 3D objects (e.g., machine parts) usually requires industrial hardware and expensive software. For example, the DIOTA company [7] offers common alternatives for hardware and software. Industrial tablets such as GETAC or head-mounted displays from HTC or ISAR Projection System are used. For the software, examples from DIOTA (DIOTA Player and DIOTA Connect) and future AR software from Siemens are in operation [7]. Most of the current solutions are mainly interesting for large industrial customers but not for SMEs.

Thus, the most important problem of the wide use of AR can be formulated as follows: many developers traditionally use rather expensive development tools and software products.

A project in the Brandenburg University of Technology Cottbus—Senftenberg (BTU) at the Chair of Automation technology [3] was initiated to solve this problem. The aim of the conducted development was to build an AR application for education and error correction. The software tools Unity3D and Vuforia, due to their availability to the project team, were used during the development process. Several new problems regarding the AR application development were identified and solved.

#### 2.1 The Problem of Data Transfer

The data transfer between industrial systems (for example, programmable logic controllers, PLCs) and Android applications is a rather complicated task when using simple development tools. The basic approaches are primarily focused on transferring data from one single PLC to one or more Android applications, for example, SIEMENS PLC support the data transfer with TCP/IP protocol [18]. But this task is getting even more complex if the solution has to be scalable, i.e., effective not only for one unit, but also for several ones [12, 19, 20]. Overall, the problem of data transfer from industrial controllers (PLC) to AR hardware is currently not well developed. Thus, a new data transfer method has to be established.

#### 2.2 The Problem of User Activity Indication

The data transfer between the programmable controller (PLC) of the plant or laboratory unit and the user is a part of the AR ergonomics problem, which is currently under development [14]. The user's Android application should not only receive data from the PLC but also transfer data back. The reason for the introduction of bidirectional information transfer is that the majority of the user's actions with the mechanical environment during the error correction or learning process cannot be detected by the environment's sensors. On the other hand, the importance of comfortable and easy communication between the user and the AR app is essential, which was explored in particular in [11]. Therefore, the user must be enabled to mark his actions with other tools, e.g., virtual keyboard, joystick, or application interface buttons (if any hand-held device as an AR hardware is used).

#### 3 Method of Development

The following steps for the development of the project have been taken:

This method of work with AR projects in BTU was formulated in the previous related work [3] and projects with AR HMI system for small laboratory plants). As for the application of the method, two important points required special attention. Firstly, the laboratory units needed a modular design to simplify possible modification.

Secondly, the supporting elements of the system were required. The complete scheme of the project released according to the method is shown in the Fig. 1.

The developed system simulated the operation of an automated garage doors facility and related maintenance processes.

#### 3.1 Hardware Components

The laboratory unit was equipped with a Siemens PLC. AR information could be displayed on a tablet, smartphone, or HMD (Human-Machine Display). The data transfer from the door control hardware (PLC) to the user hardware (smartphone and cardboard) occurred via OPC UA. The latter is a universal interface to the plant network (control components of the industrial plants). The transmitted information constituted a list of

variables and their states. In this way, the information about the system components and the operating errors would be transferred to the AR application.

Reasons for the selection of OPC UA as the (data transfer) media were the following:

In the first developing phase of the AR application, mobile hardware—an Android tablet—was used. In the final phase of the testing of the latest version of an AR app, a "wearable" device such as a smartphone and a cardboard box was used. A solution with real AR glasses, such as HoloLens from Microsoft, would be quite expensive for SMEs, therefore, such a configuration of an AR device was introduced to resemble the real future operating conditions. By using common and affordable devices such as tablets and smartphones, a distinctive balance of cost and capabilities may further be achieved.

#### 3.2 Software Components

The software system used to develop the AR app consisted of the Vuforia Augmented Reality SDK in conjunction with the Unity3D physics engine.

The reason for the selection of Vuforia as a software solution was its broad functional capabilities. With Unity3D, complex dynamic models can be realized. The control of the models is possible, as well as communication with other programs. This development environment can be used for complex tasks and is free available for research and non-commercial projects. An alternative software solution to create an AR application could be another physical engine like, for example, Unreal Engine (UE). However, UE needs a high level of experience in C programming and is not free of charge. Nevertheless, the respective software is becoming more and more popular in large projects, as it offers significantly more options in the future.

Therefore, the main system components constituted Siemens PLC, OPC/UA interface and server, AR Hardware (Android smartphone and/or tablet, VR cardboard), Unity3D and Vuforia as the development environment.

#### 3.3 Project Development Process

At the start of the project, a miniature laboratory facility (doors) was created and used. The experimentation unit contained a movable door, two user interfaces, and two tracker images. There were different tracker pictures for the positioning of AR elements and AR keyboard. The first prototype of the laboratory facility is demonstrated in Fig. 2.

The proposed system enabled the user to carry out advanced operations usually done by experts. The set of implemented standard errors and how they were solved showed the AR app opportunities for training and non-expert error correction.

Three errors could be simulated and indicated with this laboratory unit:

For the first prototype of the laboratory unit, a virtual keyboard (SVC) with two buttons was developed (Fig. 3.). This solution was created to solve a problem of the user activity indication (see Sect. 2.2). The user was required to switch between different instructions when fixing an error. The button "NEXT" could be activated to go to the next step of the error correction. The button "BACK" enabled returning to the previous step if checking off the correction action was needed.

The system under consideration was presented at the Hannover Expo in 2017 and 2018 as a part of the exhibition stand of the IHK (Industrie- und Han-delskammer) Berlin-Brandenburg. The overall number of the attendants of the exposition who tested the prototype amounted to about 100 people. Each tester was informed about the presence of simulated malfunctions in the laboratory unit, but was not informed about the possible course of actions explicitly. It was the job of the AR app to provide the users with

Fig. 3 Using virtual buttons ("NEXT" or "BACK") for step switching in the first prototype of the project

step-by-step instructions for problem resolving. Mobile hardware, namely a tablet with an Android system, was used. Such practical use cases have shown that SVC increased the user's error handling capabilities by freeing the hands from the joystick or any other remote control device. However, SVC seemed to require an extra tracker picture. It additionally reduced the user's mobility if subsequent movement around the unit was required.

The analysis of the application of the first prototype of the project allowed us to formulate a solution to the two-way communication.

#### 3.4 The Solution to the User Activity Indication Problem

A universal solution to problem of the user activity monitoring and two-way communication was not possible by software methods alone. Essentially, it required the tracking and identification of the possible actions. The latter was only feasible if the laboratory plant was equipped with additional sensors capturing each action of the user. The transfer of subsequently appearing data was also required.

Thus, the second prototype of the system was developed. The laboratory unit has been modified by introducing a new console for technicians. The SVC tracker was no longer needed. The new sensors allowed to control the user actions and the steps due to laboratory tasks.

The operator of the system could use wearable AR hardware (smartphone with an AR application and cardboard). He acquired the related status information on the door, as well as a guided manual. The outline of the user interface and the prototype overview are presented in Fig. 4.

There were three control buttons on the new user console box, which allowed for extended interaction patterns:

"Start". If this button was pressed, the garage door would open. If nothing appeared to be passing through the door, the latter waited for 12 s, and then closed.

"Stop". If the Stop button was pressed, the door stopped moving.

Fig. 4 Laboratory unit and all AR elements (small garage doors, second prototype)

"E-Button" (emergency stop button). If this button was pressed, the door was stopped, the movements of the system components were blocked, and a restart with a technical console box was required.

#### 3.5 The Solution to the Data Transfer Problem

The primary idea for the development was a direct data transfer system. As previously described (see Sect. 2.1), the flexible and scalable data transfer between industrial solutions (e.g., OPC/UA) and Android applications appears to be a major challenge. Direct data transfer from the server to the Android system is also possible, but requires extensive knowledge of software development. This contradicted the philosophy of the project by increasing the complexity for the end-user and for the developer.

Therefore, a support web server was introduced. It received data from the OPC/UA. The Unity3D Web Socket (integrated into the AR app) collected the data from the web server (Fig. 5).

Such a communication structure had a feedback time of approx. 0.3 s, but was simple in development, reliable, and offered easy maintenance and scalability (numerous AR hardware units could take information from the support web server).

Furthermore, our solution appeared to be more universal, as it made OPC-UA available for every platform that supported Web Sockets. We thus enabled OPC-UA for platforms where no ad-hoc OPC UA support library existed. The programming language used in AR app did not play a significant role (C#, JavaScript, Unity, Python, etc.). It should be mentioned that we used JSON-RPC (JavaScript Object Notation Remote Procedure Call) via Web Socket, starting the calls on the web server as a result. In other words, our solution brought OPC-UA into the browser.

#### 3.6 Resulting System Testing

To test the second prototype, we enlisted the help of laboratory visitors and university staff members. In total, the number of the participants of the experiment amounted to about 30 persons in 2019 and 2020. The same testing methodology as with the first system prototype was used.

The experimental results showed that 100% of testers were able to fulfil the service requirements in the laboratory unit for the first time in an error situation by following the AR app guidance. However, more than 60% of testers reported the problem of a the loss of position of virtual objects during sharp movements of the head. The issue was eliminated by modifying the tracking algorithms from Vuforia and using the extended tracking. With the modification under consideration, the system could work even when there were no trackers in the camera's field of view. Nevertheless, at the start of the program, the tracker image was still needed to initialize the object positions.

#### 4 Technical Maintenance Simulation (Error Correction)

To solve the introduced error with the "short" door blocking, the user received a message to clear the door's path. (Fig. 6a).

In the case of "long" door blocking, the user had to clear the door's path at first. Then the user needed to follow the AR instructions to activate the manual control mode (Fig. 6b) and close the door manually with the "Down" button (Fig. 6c).

Fig. 6 a The first error ("short" door blocking). b Switching in the manual modus. c Manual control of the door movement. d Release of E-button

As previously described (see Sect. 3.6), the system could detect errors and control every relevant step of the user and error correction. For this reason, there was automatic switching between AR instructions for individual steps.

If a door was blocked with an E-Button, the user was informed about further steps. He had to release an emergency stop button (Fig. 6d) and then (as explained before) switch the system to the manual operation before closing the door and restarting (as on the Fig. 6a and b).

#### 5 Results

The AR application developed in this project can be used as an assistance system to solve problems and errors with a laboratory unit. Such interactive facility replaces the traditional study of maintenance manuals. The visual feedback informs the user about errors in the system and whether they have been rectified.

Based on the recognized tracker pictures, AR objects can be displayed on the needed position. After the first recognition of the picture, the system can work when there are no more trackers in the camera's field of view. These AR objects reveal the error's exact location and give instructions for user guidance to resume the system's proper operation. The user is enabled to quickly and successfully follow the necessary steps of error correction. The use of the system for training and education purposes is foreseen, and this system has positive effects on reduced downtime.

#### 6 Conclusion and Future Work

This work presents a practical application of Augmented Reality for machine environment maintenance. It is shown that communication between the current hardware and software components (OPC/UA and Android applications) and, therefore, technical assistance system realization, is possible without additional expenses. Moreover, the project demonstrates the possibility of developing and using an AR application with physics engine software (Vuforia Augmented Reality SDK in conjunction with the game engine Unity3D).

In the project presented in this article, a new scalable method of data transfer has been developed and tested. The achieved solution is portable and available for every platform that supports Web Sockets. The problem of user activity indication is solved with the implementation of new hardware elements in the mechanical environment.

The authors would like to further emphasize the importance of the equipment of modern generations of systems and plants with an excessive number of sensors [13] to precisely detect each user's action and the order of these actions for the further successful implementation of AR assistance systems. The next step may be providing the support of more machine operations. Further development in this direction might include the binding of additional AR visual effects libraries, as well as the introduction of new user-machine interactions. It might enable the use of a single OPC/UA solution for collecting data from even more units and plants in the laboratory. The collected data could finally be used for more AR hardware and operators.

#### References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### Scalability of Assembly Line Automation Based on the Integrated Product Development Approach

Florian Hoffmann , Vanessa Wesskamp, Raphael Bleck and Jochen Deuse

#### Abstract

Product life cycles change, market developments and quantities are increasingly difficult to predict, as is the case in the production of charging stations. For these reasons, scalable assembly concepts with an adaptable degree of automation are becoming increasingly important. Currently, charging stations are still manufactured manually. With increasing quantities, however, manual production is no longer economical. New technologies such as lightweight robotics offer a great potential for making production more flexible in terms of quantity. At the same time, new challenges arise because these requirements must be taken into account from the very beginning of product development and process planning. Currently, there are no planning approaches and recommendations for action that take this into consideration. Therefore, the research project "Simultaneous product and process development of a charging station outlet module suitable for automation" (SUPPLy) develops an integrated, digital and simultaneous product and process development of a modular charging station suitable for automation. The aim of the project is to develop an assembly process which enables an economic production of charging stations in case of

Institute of Production Systems, TU Dortmund University, Leonhard-Euler-Straße 5, 44227 Dortmund, Germany

e-mail<EMAIL_ADDRESS>
J. Deuse

F. Hoffmann (&) V. Wesskamp R. Bleck J. Deuse

Advanced Manufacturing, School of Mechanical and Mechatronic Engineering, University of Technology Sydney, Sydney, Australia

T. Schüppstuhl et al. (eds.), Annals of Scientific Society for Assembly, Handling and Industrial Robotics 2021, https://doi.org/10.1007/978-3-030-74032-0\_23

fluctuating sales figures. The focus is not only on changes in the production process but also on a product design that is suitable for automation. The paper presents the ideas on a conceptual level.

#### Keywords

Integrated product development Assembly Scalability Simultaneous engineering Industry 4.0

#### 1 Introduction

In order to maintain the competitiveness of the German automotive and supplier industry and to survive in global competition, Germany is to become the lead market for electric mobility [1]. To achieve this challenging goal, one million electric cars are to be in use on the roads by the end of 2022 [2]. The government's goal, and with it the establishment of electromobility as part of a sustainable transport system, will only be achieved if charging infrastructure solutions are available across the board that allow electric vehicles to be charged at any time and as needed. The development in this area is currently still in a premarket phase. Accordingly, demand for such products is still at a comparatively low level, meaning that extensive investments in plant and automation technology have not yet been economically viable for manufacturers. Due to low market demand and lack of economies of scale, the components and the manufacturing processes are very cost intensive, which in turn leads to high prices for the charging infrastructure. According to various forecasts, however, there will be a rapid and sustained increase in demand in the coming years. A nationwide charging infrastructure can only be set up if the corresponding charging stations can be built economically. So that a progressive industrialization of products and the corresponding production processes will become indispensable [2–4].

This paper presents a new approach for a scalable line automation. Chapter 2 gives an overview of existing planning approaches followed by a specification of scalability in charging station production, including the importance of human–robot-interaction (HRI) in Chap. 3. The developed concept is outlined in Chap. 4 including a theoretical application scenario. The paper concludes with an outlook.

#### 2 Planning Approaches

Planning systems represent an important thematic basis for the development of scalable assembly concepts. They serve the analysis, planning, design and improvement of socio-technical work systems and can lead to more efficient processes through clever application. One of the conventional planning methods is the planning systematic according to REFA, a german association for work design, business organization and business development [5]. This contains general instructions for planning complex manufacturing and assembly systems. A central feature of the system developed by Rother and Harris [6] is the possible reduction of the degree of automation in order to enable increased flexibility and a continuous flow of employees. Dietz's [7] approach addresses the high and dynamically changing variant diversity in production systems over time and provides ideas for product structure dependent manufacturing system design. In contrary, the MABA-MABA (men are better at—machines are better at) planning approaches focus on skills-oriented process design between man and machine in the context of increasing automation [8].

Furthermore, production systems as a whole are increasingly exposed to time dynamic influencing factors, so that the planning of changeable and adaptable systems is also of increased importance for the planned research project [9].

Production oriented approaches are particularly suitable for planning manufacturing and assembly systems with high volumes and low product variance. In many cases, these methods are geared towards more capital intensive systems in order to produce the required quantities as economically as possible by exploiting economies of scale. In order to enable the production of an economically viable product even in the case of fluctuations in demand, the pure consideration of production oriented approaches is insufficient. These approaches must be combined with measures to optimize product development in order to achieve a holistic method.

In this context Hengstebeck [10] provides an insight into the different areas that are affected by the planning of semi automated work systems (e.g. product development, production planning, maintenance). Deuse identifies a great potential in the realization of cyber-physical production systems (CPPS), in which human capabilities are enhanced through the intelligent use of information and communication technologies. For this purpose, new collaborative forms of work must be developed in which humans can optimise processes and act as active decision makers [11].

Simultaneous engineering enables a shorter time-to-market through a close integration of product development and production planning, which can lead to competitive advantages. In literature there are promising approaches that show possible process models. Erlenspiel [12] describes a method that focuses on development and construction and addresses simultaneous planning across phases. Similarly, Bullinger contributes to simultaneous engineering, focusing on efficiency and cost aspects [13]. Overall, it can be stated that none of the planning approaches presented above sufficiently meets the requirements for designing a scalable production system. The core objective of SUPPLy is to develop a scalable assembly system for the production of charging stations that ensures cost-effective operation at all times during production. For this purpose, it is necessary to continuously adapt the production capacity to the respective situation on the sales market [14]. The basis for this in a high wage country, such as Germany, is a flexible adaptation of the degree of automation with the help of HRI. Product design is being rethought to maximise flexibility and aligned with the needs of HRI technology. As outlined in the next chapter, several approaches have to be combined to meet the challenges of the market.

#### 3 Quantity Dependent Charging Station Production

Although the German government has set concrete targets for the number of electric cars on the road for the next few years, it is very difficult to predict how sales will develop. Political measures, such as targets for fleet consumption, purchase premiums, or an adjustment of the motor vehicle tax to match emissions, lead to fluctuating demand in the market. [15]

Due to the prevailing market situation, specific requirements for the work system planning arise. Short planning cycles as well as a good and economic scalability are the core challenges in the planning process. Conventional sequential planning of assembly systems is not able to meet these market dynamics. In high wages countries a scalability using only personnel deployment is not economic. As explained above, an adaption of the degree of automation is a frequently used instrument. However specialised semi-automated or automated systems cannot react to short-term fluctuations. Taking increasingly shorter product lifecycles due to a fast technological progress into account, these systems are not economical either. Lightweight robots offer a new possibility for an adaptable degree of automation. Accordingly, the planning concept must combine HRI-compliant and modular product design with scalable assembly system development [16, 17].

Comparatively low investment costs and the possibility to use them with small adaption in different production systems as needed and the option to realise HRI, promises a high potential to increase flexibility. HRI offers many possibilities for automating manual assembly processes making optimal use of the strength and weaknesses of humans and robots [18]. By exploiting these synergies, the output volume can be increased by using a robot to partially automate previously manual tasks [10].

When planning and realizing a flexible automatable assembly system, greater attention must be paid to the expandability of the assembly system and the reusability of components. A modular system structure is recommended in order to minimize the resulting costs for rebuilding measures or for the restart [19]. The lightweight robots can be integrated with only minor changes to manual work systems and are able to produce especially small and medium quantities efficiently and economically [20, 21].

To realise this potential, the product must be suitable for the handling and assembly by robots as well as manual handling. The geometry of a product significantly influences the way it can be picked up by the robot end effector. Sharp edges or protruding parts make handling just as difficult as fragile plug-in connections. In many cases, bendable parts are not suitable for automated handling. In this context, design alternatives can already be considered during product development. With regard to the payload, lightweight robots capable of interaction are usually limited to single-digit kilogram ranges due to safety requirements. Due to these framework conditions, the production use of HRI is significantly determined by the product design [22, 23].

For this reason, when planning a scalable work system with a variable degree of automation, it is very important to involve the construction and design department from the outset in order to minimize scaling costs during production [24]. In this context, Nyhuis defines the five (product) characteristics universality, scalability, modularity, mobility and compatibility as so called transformation enablers. With the help of these characteristics, the requirements for a flexible assembly system and its components can be described [25].

External disturbances influence the assembly system during ongoing production operations and require constant adjustments [26]. In the present case of charging station assembly, significant fluctuations in demand must be taken into account for the reasons mentioned previously. For the first concept of a scalable charging station assembly line, scalability and modularity are particularly important in order to create an optimal basis for further flexibility-increasing measures. In the context of the project, modularity relates in particular to product development, which is at the beginning of the development process and has a direct influence on the subsequent scalability of production.

Production currently involves the assembly of numerous individual components, including their cabling. A promising approach in this context is the development of a modular product design that reduces assembly complexity and favours HRI. This in turn enables the necessary scalability of production depending on market needs. Due to the need to consider assembly possibilities and HRI suitability already in the product design, the SUPPLy project, as shown in Fig. 1 focuses on simultaneous engineering in relation to the integrated product development approach.

By linking modularity and scalability, an assembly concept is created that offers maximum adaptability depending on the quantity requirements while addressing the need for short planning cycles. For subsequent development steps, compatibility, universality and mobility must be taken into account with the aim of further increasing flexibility. By

Fig. 1 Integrated product development approach in the SUPPLy project

taking these characteristics into account during product and process design, the future expandability and changeability of the overall system is guaranteed.

#### 4 Application Concept