In addition to the above, it is important to mention other international and national classifications that are either of reference or are applicable in other countries and are inspired by or refer to the above. The International Standard Classification of Occupations [6] (ISCO) is an international classification developed by the International Labor Organization of the United Nations. This classification and, in particular, its publications ISCO-88 and ISCO-08 define groups and titles of occupations associated with tasks. In general, these database and associated documents are less comprehensive than the three previous ones, but in certain occupational databases, connections with ISCO allow occupations to be linked between databases in different countries.

One of the most common criticisms of these classifications is that they are created and adapted in high-income countries. For this reason, initiatives to map competencies in other low/middle-income countries are also being undertaken. Two examples of these initiatives are STEP (Skills towards Employment and Productivity) and PIAAC (Programme for the International Assessment of Adult Competencies). STEP is a World Bank program to improve the understanding of skills (cognitive, non-cognitive, and technical) in the labor market and relates skills to education, through household surveys and employer-based surveys. It is a program of the Organisation for Economic Cooperation and Development (OECD) that analyzes socio-labor and educational characteristics and links them to competencies at different cognitive levels such as literacy, numeracy, and problemsolving. The first cycle of PIAAC started in 2011 analyzing 42 countries and its second cycle will run until at least 2029 covering 30 countries.

All these initiatives and databases provide a stable and updated background, based on reliable sources and in many cases that can be linked together, which allows us to propose ideas and proofs of concept such as the one mentioned in the following section.

30 https://noc.esdc.gc.ca/Oasis/OasisWelcome.

31 https://www.statcan.gc.ca/en/subjects/standard/noc/2021/introductionV1.

#### **5 Proposed Solution**

The goal of the solution we have proposed in recent papers [1–3] is to make available to users, who are not always technologically prepared, the information contained within any of the resources described in Sect. 4 in the form of natural language and to help them analyze which skills and personal knowledge are most useful for their application in any job.

One of the worker upskilling resources covered in Sect. 4, the O\*NET database, contains many types of information. Our system uses the titles of the principal job categories that cover all the jobs in the North American job market, their descriptions, and the alternative titles for each one. Abilities, Skills, Knowledge, Work Activities, Tasks, Technology Skills, and Tools are the information that characterizes each job category. The number of elements for each job is constant for the first four categories and variable for the last three. The items in the first five present a score relative to their importance to the job they refer to, while the last two are lists of tools or technologies commonly used in each job category.

More in detail, the system we propose performs the following tasks:

1. Determine how much the user's resume is appropriate for the chosen job category.

The system uses NLP techniques to analyze the user's resume and to extract its most important information. The database entities and the extracted content from the CV are compared using semantic similarity. The resume score for a job is the sum of the scores of the discovered elements divided by the maximum score obtainable for the job itself and a corrective factor required to avoid penalizing jobs with few items in the entities with a variable number of elements. The system draws attention to the lacking aspects of the CV and suggests ways for users to increase their knowledge of each one.

2. Determine the user's suitability for the selected job.

The system prompts the user to enter his/her self-assessed level (none, little, medium, good, excellent) in terms of Abilities, Knowledge, Skills, Work Activities, and Tasks entities taken from the database and associated with the selected job. The system returns a percentage score, where 60% is the suitability threshold and a list of lacking knowledge or abilities that the user might improve to raise his/her score.

3. Determine what occupation the user is best suited for.

The system prompts the user to enter his/her self-assessed level (none, little, medium, good, excellent) in terms of Abilities, Knowledge, Skills, and Work Activities entities taken from the database and returns a list with five suggested job categories.

4. Perform conversation with the user.

To integrate the knowledge provided by O-NET into the conversational agent, we first chose a reduced set of questions that it must be able to answer. The base question templates are:

Then we implemented the query models necessary to retrieve the data to generate the answers.

When the conversational agent recognizes a question in the user's input that is part of the set of known questions, it requests the database for the data necessary to prepare the answer in natural language.

The system uses semantic similarity to match user input with one of the predefined question templates. If the similarity value is not high enough, the system will answer without the database knowledge, returning a default answer, or using a generative pre-trained model. More details of our system can be found in [1–3].

#### **6 Expected Challenges, Benefits, and Impact**

In terms of challenges, employee engagement in training activities and work commitments is one of the key elements. Knowing the information about their job and potential growth paths, while reducing doubts about the stability of their job, allows their adaptation and evolution, and thus engagement. Interactive chatbots and conversational tools can simplify the task of obtaining this information, for example, through informal interactions and by simply relying on the chatbot as a companion that offers relevant and appropriate information, focused on the characteristics of each occupation.

As far as technical challenges are concerned, it is still necessary to reach an adequate balance between the use of large language models (e.g., ChatGPT) and models trained with concrete content based on validated or accepted reference information. One of the main problems of LLMs is the lack of determinism in the results [11] and the potential failures in the recommendations [13]. Both can be compensated by relying on databases such as the ones mentioned above and prioritizing the results based on the information in those databases.

It is important to also mention and consider the challenges of conversational agents related to privacy. While many of the ideas mentioned in this article do not require storing worker information, and in our use cases we have demonstrated this, there are several cases, such as group data discovery or candidate analysis, where the handling of worker data could end up in the processing of private data, aggregation

of personal data and in some cases in profiling. The topic related to the concerns about user data's inappropriate use in user-chatbot interactions is being researched and surveyed [4, 8], and still there are several debates about it at the international level (e.g., in April 2023 Italy was the first Western country to block ChatGPT32 ). This calls for a critical analysis of the different uses of chatbots for upskilling and, above all, an alignment with the initiatives for the maintenance of the privacy of workers and users, from the General Data Protection Regulation to the Artificial Intelligence Act.

Regarding benefits and impact, at the scientific and technical level, and as can be seen in the state of the art, advances in conversational agents and language models are clear. All these advances have an impact on the development of solutions such as the one we present, for example, by providing better interaction capabilities or even the possibility of answering open-domain questions. At the economic level, as also discussed in the introduction, there is a demand for workers and a demand for knowledge for specific activities, and upskilling agents can facilitate this task. At the social level, it impacts workers and the unemployed, in terms of analyzing their strengths, detecting training or internship needs, and adapting their CVs to fit specific jobs.

From the worker's perspective, conversational AI agents based on standardized and reliable databases of occupational information have direct impacts and benefits.

On the one hand, these conversational solutions have a direct impact on workers' knowledge of their work activity. The workers themselves can at any time obtain reliable information about their job and learn about skills, activities, and knowledge relevant to the task they are carrying out. Moreover, the worker can ask the conversational agent which are the usual technical skills for a certain occupation or activity to imagine where they might be oriented.

On the other hand, it is beneficial for their training and career. The employees can detect points in which their profile would benefit from training and at the same time detect training needs and create training plans to adapt to the needs of their position, or to another position they plan to move to. This is one of the other benefits of these conversational agents since the workers can compare themselves with professions and positions in which they would be interested in changing or being promoted.

From a more organizational and business point of view, there are also a couple of areas where AI-based conversational agents such as the ones mentioned in this article can be beneficial. The part of assistance in hiring processes is a point in which these agents will have a direct impact: from providing candidates with information about the position, to pre-screening or determining the suitability of a candidate by using conversational agents to compare the candidate's CV with the baseline of the occupational reference database. Conversational agents are also beneficial in the detection of training needs for specific positions or the creation of group training plans, based on the needs of multiple workers.

32 https://www.bbc.com/news/technology-65139406.

Therefore, the impact on the HR area is clear: talent management, discovery of training paths, HR resources, growth programs, etc. All of them benefit in the short term from conversational agents like the ones mentioned above. Proofs of concept and use cases conducted with conversational agents, based in our proposed solution, acting as virtual interviewers, skills gap detectors or as supporters for the selection of better training, demonstrate the potential of these technologies. Not only at the level of large technology companies but the aforementioned solutions can also make a difference at the level of SMEs in industrial and manufacturing environments. Where they may not have large HR departments but need reliable, objective, and consistent feedback for resource upskilling and solutions based on standardized information, starting from the same baseline and with clear and repeatable criteria.

#### **7 Conclusions**

This chapter explores the potential uses, benefits, and limitations of conversational agents within the labor market. The chapter guides the reader through the state-of-the-art research on this topic and presents which technologies are currently employed. Furthermore, the chapter analyzes conversational agents that are disrupting the labor market, where AI tools are expected to be employed for a large variety of tasks such as learning new skills, supporting document writing, code development, language skills improvement, and so on. Along with these conversational agents, we also introduce the reader to valuable sources of information which can be leveraged to build AI systems on top. Finally, we present how potential solutions can be used for specific use cases and discuss how all these technologies can impact the labor market.

As main lessons, we would like to mention the need for both the workforce and companies to adapt to the rapidly changing world of new technologies and how this can be beneficial, for example, by taking advantage of it to better understand occupations, retain talent, hire, gain job knowledge, or adapt to new activities. It is also important to consider the need to efficiently integrate large language models and validated and reliable databases. Finally, it is essential to consider and analyze the privacy implications of these solutions. It is possible to develop privacy-aware conversational systems for the upskilling of workers, but there is a possibility that certain solutions may abuse data. For this reason, it is critical to provide these systems with consideration of Human Centricity aspects and all the factors that compose it, including privacy.

In the medium term, it is expected that these technologies will be integrated into the regular workflow of employees and will have a positive impact on employee engagement and professional development. For this reason, we expect that this chapter will foster several visions to the reader about the role of AI-based conversational agents in the labor domain and worker upskilling which might bring unprecedented development in the field.

**Acknowledgments** This work has been partly supported by EU H2020 research and innovation programme project STAR—Safe and Trusted Human Centric Artificial Intelligence in Future Manufacturing Lines (Grant n. 956573).

We acknowledge financial support under the National Recovery and Resilience Plan (NRRP), Mission 4 Component 2 Investment 1.5—Call for tender No.3277 published on December 30, 2021 by the Italian Ministry of University and Research (MUR) funded by the European Union— NextGenerationEU. Project Code ECS0000038—Project Title eINS Ecosystem of Innovation for Next Generation Sardinia—CUP F53C22000430001—Grant Assignment Decree No. 1056 adopted on June 23, 2022 by the Italian Ministry of University and Research (MUR)

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Advancing Networked Production Through Decentralised Technical Intelligence**

**Stefan Walter and Markku Mikkola**

#### **1 Introduction**

Networked production has been increasingly important for companies looking to stay competitive in today's global marketplace [1, 2]. Initially undertaken due to the requirements of profitability, many manufacturers have been focusing on their core competencies and outsourcing non-core functions to specialised suppliers [1, 3, 4]. The resulting fragmentation made increased digitalisation necessary.

By leveraging technology and digital platforms, companies can connect their manufacturing activities with other organisations, suppliers, customers, and resources, both domestically and globally, to achieve a competitive advantage [5–8]. Companies do so by accessing a broader range of suppliers and customers, reducing costs, increasing efficiency, and improving the quality of their products when networking. This also allows for greater flexibility in responding to changes in demand, supply chain disruptions, or market conditions and provides the required agility that is so important to counter economic volatility or political instability [9–11].

In addition, networked production can create opportunities for innovation and value creation, as companies can work with partners to develop new products or improve existing ones. By sharing expertise, knowledge, and resources, companies can capitalise on each other's strengths and capabilities to create new business opportunities and improve their competitive position [12, 13]. Companies can thus speed up the product development process and improve time-to-market. This position is also reflected in the concepts of networking in engineering, networked product/service engineering, and collaborative design, respectively [14].

S. Walter (-) · M. Mikkola

VTT Technical Research Centre of Finland Ltd., Espoo, Finland e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
J. Soldatos (ed.), *Artificial Intelligence in Manufacturing*, https://doi.org/10.1007/978-3-031-46452-2\_16

Thus, a significant benefit of networked production is the possibility for companies to move away from traditional sequential or linear process chains and adopt a more dynamic approach to manufacturing and support processes [15, 16]. By connecting their operations with other organisations and resources through digital platforms, companies can access a broader range of manufacturing capabilities, materials, and services. For example, companies can dynamically arrange and release manufacturing and support processes in case production requires a quick scale-up to meet a sudden increase in demand. Additionally, networked production allows and virtually demands that companies share information and collaborate with partners, suppliers, and customers in real-time. This enhances transparency and visibility across the supply chain, which can help to identify and mitigate potential bottlenecks or delays [17].

The functioning of networked production is bound to the management capabilities of the companies involved. Typically, effective management in networked production has required strong leadership and management expertise to ensure that the partners involved work together seamlessly and effectively. Companies needed to establish clear communication channels and protocols, define roles and responsibilities, and establish processes for monitoring and managing performance.

Moreover, the success of networked production depends on the compatibility of integrated, cross-organisational information chains [4]. Network partners need to ensure that they can exchange information seamlessly and securely with their partners, suppliers, and customers. This necessitated the adoption of common standards for data exchange [18]. In addition, data exchange must take place on a trusted basis and therefore requires corresponding communication security measures to protect sensitive information [19].

All this will become even more difficult with a wider expanding network. The increasing complexity of networked production presents a significant challenge to management, which requires new and intelligent approaches to effectively manage and optimise the manufacturing process. While managing networked production would require a holistic understanding of the entire value network involved, including suppliers, customers, internal operations, and other stakeholders, it is hardly possible with traditional corporate management methods based on hierarchical structures [20, 21]. This is also true for the increasingly demanded agile management involving the flexibility and adaptability to respond quickly to changes in customer demand, market conditions, or supply chain disruptions.

To effectively manage networked production, companies can leverage advanced technologies, such as artificial intelligence and data analytics, to develop new management approaches and enhance decision-making. Decentralised Technical Intelligence (DTI) is a new management approach that involves the integration of human and machine intelligence in decision-making processes. DTI allows for the creation of a network of interconnected systems, devices, and agents that operate in a decentralised and autonomous manner, reducing the need for leadership through hierarchy and potentially leading to increased efficiency and productivity [22]. The role of the workforce in this integrated system is crucial, and the human dimension adds a unique quality to the process [23].

To achieve advanced networked production with the help of DTI, a roadmap is necessary, which involves several elements linked to a vision, value promise, and development pathway. The roadmap should include major building blocks, major capabilities, characteristics, and processes typical for the effective implementation of DTI in networked production. By implementing this roadmap, companies can maintain a leadership position in future networked production, which will be crucial for maintaining a competitive edge in the global market.

The technological advancements in networked and self-controlling production can offer new opportunities for creating value, but it is important to ensure that these opportunities align with business goals and are economically viable. The business perspective provides a real-world corrective, meaning that any new technological solutions must also make sense from a business standpoint, considering factors such as profitability and feasibility. Thus, it is important to balance technological possibilities with business realities to ensure the long-term success of any innovation.

This article explores the concept of Decentralised Technical Intelligence and its meaning for advancing networked production as a competitive factor. It will also discuss possible building blocks for implementing an effective roadmap for advanced networked production through DTI. Finally, this article will emphasise the need for a business perspective to ensure that any technological advancements align with the economic viability and create real value.

#### **2 Decentralised Technical Intelligence**

Decentralised Technical Intelligence (DTI) is a concept that aims to support the deployment of decentralised, autonomous systems with embedded intelligence in manufacturing. It is a response to the management requirements of networked manufacturing and aims to enable decentralised decision-making and autonomous action while reducing centralised planning or making it redundant. It arose from a demand by the ManuFuture European Technology Platform to increase productivity and efficiency in future manufacturing [22, 24].

DTI focuses on several areas to leapfrog performance gains in terms of high performance, high quality, high resource efficiency, high speed, high flexibility, selfoptimisation and self-control, to name just a few (Fig. 1). These performance gains stretch over different areas, including ICT technologies, architectures, platforms and standards, high-performance engineering, high-performance manufacturing systems, and high value-added networked production [22, 24].

According to the Plattform Industrie 4.0 [25], decentralised intelligence is grounded on the recognition that the European industry's strength is derived from a system of innovation and commerce fuelled by diversity, heterogeneity, and specialisation. These elements are integral to the European industrial society. A decentralised regime of open and adaptable systems creates optimal conditions for shaping the digital economy while adhering to the principles of a free and sociallyoriented market economy [25].

**Fig. 1** Utilising technologies to enhance universal impact (adapted from Sautter [22])

**Table 1** DTI as the next evolutionary step in industry performance (adapted from Sautter [22])

DTI is adopting this position and is developed as a systemic and interdisciplinary approach that represents a key factor for the macro- and multidimensional transformation of manufacturing systems. It builds on previous developments in systematic and automated manufacturing and can be seen as an evolutionary step towards more advanced manufacturing systems (Table 1). Earlier steps included for example control theory and systems engineering, involving feedback loops, artificial intelligence, cyber-physical systems as a connection of informational and electromechanical components, and holonic manufacturing, which incorporates autonomous and collaborative agents creating an adaptive and flexible manufacturing system [22].

By integrating human knowledge and experience with artificial intelligence and other advanced technologies, DTI aims to create a more intelligent and adaptable manufacturing system that can respond quickly to changing conditions and optimise performance. The concept recognises that humans and machines each have unique strengths and limitations, and that by combining their abilities and expertise, a more effective and efficient system can be created. This requires a multi-agent architecture, where different agents (including humans and machines) work together in a coordinated manner to achieve common goals. The goal is to create a system where each agent contributes to the overall intelligence of the system and helps to create a more efficient and effective manufacturing process.

Consequently, DTI consciously aims for holism and recognises the importance of humans in industrial manufacturing, which puts the concept in contrast to the earlier conceptualisations that were strictly technological in their orientation. DTI recognises that humans play a crucial role in manufacturing and seeks to empower them with the necessary tools and technologies to make informed and effective decisions. By integrating humans into the manufacturing process, DTI enables them to work alongside machines and systems more collaboratively and efficiently.

Sautter [22] compares DTI as being similar to our nervous system. The body's nerves make up part of the main control structure, which provides information on the environment and inner body functioning. This way, the autonomic nervous system controls many body functions and processes in a seemingly automatic way with little or no conscious involvement. Similarly, DTI ensures the smooth functioning and good performance of complex technical systems such as production systems. It allows the production system to execute tasks autonomously in interaction with the environment and keeps the system in optimal condition. DTI components can be used to make the system perform tasks autonomously and optimise its performance without central or conscious control. Therefore, DTI is concerned with self-organisation and self-steering aspects in production systems. Mere artificial intelligence, in turn, is a simulation of human intelligence processes by machines based on advanced data analytics and algorithms that emulate and simulate processes and systems. It is often used as a tool to support human decisionmaking.

The DTI concept can also be considered a forerunner or prerequisite for the industrial metaverse. The industrial metaverse, as a comprehensive digital twin of every relevant process or artefact in a factory or production network and including the human workforce, will make use of existing and developing technologies such as artificial intelligence and machine learning, extended reality, advanced data management structures and cloud and edge computing. These technologies create an interface between the real and digital worlds. This interface aims at offering fully immersive, real-time, interactive, persistent, and synchronous representations and simulations of complex systems such as machines, factories, cities, and logistics

networks in a decentralised way [26]. While specific applications in the networked production realm are still to be explored, possibilities include interactive assembling, involving, for example, network partners such as suppliers and customers, collaborative development of products and services, their design, engineering and testing, product presentations and evaluations and product-related services, sales activities and tracking of material flows, and training of personnel [27].

#### **3 Implications for Networked Production Management**

Traditionally, the objectives of networked production included the cutting of logistics costs and high inventories, in particular, by improving the coordination and integration of production and logistics activities across the network, the shortening of lead times, especially by improving the availability of information and the flow of materials across the network to generate quick responses, the enhancement of service levels, for example by improving the coordination, collaboration and understanding between companies and of companies' needs in the network, and enhancing competitiveness, in particular, by increasing the capacity of industrial companies to operate globally in an agile manner through the network [1, 2, 4, 28, 29]. All of these remain true, and these aspects are joined by more recent perspectives on collaborative design and engineering and the management of services.

Achieving these goals requires collaboration among the companies involved. Therefore, all participants need to see benefits from their effort in collaborating. Although companies will try to maintain their autonomy as actors in the network, emerging technologies enabling DTI are likely to dilute the perception of organisational boundaries in the future. Hence, production will largely take place in non-hierarchical networks where decision-making is decentralised [30, 31]. This requires the development of methods and tools for collaborative planning, management, and optimisation of production resources, as well as distributed planning and scheduling [32–34]. Other important tools for management include monitoring production and equipment and their maintenance. Increasingly, tools also need to support the requirements of the circular economy.

It is essential to ensure interoperability between the different technologies and software solutions, business and production processes, and organisational structures [35]. All new methods and tools should follow the principle behind DTI, which allows operating in a decentralised manner so that companies can operate in several production networks simultaneously. Conventionally, these functions have been relying on regular updates to a central database with limited learning capacity builtin and neither with consideration of external data. For advanced management of networked production and its monitoring, decentralised ICT systems are needed

[36, 37]. These systems use distributed knowledge and integrate heterogeneous data sources to evaluate the state of monitored systems [14].

Competitiveness is based on various performance objectives such as quality, price, delivery service, responsiveness, and flexibility. With shorter product life cycles and a greater variety of models, manufacturers need to respond quickly to customers' needs and handle capacities flexibly. The technologies driving DTI allow companies to move towards decentralised operations, take advantage of available resources and, for example, be closer to their markets.

Taking advantage of available resources via decentralised operations and intense network interactions offers, in particular, opportunities to run make-to-order and engineer-to-order strategies [38, 39]. Both strategies begin their specific operations only after a customer has placed an order. It represents in some way an orientation towards lean management because it consequently aims at saving resources while avoiding producing to stock. On the other hand, the close customer relationship targets customer wishes, flexibility and responsiveness to individual needs and, therefore, reflects the need for customisation [4, 40]. The mentioned strategies will help companies maintain profitability whilst operating with smaller batch sizes and shorter lead times for customer benefit [14].

The technologies associated with decentralisation enable coordination and synchronisation of production and decision-making processes, which are required in non-hierarchical networks [41–44]. The production strategies mentioned above are most effective when typical production functions such as engineering, capacity planning, order management, manufacturing, and logistics are integrated comprehensively. DTI technologies allow the inclusion of network partners in all information flows and enable sharing of appropriate resources. When leveraging advanced technologies such as AI, machine learning and edge computing to enable decentralised decision-making and coordination of production processes, humans contribute their domain expertise, intuition, and creativity. They possess contextual knowledge and problem-solving abilities and can provide oversight, make strategic decisions and handle complex and non-routine tasks that require creativity, adaptability and critical thinking. AI, on the other hand, can assist humans by automating routine tasks, analysing large datasets, identifying patterns and anomalies and providing recommendations for optimised decision-making. DTI, thus, facilitates the integration of humans and AI through technologies like edge computing, where AI algorithms can be deployed on edge devices, and cloud computing, where AI models can be trained and deployed centrally. This allows for real-time decision-making at the edge and enables seamless coordination and synchronisation across the production network.

#### **Case Study: Production Scheduling Pilot**

In the context of production and supply chain management, coordination of processes and their consequences is crucial for maintaining robustness and resilience. The EU research project *knowlEdge - Towards Artificial Intelligence powered manufacturing services, processes, and products in an edge-to-cloudknowledge continuum for humans [in-the-loop]* [45] aims to unlock the full potential of AI in manufacturing by developing AI methods, systems and data management infrastructure, with a focus on edge-cloud computing and collaboration between humans and AI. In a production scheduling use case, the project optimises supply chain planning, demand forecasting and production batch optimisation in an industrial environment. DTI technologies are integral to the solution.

Real-time data from production facilities is collected using sensors and data interfaces at the edge, enabling monitoring, analysis and informed decisionmaking. Machine learning algorithms analyse historical and real-time data to predict demand, optimise production schedules and identify bottlenecks. Process simulation and digital twins are used to simulate and predict future scenarios, allowing proactive adjustments to production schedules.

The integration of AI and data analytics empowers employees to make informed decisions, reducing errors and saving time. The AI system continuously learns and improves, leveraging insights from documented deviations, errors and solutions. User-friendly interfaces facilitate human-centred interaction and visualisation, enabling monitoring, process adjustments and coordination at the edge. This holistic approach enhances production, facilitates proactive decision-making and improves coordination and collaboration between humans and AI agents.

Essentially, this points to smart and collaborative manufacturing, which involves collaborative planning, management and operation of networked production. The approach deliberately utilises the network as a resource for distributing tasks and acquiring means of production and could also be seen as in line with what Jovane et al. call a plug-and-produce network of production services [14]. This way, the network can respond to immediate challenges in a much better way than under a regime of hierarchical planning and management. Thus, the resulting inter-organisational integration of companies reflects the regime of networked production of the future. This becomes particularly obvious when considering the self-organisational and autonomous operations of DTI agents, which, based on predefined process targets, can easily transect organisational boundaries to carry out tasks. Consequently, in a resource constraint world, the focus on networked production and the potential to expand the network when opportunities arise, companies can leverage the strengths of different network partners and achieve a more robust and resilient production, whilst unlocking new opportunities for growth and innovation [46–48].

Following the DTI logic, future production networks will be following a selforganising and self-forming paradigm [22]. Rather than being pure manufacturers of goods, companies will be more and more providers of service, which solve a problem for a user of the manufactured product [49–51]. The shift towards a self-forming and decentralised supply chain requires the development of a conceptual framework for a manufacturing-as-a-service (MaaS) approach. This approach involves the provision of value-added services throughout the entire life cycle of a product, including planning, installation, operation, and alterations as well as the recirculation of used product and associated raw materials [52, 53].

In line with the principles of decentralisation and networked production, manufacturing capabilities and resources are provided as on-demand services to companies or individuals. Essentially, MaaS can be facilitated through the integration of technologies such as edge computing, cloud computing and IoT. These technologies enable the connection and coordination of manufacturing resources across a network of distributed nodes.

The decentralisation of activities allows companies to define and execute selfcontained service packages independently, supporting the flexibility and adaptability of the network [14]. This means that manufacturing tasks can be outsourced to specialised service providers who offer specific capabilities or expertise. These providers can be in different geographical locations, allowing for flexibility in production and leveraging the strengths of each partner. MaaS may include the integration and coordination of typical manufacturing processes, such as design and engineering, production planning, procurement, assembly, quality control and logistics. Additionally, DTI enables continuous monitoring and control of manufacturing operations, ensuring general visibility, heightened transparency and adaptability throughout the production process.

Additionally, there is a growing demand for "all-inclusive" solutions that incorporate both products and related services, which presents an opportunity for collaboration between larger and smaller companies to design and offer specialised services for new markets [54]. Maas in this context also enables smaller companies or individuals to access manufacturing capabilities that were traditionally available only to larger organisations.

#### **4 Building Blocks and Implementation Roadmap**

A technology roadmap is a strategic planning tool that helps organisations to align their technology development efforts with their business goals and objectives [55, 56]. It can also serve as a research agenda for continuous research. This way, the elements of the roadmap, building blocks, required intermediate steps and envisioned outcomes, can be further refined and adapted to new insights. Thus, through the roadmap the major capabilities, characteristics, and processes that are needed to create a successful DTI-enabled and networked production can be understood.

The roadmap for realising advanced networked production with the help of DTI is comprehensive and involves several building blocks, each with its vision, value promise and development pathway. To achieve the desired goals, it is important to have a clear pathway for each building block, divided into near-term (0–5 years), mid-term (5–10 years) and long-term (over 10 years) goals and performances (Fig. 2).

The first building block, *Universal Transparency*, aims to enable end-to-end transparency and accountability across value networks, generating added value through comprehensive information available for all involved. The goal is to enable end-to-end transparency and visibility in networked production. An increase in coordination can help to ensure that the different activities within the network are better aligned, which can lead to greater efficiency and better results [57, 58]. This is achieved using various technologies and tools that enable comprehensive availability of information and real-time monitoring of the network's performance. Universal transparency also promotes customer value by providing more visibility and transparency in the production process, which can help build trust [59]. By building trust, the different partners within the network can better collaborate and achieve their goals together. It also enables social sustainability by promoting accountability and compliance with legal requirements and ensuring that all participants in the network act ethically and responsibly [4]. The near-term goals include progress against sustainability and compliance goals, data management protocols, and information security protocols. Mid-term goals include data analysis for transparency goals involving x-tier suppliers, while the long-term goal is to achieve end-to-end transparency in value chains that extends beyond legal requirements.

*Cooperate to Compete* is another building block that involves knowledgebased continuous development and network innovation. It involves co-design, coengineering, and co-production of customised products and services. This building block generates added value by capturing the benefits of collaboration, such as sharing knowledge and resources [60]. The integration of product development into the network is especially important during the early phases of development when collaboration between different actors in the production network can have a decisive influence on competitive factors, such as reducing costs, shorten time-to-market, improve product quality, increase flexibility in production, promote innovation, and develop sustainable solutions [4]. Near-term goals include hybrid systems for knowledge-based engineering and process development, while mid-term goals include inter-factory organisation for enhanced collaborative design and engineering. Long-term goals involve advanced interfaces and methodologies for domain knowledge capturing and situation/context-dependent systems for engineering and process development across factory boundaries.

*Sustainable and Circular Operations* is a building block that focuses on closedloop cycles, greater efficiencies, and new ways of sourcing materials. The added value is particularly produced through resource efficiency, extended machine operation, and securing new resources. In terms of resource efficiency, the focus is on optimising the utilisation rate of machines and equipment, avoiding downtime and production disruptions and reducing waste and energy consumption [4, 14].

Sustainability-wise, the emphasis is on minimising the environmental impact of production processes by reducing waste and emissions, using alternative sources for raw materials, such as secondary sources, and considering circular economy principles for material and resource flows. Near-term goals include machinery and equipment to design and engineer products for a circular economy. Mid-term goals include hybrid systems for knowledge-based efficiency gains, and machinery and equipment designed and engineered for long service life. Long-term goals involve decentralised management for reduced risks and increased resilience, minimal manufacturing and maximal servicing paradigm for sustainability, and emphasis on secondary raw materials.

*Intelligent Control of Value Networks* involves decentralised, autonomous management of value networks, comprising various agents including human and artificial agents. This building block generates added value through better anticipation and agility. In a learning cognitive production network, assets in the network are interconnected and can self-optimise. Depending on the context, control is carried out automatically or autonomously within the collaboration between humans and artificial intelligence [61]. Predictions can be made based on data sources and amounts that would be hidden from humans alone [4]. Near-term goals include AI solutions applied in various parts of the network, while midterm goals involve integrated platforms and architectures for network-level AI. Long-term goals involve human and artificial agents working seamlessly together, and increased capacity for adaptation, anticipation, agility, and resilience due to decentralised management.

Finally, *Integration with (New) Network Partners* (or their dissolution of redundant relationships) involves on-the-fly integration, separation, and interoperation, federated architectures, and manufacturing-as-a-service [62]. This building block creates added value through overall network performance, operational efficiency, customer satisfaction, and enhanced products. Companies can expand their capabilities and access new markets, assets, technologies and resources. This leads to enhanced network performance and will also involve strong collaboration between large companies and smaller, niche manufacturers [14]. Near-term goals include continued digitalisation of supply chain planning, connecting internal production and planning systems, and digitalisation of product information. Mid-term goals involve integration of factories with suppliers and customers, while long-term goals involve autonomous transaction processes and restructuring, and manufacturing-asa-service and experience.

The roadmap for realising advanced networked production with the help of DTI is complex considering, for example, the variety of technologies involved and requires a long-term vision. In addition, it is important to recognise and exploit new business models and opportunities for creating value, while keeping in mind the viability of the proposed solutions from a business perspective.

#### **5 DTI Deployment from Business and Organisational Perspective**

The industrial evolution towards networked and self-controlling manufacturing concepts enables and generates groundwork for new business models, as well as sets new requirements for the skills and expertise of the personnel. In addition to the need to maximise profits, new challenges such as acting properly in the face of environmental damage cannot be ignored. It is therefore a matter of recognising and exploiting new opportunities for creating value throughout a product's life cycle. Due to its cross-company nature, networked manufacturing offers the appropriate conditions for this, for example, partnerships for knowledgebased and sustainable extraction of necessary resources, manufacturing and product development, and innovation development. At the same time, new business models must also recognise new forms of work and the development of skills, learning and education.

Digitalization of companies' and organisations' processes enables the separation of data and information from their physical/real operational environment. This in turn enables new ways to organise work within and between organisations, which also leads to new business model opportunities. In practice, the so-called cyberphysical systems can be implemented where physical processes are monitored by creating a virtual counterpart of the physical world in which decentralised decisions can be made by communicating and cooperating with each actor in real-time using the Internet and IoT components [63].

From an organisation point of view, the DTI vision brings about changes to the distribution of work between human employees and computer systems. More and more human processing can be automated to be done by smart ICT systems. This provides, on the one hand, an opportunity to increase the efficiency of the organisation, and on the other hand, develop new kinds of data-based services and new ways of connecting customers, partners, and suppliers. Furthermore, the skills and competencies of the personnel must be developed and adapted to meet the new requirements the DTI deployment entails. In general DTI increases the connectivity of the different stakeholders, which means that the actors need to broaden their perspective on their business environment to include partners that are further away from them in the value chain, e.g., they need to consider the operations of their customer's customer. More systemic thinking is required and skills and competences to realise it. An example of the business opportunities this enables is a condition based maintenance service concept, where a component or sub-system provider is connected to the equipment in use either directly or via OEM's systems to follow the product performance during its life cycle. All these connections, data availability and security issues need to be understood by all partners in the connected system. This requires enhancement of two kinds of competences: (1) competences to understand the technological system and its operation, and (2) competences to understand the operational environment of the end-product at end-user location.

In the digital world, the DTI envisions new services, such as platforms, that are required to enable the myriad of potential connections of digital data flows between different stakeholders. This digital world forms its market environment where new platform providers as well as traditional business ecosystem actors with different resources, competitive advantages and strategic aims are positioning themselves. The digital marketplace is much more flexible and dynamic than the traditional as digital relationships between actors are easier to establish and terminate. Digital platforms have a specific characteristic of aiming to orchestrate a large number of business actors, i.e., their business is based on connecting other actors. This challenges the traditional strategic planning of organisations rooted in the resource-based view of the firm, which focuses on achieving competitive advantage by owning and controlling resources [64]. These resources should have the so-called VRIN characteristics, i.e., be valuable, be rare, be inimitable, or be non-substitutable. Digital platform business models also challenge the positioning framework of strategic management [3], which focus on competitive advantage by raising structural barriers to entry and increasing bargaining power. The digital marketplace enables more dynamic competition, where, e.g., disruptive innovation from start-ups and entry by firms from different industry sectors can bring completely new competitive pressure to existing business models [65]. Instead of competition between firms, the competition is between business ecosystems consisting typically of one or two central (platform) companies and other firms providing complementary goods and services. In this kind of context, the value capture component of a business model should balance the focal firm's profits and the profitability of the ecosystem partners [65].

While creating new business opportunities the dynamic digital marketplace creates new challenges as well. Many new digital solutions and services are based on combining data from different sources, i.e., from different stakeholders. The knowledge and capabilities of different stakeholders are combined to deliver a specific offering or solution. This can create a challenging business landscape where the perspective on specific assets and complementary assets may be viewed differently by different stakeholders, especially between platform providers and linked service providers [64]. This can bring about risk, especially to the linked parties of the digital ecosystem around the platform. On the one hand, the platform provider may be enticed to integrate successful solutions provided by the linked parties into its offering. On the other hand, the platform operator may stop providing required data sources to the linked parties, thus destroying their digital business model. Naturally, the platform provider has to consider the risk these kinds of actions can cause to its business reputation [64].

Another challenge posed by the digital integration of businesses is the inclusion of the smaller actors in the value chains and business ecosystems, especially considering their digitalization skills and competencies [66]. SMEs are often an important part of a value chain in the manufacturing sector. Thus, to take full advantage of the DTI their integration is essential. Lack of resources and limited competencies are generic barriers for SMEs to implement new technologies. On the one hand, these barriers can be overcome by introducing scalable technology solutions that

are easier to be deployed by SMEs [66]. On the other hand, new services and service providers can be established to support the SMEs in digitalization, from understanding the systemic change it brings to their business to running specific technology services for them [67].

#### **6 Discussion**

This article discusses the use of Decentralised Technical Intelligence (DTI) to improve networked production. While the primary focus is on the network, the transition is built on a multilevel perspective that spans from technical processes on the factory floor to networking and business operations at the upper level. This requires the integration of various micro and macro levels in real-time through both company internal and external networks of information and communication technologies, necessitating international standards.

Consequently, it is essential to develop and integrate a multilevel architecture for control and management, which includes implementing decentralised intelligence through edge-cloud solutions. This integration allows for the effective combination of physical levels with ICT control structures.

This development implies that organisations involved in the networked production will integrate more and their roles may transform into functional specialisation or strategic competencies within the network. This integration leads to a synergistic effect among collaborating companies, resulting in significant efficiency gains and cost savings across different categories of costs. Achieving advanced networked production through DTI requires following the described roadmap and implementing the building blocks of DTI in a way that is transparent, trustworthy, cooperative, sustainable, and intelligent.

The figure (Fig. 3) presented in this article illustrates the approach's goal to surpass the current fragmented approaches that mainly involve incremental

**Fig. 3** Progress towards advanced networked production based on DTI (adapted from ManuFuture [24] and Sautter [22])

improvements. Despite witnessing progress within the different levels of the production system and the interconnections between these levels, disciplinary boundaries often limit us. With DTI, we can surpass these limitations by enabling autonomous activities of DTI agents, which include both humans and machines. This can lead to various benefits such as self-organisation, self-optimisation, and self-maintenance. All of these actions occur within a predefined performance space, with a focus on situational and contextual requirements.

Moreover, leveraging DTI technologies enables a shift away from strict hierarchies towards more flexible and adaptive operational management. With DTI, employees are empowered to make decisions based on their expertise and the realtime information provided by AI systems. This empowerment allows for quicker response times, as employees can take immediate action and adapt to unforeseen events without waiting for approval from higher levels of management.

Thus, DTI involves converging towards heterarchy, which refers to a nonhierarchical organisational structure where decision-making authority and control are distributed among various nodes or agents rather than being concentrated at the top of a hierarchical chain. Heterarchies emphasize collaboration, decentralised decision-making and empowered employees [68].

As a result, DTI facilitates a systemic and transcending approach that dissolves the boundaries of organisations and disciplines, allowing for a fundamental transition in production. DTI catalyses leapfrogging performance, as Sautter [22] describes it. DTI also provides a novel understanding of process quality, flexibility, and resilience, enabling the production system to adapt to changing situations.

From a business implementation perspective, it is essential to develop incentive systems that support the collaboration of the different stakeholders in the network. Each actor should have a clear business value proposition to engage in the activities DTI entails, i.e., collaboration, commitment to common goals, and sharing of knowledge and information. This may require, e.g., further development of pricing methods and mechanisms for knowledge and data.

#### **7 Conclusion**

DTI has wide-ranging implications for long-term development in industrial production. The concept has the potential to alter operative management and enhance flexibility, adaptability, robustness, and resilience in production networks.

Understanding the benefits will coincide with a surge in strategic management efforts, research, development and deployment of technologies, which provide an underpinning to the decentralisation of intelligence and agents' actions and responses.

DTI is an overarching visionary concept for the future of manufacturing. Implementing it is a long-term process that requires a mindset change from all the actors in manufacturing ecosystems. Changing all at once is not viable,

instead, incremental development steps in collaboration with relevant actors in the manufacturing networks should be taken.

The roadmap serves as a guiding framework that reflects the need for incremental development in the implementation of DTI and emphasises the importance of taking progressive steps forward while allowing for continuous refinement and adjustment. Stakeholders can align their efforts towards realising the long-term vision of enhanced flexibility, adaptability, robustness and resilience in production networks. This will position them at the forefront of the future of manufacturing.

**Acknowledgments** The research leading to these results has received funding from Horizon 2020, the European Union's Framework Programme for Research and Innovation (H2020/2014-2020) under grant agreement no. 957331 (www.knowlEdge-project.eu).

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Part III Trusted, Explainable and Human-Centered AI Systems**

## **Wearable Sensor-Based Human Activity Recognition for Worker Safety in Manufacturing Line**

**Sungho Suh, Vitor Fortes Rey, and Paul Lukowicz**

#### **1 Introduction**

The advent of Industry 4.0, introduced in 2011, aimed to revolutionize manufacturing by incorporating advanced technologies to achieve operational efficiency and productivity gains [24]. However, as the focus shifted from technology-driven advancements to a more human-centric approach, the concept of Operator 4.0 emerged. Operator 4.0 envisions a workforce assisted by systems that alleviate physical and mental stress while maintaining production objectives [14]. This shift in perspective laid the foundation for the development of Industry 5.0, a valuedriven manufacturing paradigm that places worker well-being at the forefront of the production process [30].

Industry 5.0 encompasses two main visions: one involving human–robot collaboration and the other centered around a bioeconomy utilizing renewable biological resources. As the researchers explore the potential of Industry 5.0, it becomes crucial to investigate how technology can support the industry while prioritizing worker safety and productivity. This necessitates meeting the human needs outlined in the Industrial Human Needs Pyramid, ranging from workplace safety to the fulfillment of human potential through a trustworthy relationship between humans and machines.

In this context, wearable sensor-based human activity recognition (HAR) emerges as a vital component of Industry 5.0. By continuously and unobtrusively monitoring workers' activities, wearable sensors enable a synergistic collaboration between humans and machines [23, 29]. This collaboration enhances productivity

German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany

S. Suh (-) · V. F. Rey · P. Lukowicz

Department of Computer Science, RPTU Kaiserslautern-Landau, Kaiserslautern, Germany e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
J. Soldatos (ed.), *Artificial Intelligence in Manufacturing*, https://doi.org/10.1007/978-3-031-46452-2\_17

while empowering workers to unleash critical thinking, creativity, and domain knowledge. Simultaneously, machines autonomously assist with repetitive tasks, reducing waste and costs.

To foster the development of trustworthy coevolutionary relationships between humans and machines, interfaces must consider the unique characteristics of employees and organizational goals. An example of such collaboration is evident in the use of cobots, where machines share physical space, perceive human presence, and perform tasks independently, simultaneously, sequentially, or in a supportive manner [8].

Within this framework, this book chapter focuses on the crucial intersection of wearable sensor-based HAR and Industry 5.0. Especially, we explore the transformative role of sensor-based HAR in promoting worker safety and optimizing productivity in manufacturing environments. We aim to study the intricacies of wearable sensor technologies, sensor data fusion techniques, and advanced machine learning algorithms to effectively capture and interpret workers' activities in real time. By utilizing wearable sensors, manufacturers can gain real-time insights into the physical movements and behaviors of workers, enabling them to identify potential safety hazards, proactively intervene in hazardous situations, and implement preventive measures.

Moreover, the applications of sensor-based HAR extend beyond worker safety. The data collected from wearable sensors can facilitate the optimization of manufacturing processes by identifying bottlenecks, streamlining workflows, and minimizing errors [29]. By analyzing workers' activities, manufacturers can uncover insights into the ergonomics of workstations, leading to improvements in the job design and a reduction in musculoskeletal disorders. Furthermore, the knowledge derived from sensor-based HAR can inform training programs, enabling targeted interventions and skill development to enhance worker efficiency and job satisfaction. Likewise, the integration of HAR using wearable sensors in manufacturing environments aligns with the growing interest in smart manufacturing and Industry 4.0 [23, 29]. In the manufacturing line, HAR can be utilized to quantify and evaluate worker performance [28], understand workers' operational behavior [1], and support workers' operations with industrial robots [22]. Activity recognition for worker safety in the manufacturing line is becoming increasingly important, as it provides the ability to quickly identify workers' needs for assistance or prevent industrial accidents.

In this book chapter, we study the intricacies of wearable sensor technologies and their integration into manufacturing environments. We explore various sensor modalities, including inertial sensors, motion sensors, and body capacitance sensors, and discuss their relevance in capturing a comprehensive view of workers' activities. We also examine sensor data fusion techniques to effectively integrate and interpret data from multiple wearable devices, enabling a holistic understanding of workers' actions. The outcomes of this research shed light on the transformative capabilities of wearable sensor technologies and open new avenues for future research in this field, aligning with the principles of the Industry 5.0 paradigm.

To demonstrate the practical application of sensor-based worker activity recognition, we present a use case in a smart factory testbed. In this use case, we deploy and test HAR using wearable sensors to predict workers' movement intentions and plan optimal routes for mobile robots in a collaborative environment. By anticipating workers' actions, collision risks between workers and robots can be minimized, ensuring both high production levels and worker safety.

To further improve the performance of HAR, we explore deep learning techniques such as adversarial learning [26, 27] and contrastive learning [9, 12]. These approaches have shown promise in enhancing activity recognition by leveraging additional information and improving the generalization capabilities of the models.

In addition to traditional wearable sensor modalities, we introduce the use of body capacitance sensing as an alternative modality for HAR. Body capacitance sensing captures the electric field feature between the body and the environment, providing unique insights into body movement and environmental variations [6]. By combining body capacitance sensing with inertial measurement unit (IMU) based activity recognition, we aim to extend the sensing capabilities and improve the accuracy of activity recognition systems.

Throughout this chapter, we showcase real-world applications of sensor-based worker activity recognition in manufacturing environments. From real-time monitoring of workers' movements to detecting unsafe actions and alerting workers and supervisors, the potential benefits are substantial. We also discuss the limitations and potential ethical considerations associated with wearable sensor systems, emphasizing the importance of privacy, data security, and worker consent.

In the following sections, we provide a comprehensive review of related works in Sect. 2. Section 3 presents the wearable sensor-based worker activity recognition in a manufacturing line with IMU sensors and body capacitance sensing module, including data fusion approaches and a use case in a smart factory testbed. Section 4 introduces deep learning techniques for improving the performance of human activity recognition, such as adversarial learning and contrastive learning. Finally, Sect. 5 concludes this book chapter.

#### **2 Background**

The Internet of Things (IoT) has revolutionized various industries, including manufacturing, by enabling the integration of physical devices and digital systems. In the context of smart factories, IoT technologies play a crucial role in creating intelligent and automated production environments. With the advancement of sensing technologies, it has become increasingly feasible to recognize human activities using various sensors such as IMU sensors [26], microphones [5], cameras [20], and magnetic sensors [19]. These sensors capture valuable data about human movements, interactions, and environmental factors, which can be utilized to enhance productivity, safety, and efficiency in smart factory environments.

Supervised machine learning techniques have been widely used for recognizing activities using labeled training data. However, collecting labeled data can be time-consuming and labor-intensive. To overcome this challenge, researchers have explored methods that reduce the effort required for data collection. Among the popular approaches are transfer learning, activity modeling, and clustering techniques. These methods leverage existing knowledge or unsupervised learning to recognize activities with minimal labeled data.

The concept of smart manufacturing, often associated with Industry 4.0, has gained significant attention in recent years [21]. Researchers have focused on recognizing and supporting factory work using sensor technologies [2, 3]. In the manufacturing domain, the use of sensors for activity recognition has been extensively explored. For example, Koskimaki et al. [15] utilized a wrist-worn IMU sensor and a K-Nearest Neighbors model to classify activities in industrial assembly lines. Maekawa et al. [17] proposed an unsupervised method for lead time estimation of factory work using signals from a smartwatch with an IMU sensor. These studies demonstrate the potential of sensors in recognizing worker activities in manufacturing environments, particularly with the application of machine learning techniques.

Sensor-based HAR is a crucial aspect of wearable technology. The IMU has been the dominant sensor in wearable devices, providing motion-sensing capabilities. However, the IMU's ability is limited to capturing the wearer's movement patterns and does not account for body–environment and body–machine interactions, which play critical roles in security and safety in manufacturing environments. To extend the motion-sensing ability of wearables, the researchers have explored alternative sensing sources, such as the body capacitance [6]. The body capacitance describes the electric field between the body and the environment, and variations in this field can provide valuable information for pattern recognition. For example, capacitive sensing has been used to detect touch patterns between human fingers and different objects. This alternative sensing approach benefits from low cost and low power consumption and extends the sensing ability beyond IMU-based activity recognition.

Machine learning models play a crucial role in wearable activity recognition, enabling the extraction of meaningful patterns and features from sensor data [13]. Classical machine learning methods extract handcrafted features from sensor data, such as time-domain and frequency-domain features [16]. These methods rely on expert knowledge and domain-specific feature engineering. In recent years, deep learning-based methods have gained significant attention in activity recognition [18]. Convolutional neural networks (CNNs), recurrent neural networks (RNNs), and hybrid models have been developed to capture temporal correlations and learn sensor representations for improved activity recognition accuracy. Additionally, multitask learning and generative adversarial learning have been introduced to address different data distribution problems and enhance recognition performance [4, 7]. These advancements in machine learning techniques have paved the way for more accurate and robust activity recognition in wearable devices.

To bridge the gap between wearable sensor-based HAR and the principles of Industry 5.0, this book chapter offers unique contributions and addresses existing research gaps. It presents a comprehensive case study on wearable sensorbased worker activity recognition in a manufacturing line with a mobile robot. The investigation includes the exploration and comparison of sensor data fusion approaches using neural network models to effectively handle the multimodal sensor data obtained from wearable devices. Additionally, several deep learningbased techniques are introduced to enhance the performance of human activity recognition. By harnessing the potential of wearable sensors for human activity recognition, this chapter provides valuable insights into improving worker safety on the manufacturing line, aligning with the principles of the Industry 5.0 paradigm. The outcomes of this research shed light on the transformative capabilities of wearable sensor technologies and open new avenues for future research in this field.

#### **3 Wearable Sensor-Based Worker Activity Recognition in Manufacturing Line**

#### *3.1 Use Case at the SmartFactory Testbed*

The SmartFactory Testbed, developed by the Technology Initiative SmartFactory KL, is a collaborative effort between the Department of Machine Tools and Controls (WSKL) at the TU Kaiserslautern and the Innovative Factory Systems (IFS) research unit at the German Research Center for Artificial Intelligence (DFKI). This non-profit organization focuses on advancing manufacturing technologies with industry specialists known as factory innovators.

The SmartFactory Testbed offers unique manufacturer-independent demonstrations that enable the development, testing, and deployment of innovative ICT technologies in a realistic industrial production environment. It serves as a vital test environment, particularly for the European GAIA-X subproject SmartMA-X, where flexible production systems can be arranged and integrated in highly customized configurations, enhancing the dynamism of the industrial environment.

In this chapter, we utilize the SmartFactory Testbed as a real-world deployment setting to evaluate and validate the Human Action Recognition module within a specific use case referred to as "Human Action Recognition and Prediction in the Respective Environment." The use case focuses on a worker's activity pipeline, including their presence in the pilot area and collaboration with different modules and robots across 20 diverse activities. The accurate prediction of human actions within the production lines is crucial, especially when workers interact with moving robots or when robots generate paths to avoid potential collisions in the layout. By leveraging information about the worker's location and their next anticipated action, the robot's movement path can be manipulated to minimize the risk of collisions. This proactive approach ensures the safety and reliability of the industrial environment, particularly when humans are present.

#### *3.2 Data Acquisition*

In this chapter, we conducted an experiment involving 12 volunteers, from diverse cultural backgrounds and genders, who wore a wearable sensing prototype that we designed, as well as an Apple Watch, currently the top-selling smartwatch [10], while performing various tasks that simulated typical worker scenarios during their daily work. These tasks included opening and closing doors, walking, checking parts inside a module, and interacting with a touch screen. To ensure robust results, the experiment consisted of five sessions, each lasting between 2 and 3 minutes. Some sessions were conducted and recorded in a different direction from the flow chart. Prior to participation, all volunteers signed an agreement in compliance with the university's committee for the protection of human subjects policies. The experiment was video recorded to enable further confidential analysis, including ground truth activity annotation. Both the observer and the participants adhered to an ethical and hygienic protocol in accordance with public health guidelines.

Figure 1 illustrates the wearable sensors attached to the participants. The prototype sensors were placed on both wrists while the Apple Watches were attached to both wrists, and an iPhone mini was attached to the left arm. The

**Fig. 1** The wearable sensors attached to the participants. (**a**) The wearable sensing prototype we designed on both wrists and (**b**) Apple Watches on both wrists and the iPhone mini on the left arm

**Table 1** Comparison of the data distribution of activities in the prototype sensor and Apple Watch datasets

collected data from the wearable sensing prototype consisted of 10 channels per sensor, resulting in a total of 20 channels, including three channels of acceleration, three channels of gyroscope, three channels of magnetometer data, and one channel of body capacitance data. The data collected using the Apple Watch and iPhone mini comprised 9 channels per sensor, totaling 27 channels, including 3 channels of acceleration, 3 channels of gyroscope, and 3 channels of magnetometer data. To synchronize the recorded video data, left wrist sensor data, and right wrist sensor data, we performed five claps at the start and end of each session. Based on the video data, we manually annotated the users' activities, resulting in 12 different activities, including a Null class. The prototype sensors provided IMU sensor data and body capacitance data with a sampling rate of 25 Hz, while the Apple Watch only provided IMU sensor data with a sampling rate of 100 Hz. This experiment allowed us to assess the performance of the sensor hardware and collect a dataset suitable for developing and testing algorithms for human intention recognition.

Table 1 presents the data distribution of activities in the prototype sensor and Apple Watch datasets. The prototype sensor dataset involved data collection for a total duration of 142 minutes and 24 seconds, while the Apple Watch dataset had a total duration of 170 minutes and 3 seconds. To focus on preventing collisions between workers and mobile robots, we decided to collect additional sensor data from the Apple Watches for the walking class after collecting data from the prototype sensor. As a result, the Apple Watch dataset had a higher proportion of walking class data compared to the prototype sensor dataset.

**Table 2** Comparison results between Apple Watches and the prototype sensing module combining IMU and body capacitance sensors in terms of the testing accuracy and macro F1 score. The numbers are expressed in percent and represented as *mean* ± *std*

To evaluate the prototype sensor hardware and neural network models for worker activity recognition, we annotated the user's activities based on the workflow of the use case at the SmartFactory Testbed. The workflow classified the sensor data into 12 activities, including Null, opening/closing doors, checking machines, walking, pressing buttons, and placing back keys.

In our application, each instance is a sliding window of sensor data. For the Apple Watch data, we used a window of length 100 (1 second) and a step size of 4 (0.04 seconds). For the prototype sensor data, we employed a window of length 25 (1 second) and a step size of 1 (0.04 seconds). That is, in both cases, we use data from 1 second, with a step size of 0.04 seconds, which maximizes the number of windows while keeping the step size suitable in both sensor scenarios.

#### *3.3 Worker Activity Recognition Results*

To process the collected sensor data, we used neural networks based on convolutional neural networks (CNNs) [25]. For training and validation of the neural network models, we employed a leave-one-session-out scheme. In each fold, one session was allocated for testing, another for validation, and the remaining three sessions for training.

Table 2 presents the comparison results between Apple Watches and the prototype sensing module, which combined IMU and body capacitance sensors, for the worker's activity recognition at the smart factory testbed. The evaluation metrics used include accuracy, macro F1 score [26], and the accuracy of the walking class, which is critical for preventing potential collisions between workers and mobile robots. The results demonstrate that the prototype-sensing module outperformed the Apple Watch in terms of accuracy, macro F1 score, and walking class accuracy.

#### **4 Deep Learning Techniques for Human Activity Recognition Improvement**

In this section, we focus on exploring advanced deep learning techniques to improve the performance of worker activity recognition in the manufacturing line for worker safety, building upon the findings presented in the previous section. Two specific

**Fig. 2** Challenges in activity recognition: accounting for diverse behavior patterns across individuals

techniques, namely adversarial learning and contrastive learning, are discussed in detail. These techniques offer innovative approaches to improve the accuracy and robustness of the recognition system, addressing the challenges associated with complex and dynamic industrial environments [9, 27].

#### *4.1 Adversarial Learning*

Despite the successful digitalization of worker activities through wearable sensors and their recognition by simple CNN models, achieving generalization to unseen workers remains a significant challenge. Numerous studies have demonstrated that individuals perform the same activities in different ways, posing a challenge for activity recognition, despite the feasibility of user recognition [26], as illustrated in Fig. 2. This discrepancy becomes evident when evaluating performance by leaving out subjects rather than leaving out sessions.

To deal with this problem, we present an adversarial learning-based method for user-invariant HAR in this subsection, as illustrated in Fig. 3. Inspired by generative adversarial networks (GANs) [11], adversarial learning has been introduced as a technique to enhance the model's ability to discriminate between different worker activities and improve generalization. This technique, described in [26] and [27], employs four independent networks: a feature extractor, a reconstructor, an activity classifier, and a subject discriminator. The feature extractor maps sensor data to a common embedding feature space, while the reconstructor reconstructs the original signal from the embedding features. The activity classifier predicts activity labels based on the embedding features, and the subject discriminator differentiates between subjects based on the embedding features. The feature extractor and subject discriminator are trained by adversarial learning, with the subject discriminator aiming to distinguish subjects and the feature extractor aiming to deceive the subject discriminator. The method also incorporates a reconstruction loss to minimize the

**Fig. 3** The overall framework of the user-invariant HAR method using adversarial learning

difference between the original and reconstructed signals, along with a classification loss to train the feature extractor and activity classifier using activity labels. Additionally, the proposed method employs maximum mean discrepancy (MMD) regularization to align the distributions among source and target subjects, thereby enhancing the generalization of the embedding feature representation.

The user-invariant HAR method has demonstrated improvements of up to 7% in accuracy and 28% in macro F1 score compared to the baseline using the CNN model [26]. By using this adversarial learning technique, we expect to improve worker activity recognition in the manufacturing line for worker safety. For further details on the methods and experiments, please refer to [26] and [27].

#### *4.2 Contrastive Learning*

We tackle a common challenge in wearable HAR, where sensor locations that are commonly used for everyday wear provide inadequate information for accurate activity recognition. For instance, in our manufacturing work scenario, we found that sensors placed on the wrist and arm are not optimal for recognizing activities like walking, which would be better detected with an IMU sensor placed on the leg. This is a well-known limitation in HAR, where sensors deployed for long-term everyday use often result in poor or noisy data for the intended application.

To address this issue, we propose a method outlined in our work [9] that aims to improve the representation of the deployed sensors during training. The idea is to leverage additional sensors that are available only during the training phase to build a better representation of the target (deployed) sensor. This approach allows us to capture more relevant information about the activities being recognized, even if the

**Fig. 4** Step 1: Training representations with paired sensor data by contrastive learning

target sensor alone may not provide sufficient information, we can improve it by guiding its representation with through contrastive learning with the source sensor.

In our proposed method, we collect temporally paired data from both the source and target sensors. The source sensor refers to a sensor that is available during training but not during deployment, while the target sensor is the sensor that will be deployed for activity recognition in real-world scenarios. Through contrastive learning, we learn a mapping between the representations of the source and target sensors. This process enables us to capture the relationship and similarities between the activities observed by the source and target sensors, enhancing the representation of the target sensor.

The training of representations and the mapping between them are depicted in Fig. 4. Each sensor's data are processed separately through deep neural network encoders to obtain their respective representations. The translation between representations is facilitated by translation networks, which learn to transform the source sensor representation to align with the target sensor representation. This contrastive learning step can be performed using unlabeled data, where the network learns to align the representations based on the temporal correspondence between the sensors.

Once the representations are learned, we proceed to the next step using labeled data from both sensors. In this step, we train our activity classifier using data either from the target sensor in its learned representation or from the source sensor by translating it to the target representation using the translation network. This joint training process allows the classifier to learn to recognize activities based on the enhanced representations from both sensors. The overall training process is illustrated in Fig. 5. For evaluation, we utilize only the target sensor data, as it represents the real-world deployment scenario. By applying the learned representations and the trained classifier to the target sensor data, we can more

**Fig. 5** Step 2: Training representations and classifier by minimizing classification loss

**Fig. 6** Step 3: Testing with data of the target sensor only in order to evaluate the method

accurately recognize and classify activities in practical settings. The evaluation process is illustrated in Fig. 6.

We evaluated our method on two benchmark datasets for HAR: PAMAP2 and Opportunity. The results demonstrated significant improvements in activity recognition performance. We achieved an average macro F1 score increase ranging from 5 to 13 percentage points across all activities compared to traditional approaches that only rely on the target sensor. Notably, in specific scenarios where the source sensor provides highly informative data compared to the target sensor (e.g., recognizing walking with an ankle sensor as the source and a wrist sensor as the target), we observed even greater improvements, reaching up to 20 to 40 percentage points for certain activity classes.

This method using contrastive learning has important implications for real-world applications beyond IMU sensors. By using contrastive learning and knowledge transfer between sensors, our approach enables the development of more robust and accurate HAR systems. This has potential applications in various domains, including human–robot collaboration in manufacturing lines, where HAR plays a crucial role in improving productivity, quality assurance, and worker safety.

#### **5 Conclusion**

In this book chapter, we explored the role of wearable sensor-based HAR in promoting worker safety and optimizing productivity in manufacturing environments. We discussed the importance of worker safety and the potential benefits of using wearable sensors to monitor and recognize workers' activities on the manufacturing line. We presented a case study on wearable sensor-based worker activity recognition in a manufacturing line with a mobile robot, using sensor data fusion approaches and neural network models. By combining data from different sensor modalities, such as inertial sensors and body capacitance sensors, we were able to capture a comprehensive view of workers' activities and improve the accuracy of activity recognition. Furthermore, we introduced several deep learning-based techniques to enhance the performance of HAR, including adversarial learning and contrastive learning. These approaches have shown promise in improving activity recognition accuracy and generalization capabilities.

The use case in the SmartFactory Testbed demonstrated the practical application of sensor-based worker activity recognition in a real-world manufacturing environment. By accurately recognizing workers' activities and predicting their movement intentions, collision risks between workers and robots can be minimized, ensuring both worker safety and high production levels. Throughout this chapter, we discussed the potential benefits of sensor-based HAR beyond worker safety, including process optimization, ergonomics improvement, and worker training. We also highlighted the challenges and limitations associated with wearable sensor systems, such as privacy concerns and data security.

While our studies demonstrate the potential of wearable sensor-based HAR in manufacturing environments, we acknowledge that there are still challenges to overcome. These challenges include ensuring the robustness and reliability of sensor data, addressing issues related to real-time processing and inference, and managing privacy concerns in the collection and storage of personal data.

The results obtained from the adopted research can be applied in the practice of manufacturing companies by providing real-time insights into workers' activities. By accurately recognizing workers' activities and predicting their movement intentions, manufacturers can identify safety hazards, optimize processes, and enhance overall operational efficiency. The implementation of wearable sensor-based HAR systems can lead to improved worker safety, reduced workplace accidents, enhanced productivity, and more efficient resource allocation.

In conclusion, wearable sensor-based HAR holds significant potential for improving worker safety and productivity in manufacturing environments. By leveraging the wealth of data collected from wearable sensors and utilizing advanced machine learning techniques, manufacturers can gain real-time insights into workers' activities, identify safety hazards, optimize processes, and enhance overall operational efficiency. Future research in this field should focus on addressing the challenges and limitations of wearable sensor systems and exploring novel sensing modalities and machine learning approaches to further improve the performance of HAR in manufacturing environments.

**Acknowledgments** This work was supported by the European Union's Horizon 2020 program projects STAR under grant agreement number H2020-956573. Carl-Zeiss Stiftung also funded it under the Sustainable Embedded AI project (P2021-02-009).

#### **References**

2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 1088– 1099 (2016)

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Object Detection for Human–Robot Interaction and Worker Assistance Systems**

**Hooman Tavakoli, Sungho Suh, Snehal Walunj, Parsha Pahlevannejad, Christiane Plociennik, and Martin Ruskowski**

#### **1 Introduction: Why Object Detection in the Industrial Environment is Helpful?**

The integration of Object Detection (OD) technology into industrial environments offers significant changes and improvements by addressing critical challenges and optimizing operations. By accurately detecting and recognizing objects in real time, OD systems play an important role in ensuring safety, streamlining workflows, and efficient assistance for humans. In today's complex industrial landscape, accurate OD is pivotal as it serves as the foundation for various safety mechanisms, such as identifying obstacles and hazardous materials, thereby reducing accidents and downtime.

Efficiency, productivity, and worker safety are paramount in industrial settings, making accurate object detection an essential component of worker assistance systems. Leveraging advanced algorithms, sensor fusion techniques, and machine learning methodologies, industries can achieve improved automation, enhanced human–robot interaction, and optimized processes. OD technology enables indus-

S. Suh

German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany

H. Tavakoli (-) · S. Walunj · P. Pahlevannejad · C. Plociennik · M. Ruskowski German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany

Technologie-Initiative SmartFactory, Kaiserslautern, Germany e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Department of Computer Science, RPTU Kaiserslautern-Landau, Kaiserslautern, Germany e-mail<EMAIL_ADDRESS>
trial modules and agents to perceive and analyze their surroundings, leading to optimized operations and safeguarding the well-being of workers.

By integrating OD technology with human assistance and collaboration mechanisms, safer and more productive interactions are fostered in industrial environments. Collaborative robots, or Cobots, are increasingly deployed to augment the capabilities of human operators and improve overall productivity. Leveraging object detection technology, Cobots accurately perceive and respond to the presence of humans, ensuring safe and seamless cooperation within shared workspaces. This prioritizes safety and productivity in human–robot interactions, driving innovation and efficiency in the industrial domain.

Assistance systems play an important role where manual work is prevalent. In industrial processes such as assembly, training, or maintenance processes, these systems support minimizing the workload of humans. There are different types of assistance systems, ranging from manual workstations equipped with cameras and displays to immersive assistance systems over head-mounted devices or smart devices. Computer vision techniques such as object detection help understand the worker's environment from visual data. This understanding can be used to enrich existing software systems with object information to achieve multiple goals. With applications ranging from healthcare to the automotive industry, object detection models such as Yolo [22] and Faster RCNN [7] have gained popularity on account of their real-time detection performance. For example, in [15], an Advanced Driver Assistance System (ADAS) is equipped with real-time object detection to provide safety and a better driving experience. In use cases such as ADAS systems encountering moving objects, real-time object detection becomes even more important. Object detection also finds applications for solving industrial problems such as quality inspection.

An Augmented Reality (AR)-based assistance system uses the context of the existing reality or real environment and intends to augment it with useful information. In order to capture the real-world context, camera sensors play an integral part in the AR system. They provide real-world data, and the display serves as the counterpart helping the user visualize the system together with augmented information. The input visual data are understood using deep learning methods such as object detection and pose estimation, the output of which can be used for solving a large spectrum of problems in the form of an assistance system.

Object detection outputs can be utilized to create effective assistance systems that provide real-time recommendations and guidance. By integrating OD with headworn devices, workers can receive valuable information and instructions related to their tasks. For instance, an OD system can detect and recognize objects in the assembly place, or potential hazards in the worker's surroundings. The system can then analyze these data and provide recommendations, such as the next step in the assembly pipeline. It can also alert the worker to the presence of a hazardous object. These recommendations are displayed on the worker's head-worn device, providing immediate and personalized assistance. By leveraging OD in human assistance systems, workers can benefit from increased safety, improved efficiency, and enhanced situational awareness, ultimately leading to a more productive and secure work environment.

Supervised learning approaches such as object detection are highly dependent on data. A large amount of image data are required for training. Synthetically generated and labeled dataset offers several advantages in this regard. It simplifies and economizes the generation of large datasets, eliminating manual labeling and reducing human errors. Its flexibility allows for easy manipulation according to specific requirements, enabling researchers to control factors such as lighting, camera angles, and object placements. This enhances the model's ability to learn from diverse scenarios and improves its generalization capabilities.

Furthermore, synthetic data are well-suited for various computer vision tasks, including the detection of small or rare objects that may be challenging to capture in real-world datasets. Its ability to simulate objects of any scale or size makes it invaluable in training models. Synthetic data can be generated in large quantities, providing ample training samples for deep learning models that require extensive labeled data. Moreover, it allows for easy augmentation, increasing dataset diversity and variability. These advantages contribute to the development of more accurate and robust deep learning models in computer vision.

In the subsequent sections, we will delve into the following aspects:

#### **2 Background**

The industrial environment is undergoing a profound transformation toward greater intelligence and autonomy, thanks to the emergence of machine learning (ML) approaches. With access to abundant data and powerful hardware resources, deep learning techniques and artificial intelligence methods have become increasingly valuable. In this context, leveraging AI-based methods within industrial settings offers significant potential for reducing human errors and enhancing safety [4], particularly in scenarios involving human–robot collaboration or shared workspaces with close proximity between these two crucial agents. The exploration of human– robot interaction in complex and unpredictable environments has become a prominent research area, where AI methods can be effectively harnessed [13]. Within this landscape, deep models serve as powerful tools that excel in addressing intricate challenges encountered in the industrial domain. Their ability to learn hierarchical representations directly from raw data positions them as ideal solutions for a wide range of applications, encompassing object detection, anomaly detection, predictive maintenance, quality control, and optimization.

Deep models, capable of addressing diverse challenges encountered in industrial settings, have emerged as highly effective tools. These encompass vision-based approaches [25], natural language processing (NLP) techniques [1], as well as human wearable sensor-based approaches for HRI [16], among others. Notably, both conventional and deep-learning-based ML models have recently been harnessed for video streaming analysis with the aim of object detection [6]. Deep models, characterized by their end-to-end nature, aim to address the challenge of laborious and time-consuming feature extraction from data [11].

Vision-based AI approaches have become increasingly valuable in addressing industrial problems due to their ability to interpret and analyze visual data in real time. These approaches leverage deep learning algorithms and computer vision techniques to process images or video streams captured by cameras or sensors installed in industrial settings. By employing vision-based AI, various industrial challenges can be effectively tackled. Hence, vision-based approaches are widely recognized for their significant value across diverse scenarios. One prominent vision-based approach is object detection, which involves the classification and localization of multiple objects in the target frames or images. It has proven to be highly applicable in various scenarios, contributing to tasks such as improving safety, facilitating human–robot interaction [25], aiding in error identification for workers, and optimizing task completion time.

Given the wide range of challenges and numerous common use cases encountered by vision-based AI approaches in industrial settings, it is imperative to thoroughly investigate various aspects of AI-based approaches, especially in industrial environments. These aspects encompass examining the architecture of deep learning models, conducting comprehensive analysis and assessment of the data sources, and addressing other significant factors to ensure optimal performance and effectiveness. In the following, we dig into some of these challenges and solutions.

#### *2.1 Dataset*

In the case of industrial scenarios, there is difficulty in terms of real-world data collection. Especially in vision problems, collecting images with diverse conditions and viewpoints, and also labeling them, is an effort-intensive as well as a timeconsuming task. In some cases, the frames used as the source of input can originate from various cameras within the environment. These cameras can include ceilingmounted cameras, robot cameras, as well as head-worn cameras of mixed reality devices like HoloLens that capture workers' point of view [24]. Also in some cases, the data collection cameras are different from the edge devices on which an object detection model needs to be deployed. In such a situation, synthetic data generation becomes inevitable. Also, CAD data play an eminent role in the complete product life cycle of a manufacturing product, and for the product as well as the machine, it is readily available. These CAD models can be exploited to generate synthetic datasets. However, using CAD data directly cannot serve as a reliable solution. The CAD data only resemble the real objects in geometry; however, they lack materials and texture. Thus, there exists a large amount of distinction in the appearance of the real and CAD data. If we use CAD-based synthetic images for training, which is to be tested on real-world objects, there is a problem of domain difference, i.e., the real and synthetic data domain. In an attempt to solve this issue, the technique of domain randomization is demonstrated in [19]. Domain randomization can be achieved by randomizing various aspects of the simulation scene such as the backgrounds, illumination, orientation of the objects, etc. [3].

#### *2.2 Architectures*

Numerous prominent object detection architectures, including the R-CNN pipeline [7], Fast-RCNN [8], and Faster-RCNN [9], SSD [12], You Only Look Once (YOLO) approaches [22], and its following versions such as YOLOv7 [28], provide well-defined pipelines for detecting objects across different scenarios. These architectures offer robust methodologies for accurately identifying and localizing objects within the visual data.  This capability is of utmost importance in industrial environments, where the ability to detect objects promptly is crucial for effective use cases. By leveraging these advanced object detection methods, industries can enhance their operational efficiency, safety, and decision-making processes.

#### *2.3 Application in Industrial Environment*

Object detection is highly practical and beneficial in various broad use cases. For instance, in [21], object detection pipelines were employed to automate logistic processes within industrial environments. Saeed et al. [23] addressed the challenge of detecting faults in industrial product images, particularly focusing on smallobject detection. Another case study is conducted by Usamentiaga et al. [27] for evaluating the state-of-the-art deep-based object detection models as well as semantic segmentation in the scenario of automated surface inspection in metals. In the field of robotics, [2] researched the integration of an object detection CNN-based model to leverage a robot in a sorting task.

#### *2.4 Challenges*

It is obvious that object detection in the industrial environment can facilitate many tasks especially in which a robot and a human need to collaborate and work in a shared workplace, and promotes safety by facilitating interaction between humans and robots. However, object detection in the industrial environment does face some challenges. Object detection in a complex and unpredictable industrial environment, in which interference objects similar to the goal objects can be found easily due to the similarity in shape, size, and color, and random positioning and orientation of different objects make this detection much more difficult [2]. Also, detecting small objects poses a significant challenge due to their limited representation of features.

An overview of concepts and terminology related to artificial intelligence (AI) is given in ISO/IEC 22989:2022 Information Technology—Artificial intelligence— Artificial intelligence concepts and terminology" [10].

#### **3 Scenarios**

In this section, we explore the scenarios in both projects, STAR and InCoRAP for object detection in the factory environment with specific emphasis on safety in human–robot collaboration and interaction, and human assistance systems. We study the use cases in which object detection can be utilized for safety in the industrial environment. Additionally, we investigate the application of object detection in human assistance systems. Furthermore, we will explain our recent work on context-based object detection methods, particularly for small objects in the assembly use case within the industrial environment. We dive into the details of how context-based approaches can effectively detect and identify small objects in assembly scenarios.

#### *3.1 Object Detection for Human–Robot Interaction in the STAR Project*

Object detection for human detection can play a crucial role in enhancing safety in industrial environments. Although human detection can be considered as a subtask of object detection, it is facing some more complexity due to the wide range of possible appearances on account of the articulated pose, and clothing, to name a few [20]. Hence, studying human detection is vital from a safety perspective, especially in a complex, unpredictable, and dangerous industrial environment. Here are a few ways human detection will be utilized:

– Enhancing Worker Safety: Implementing object detection systems that are capable of accurately identifying and tracking human presence can greatly contribute to worker safety. These systems enable the implementation of proactive safety measures to prevent accidents or potentially dangerous situations. For instance, if a worker enters a restricted area or approaches a hazardous machine, the object detection system can promptly detect their presence and trigger warnings that can be raised in the HoloLens head-worn device or automatically shut down the equipment to mitigate potential accidents.

– Collision Avoidance: Object detection can be utilized to detect the presence of humans in the vicinity of moving machinery or vehicles. This information can be used to alert operators or autonomous systems to slow down, change direction, or stop to avoid collisions and ensure worker safety.

Our pilot study aims to explore the practical application of ceiling cameras or robot cameras in hazardous accident scenarios occurring within an industrial environment. Our focus is specifically directed toward examining the collaboration between a moving robot and human workers. The moving robot's primary role is to assist in the transportation of objects from the warehouse to the workstation, as well as facilitating the transfer of objects between various production lines, which often occurs in an unpredictable timescale.

To ensure the safety of workers and prevent any potential accidents, we employ localization and classification from object detection techniques for both the robot and human agents. These techniques enable us to effectively localize the position of the robots and humans within the workspace through the ceiling cameras in the environment. By real-time monitoring and analyzing of the robot and human locations, we can promptly identify and mitigate potential collisions or unsafe situations, which leads to a secure and accident-free working environment.

Overall, object detection for human detection in industrial environments brings safety by enabling proactive measures and collision avoidance. It helps create a safer working environment, reduces the risk of accidents, and enhances the well-being of workers.

#### *3.2 Object Detection for Manual Assembly Assistance System in InCoRAP*

In the InCoRAP use case, the AR-based Assistance system observes the egocentric point of view of the worker. The goal is to observe worker activity in the assembly process in order to support them through a mobile robot collaboration. The object detection model is a part of the assistance used to observe the assembly state based on the detected objects. For AR applications, the head-mounted device: HoloLens2 is used. The detected objects are the observations that correspond to the assembly steps, and these observations are later used to support the worker.

Research focusing on the evaluation of AR systems has proven AR-based assistance systems advantageous over conventional instruction manuals [5]. Thus,

**Fig. 1** The pipeline of the 3 experiments within the training phase with testing on the HoloLens2 frames transferred to the edge server for detecting target objects [25]

AR-based assistance places less workload on the worker or user compared to document-based guidance.

The field of small-object detection has garnered considerable research interest and has become increasingly popular. In vision-based object detection methods, the texture and arrangement of objects play a pivotal role in enabling the object detection pipeline to extract relevant features. It is worth noting that smaller objects may pose a challenge, as their size can be reduced significantly during the feature extraction process. For example, an object with dimensions of 32 × 32 pixels, after passing through five pooling layers in the VGG16 model, would be represented as a mere 1-pixel [17]. The concept of small objects, as defined in [26], encompasses objects sized at 32 × 32 pixels within the context of image analysis.

We specifically focus on small-object detection within an assembly scenario, where workers assemble various electrical components (e.g., buttons, resistors, LED, buzzers) on a breadboard to create a final product (Fig. 3b) [25]. In this scenario, a robot assists the worker by delivering parts from the warehouse, and an assistance system detects the current assembly steps and suggests the next probable part to be installed on the breadboard. The frames captured from the worker's point of view (POV) are seized using the HoloLens2, which the worker wears during the assembly process. The pipeline for this approach is illustrated in Fig. 1. In the testing phase, the frames are transmitted to the server where the object detection model performs inference. The detected objects provide information such as class identification and bounding box coordinates. This information is then communicated from the server to the HoloLens2 device using Unity communication. Subsequently, the HoloLens2 device generates holographic representations, displaying the class identification and bounding box information as augmented data. Workers in the environment can observe these holograms when they focus their gaze on the respective objects.

**Fig. 2** Synthetic dataset generation pipeline [25]

#### *3.3 Methodology: Context-Based Two-Step Object Detection*

For computer vision problems such as image classification or object detection, the foremost task is the collection of suitable image data and then labeling it. However, this process can be automated using a game engine such as Unity [3]. In a Unity renderer space, there is a game camera that consists of the attributes of a physical camera. There are two types of views obtained in such a game development environment, one being the developer view and the other being the game view. These features allow for creating a simulation scene for synthetic image data generation. As introduced in [3], the Unity Perception allows for customization and user-defined feature development with the goal of synthetic data generation in Unity. We developed a scene similar to that of the Unity Perception Package that allows for multi-object detection dataset generation. The 3D models of the desired object are CAD models converted into Unity-compatible format and imported into the scene.

The synthetic dataset generation pipeline shown in Fig. 2 shows the CAD model of the object imported into a Unity scene. On importing into the scene, the CAD models are processed to achieve a photo-realistic appearance using the renderer features. The scene is developed in such a way that it simulates a systematic imagecapturing process [3] to deliver the desired dataset. A script has been written for the game camera to capture the object images from various viewpoints and distances to the object during the simulation. From a set of background images, the backgrounds of the objects are randomized in the simulation. By rendering multiple views of the scenes, we generate a diverse set of synthetic images that mimic real-world conditions. Additionally, data augmentation techniques such as object rotation, scene illumination, and object occlusion are applied. Furthermore, the labeling tool within this Unity scene automatically annotates the generated images with bounding boxes, providing ground truth information for training and validation.

In Table 1, the benefits of using synthetic data were demonstrated in the context of installing small objects on a breadboard as an assembly process. Additionally, utilizing synthetic data presents advancements in the two-step detection approach. Table 1 illustrates the Mean Average Precision (mAP) for the detection of the small button on the breadboard for our three experiments. The mAP is a widely used evaluation metric in object detection tasks. It measures the accuracy and precision

**Table 1** The mAP results for all 3 experiments for different Intersection of Union (IoU) are illustrated and confirmed that the two-step detection improved the mAP for the buttons, the target small object in the frames [25]

**Fig. 3** (**a**) Synthetic image for assembly objects, (**b**) object detection on a video captured from HoloLens2 for the corresponding real objects

of object detection algorithms by calculating the average precision for each class and then taking the mean across all classes. The mAP is calculated according to [18]. The first experiment is trained on the 295 conventional images from buttons and tested on the 90 images containing 221 tiny buttons. The same setup is applied in the 2nd experiment, but in the training phase, we utilized the 1300 synthetic images of the breadboard and 2500 images of tiny buttons installed on the breadboard. It is clear that the mAP for the second experiment is considerably higher than the first experiment, which is around 0% for almost all different IoUs. In the third experiment, we utilized the same training and testing data as in the second experiment. However, during the inference phase, we adopted a different approach. Instead of resizing the entire frame, we first detected the breadboard and then cropped it based on the context (specifically, the Breadboard). The resulting cropped frame was then forwarded to the YOLOv4 object detection pipeline to detect the buttons as small objects. Through the third experiment, we demonstrated how employing a context-based cropping approach on the frames leads to a significant improvement in mAP (Table 1). More details can be found in [25] (Fig. 3).

#### **4 Ongoing Research**

In this section, we present the current research endeavors conducted at our research center in the field of object detection. We will highlight the ongoing projects and studies that are directly aligned with our focus on advancing object detection techniques.

#### *4.1 Hybrid Dataset*

Considering the advantages that synthetic data offer with respect to reduced human effort and time consumption, it has gained research interest. However, it can encounter challenges related to optimizing various aspects of the scene, including CAD models, lighting conditions, backgrounds, and object textures. In our recent research, we are specifically investigating the utilization of real data in combination with synthetic data to enhance the precision of object detection. By leveraging real data in this manner, we aim to optimize the object detection performance and facilitate the creation of datasets that meet our specific requirements.

#### *4.2 Continual Learning (CL)*

An additional challenge in this chapter is the need to update previously trained object detection models to accommodate new tasks, rather than retraining the model from scratch with both old and new data. This process of continual learning aims to address the issue of catastrophic forgetting, where the model's performance on previous tasks significantly declines as it is trained on new tasks [29]. This presents an interesting avenue for further exploration within the field of study.

Both domain incremental learning and task incremental learning [14] offer potential research approaches that can be applied to our specific environmental scenarios. These methods enable the model to adapt to new tasks while retaining knowledge from previous tasks. Investigating and leveraging different continual learning techniques can contribute to the development of a more flexible and efficient object detection approach.

Figure 4 depicts the utilization of continual learning to optimize the object detection process using the synthetic data in our scenarios. The procedure begins by importing CAD models of new objects and subsequently generating synthetic image datasets. These datasets are then passed to the next phase, where the previously trained YOLOv7 model is retrained using the replay approach of continual object detection. Finally, in the evaluation phase, frames captured from HoloLens are processed by the new object detection model, and the performance of this model is assessed. Thus it is an iterative process, starting from importing synthetic data to dataset generation, model retraining, and evaluation phase, aiming to expand the range of OD classes by updating the previously trained model.

**Fig. 4** The pipeline demonstrates the sequential process of generating synthetic data, conducting training and testing using the HoloLens2 World Camera, with the aim of establishing a continual learning pipeline for model updates

#### **5 Conclusion**

This chapter provides an exploration of object detection in industrial environments, specifically focusing on various scenarios involving human assistance systems, and safety requirements for human–robot collaboration. Additionally, we examine the application of object detection in conjunction with augmented reality devices, which offer intuitive communication interfaces for workers in these environments.

Moreover, we emphasize the substantial advantages gained from the incorporation of synthetic data in expediting the laborious data generation process and enhancing object detection outcomes. Through the utilization of synthetic data, we achieve improved efficiency and precision in object detection results. Additionally, we present our recent research focusing on the detection of small objects within industrial environments, which poses a significant challenge. We demonstrate that our approach significantly enhances the detection of small objects in manual assembly scenarios, resulting in notable improvements in performance.

Ultimately, this chapter not only sheds light on some solutions to object detection barriers in industrial settings but also paves the way for further exploration in related fields. This includes the development of sustainable practices within our scenarios and the establishment of a more generalized process that encompasses data generation and the preparation of object detection models.

**Acknowledgments** This work was supported by the European Union's Horizon 2020 program project STAR, under grant agreement numbers H2020-956573, and by the BMBF (German Federal Ministry of Education and Research) project InCoRAP (0IW19002).

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Boosting AutoML and XAI in Manufacturing: AI Model Generation Framework**

**Marta Barroso, Daniel Hinjos, Pablo A. Martin, Marta Gonzalez-Mallo, Victor Gimenez-Abalos, and Sergio Alvarez-Napagao**

#### **1 Introduction**

The field of manufacturing is increasingly interested in adopting Artificial Intelligence (AI) due to its ability to revolutionize operations, improve efficiency, and drive innovation. AI algorithms can effectively analyze and extract valuable insights from data, enabling manufacturers to optimize processes, detect anomalies, and make data-driven decisions. In addition, they bring automation and predictive capabilities, by enabling the automation of tasks, such as quality control and predictive maintenance, leading to faster inspections, reduced downtime, and improved operational efficiency.

However, despite all these benefits, the adoption of AI from manufacturing side is not happening as quickly as expected [1], and it comes with its own set of challenges and difficulties. While it holds great promise, several challenges need to be addressed for successful integration into the manufacturing processes. These challenges include issues related to data availability and quality, integration with existing infrastructure, shortage of skilled personnel, ethical and regulatory considerations, and change management [2]. Manufacturing environments often entail complex data ecosystems, requiring proper data collection and preparation processes to ensure the availability of high-quality and relevant data usable to produce intelligent models. Furthermore, integrating these models with existing infrastructure may require system upgrades and compatibility assurance. The shortage of skilled professionals with simultaneous expertise in both AI and manufacturing processes hinders adoption, while ethical and regulatory concerns

M. Barroso (-) · D. Hinjos · P. A. Martin · M. Gonzalez · V. Gimenez · S. Alvarez-Napagao High Performance and Artificial Intelligence, Barcelona Supercomputing Center, Barcelona, Spain

e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
require robust governance policies. Change management is also crucial, requiring cultural shifts and addressing employee concerns to foster a positive AI adoption environment.

To overcome these challenges, AI providers should also facilitate adoption through the development of user-centered tools, fostering partnerships between experts and manufacturing professionals, and showcasing the tangible benefits of implementing machine and deep learning algorithms. By making AI accessible, tailored, and demonstrably valuable, the manufacturing industry can overcome barriers and embrace these technologies to unlock their full potential for improving operational efficiency, productivity, and competitiveness.

In this matter, there are two research areas that can play significant roles in nurturing this adoption: Automated Machine Learning (AutoML) [3] and Explainable Artificial Intelligence (XAI) [4].1 AutoML simplifies the process of developing models by automating tasks such as feature engineering, model selection, and hyperparameter tuning. This automation reduces the need for extensive data science expertise, making it easier for manufacturing professionals to leverage AI capabilities. By providing user-friendly interfaces, pre-built algorithms, and automated workflows, AutoML tools enable manufacturers to quickly and efficiently build accurate and robust intelligent models tailored to their specific requirements.

On the other hand, XAI addresses the critical need for transparency, interpretability,2 and trust in AI systems. Manufacturing environments require clear explanations of model decisions, especially when it comes down to critical factors such as quality control, predictive maintenance, and process optimization. XAI techniques allow manufacturers to understand the inner workings of models, identify the factors influencing predictions, and detect potential biases or risks. By providing interpretability and explanations, XAI helps build trust in AI systems and facilitates their adoption by manufacturing professionals.