**Lecture Notes in Networks and Systems 710**

Cezary Biele · Janusz Kacprzyk · Wiesław Kopeć · Jan W. Owsiński · Andrzej Romanowski · Marcin Sikorski Editors

# Digital Interaction and Machine Intelligence

Proceedings of MIDI'2022 – 10th Machine Intelligence and Digital Interaction – Conference, December 12–15, 2022, Warsaw, Poland (Online)

## **Lecture Notes in Networks and Systems 710**

## Series Editor

Janusz Kacprzyk, *Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland*

## Advisory Editors

Fernando Gomide, *Department of Computer Engineering and Automation—DCA, School of Electrical and Computer Engineering—FEEC, University of Campinas—UNICAMP, São Paulo, Brazil* Okyay Kaynak, *Department of Electrical and Electronic Engineering, Bogazici University, Istanbul, Türkiye* Derong Liu, *Department of Electrical and Computer Engineering, University of Illinois at Chicago, Chicago, USA Institute of Automation, Chinese Academy of Sciences, Beijing, China* Witold Pedrycz, *Department of Electrical and Computer Engineering, University of Alberta, Alberta, Canada Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland* Marios M. Polycarpou, *Department of Electrical and Computer Engineering, KIOS Research Center for Intelligent Systems and Networks, University of Cyprus, Nicosia, Cyprus* Imre J. Rudas, *Óbuda University, Budapest, Hungary* Jun Wang, *Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong*

The series "Lecture Notes in Networks and Systems" publishes the latest developments in Networks and Systems—quickly, informally and with high quality. Original research reported in proceedings and post-proceedings represents the core of LNNS.

Volumes published in LNNS embrace all aspects and subfields of, as well as new challenges in, Networks and Systems.

The series contains proceedings and edited volumes in systems and networks, spanning the areas of Cyber-Physical Systems, Autonomous Systems, Sensor Networks, Control Systems, Energy Systems, Automotive Systems, Biological Systems, Vehicular Networking and Connected Vehicles, Aerospace Systems, Automation, Manufacturing, Smart Grids, Nonlinear Systems, Power Systems, Robotics, Social Systems, Economic Systems and other. Of particular value to both the contributors and the readership are the short publication timeframe and the world-wide distribution and exposure which enable both a wide and rapid dissemination of research output.

The series covers the theory, applications, and perspectives on the state of the art and future developments relevant to systems and networks, decision making, control, complex processes and related areas, as embedded in the fields of interdisciplinary and applied sciences, engineering, computer science, physics, economics, social, and life sciences, as well as the paradigms and methodologies behind them.

Indexed by SCOPUS, INSPEC, WTI Frankfurt eG, zbMATH, SCImago.

All books published in the series are submitted for consideration in Web of Science.

For proposals from Asia please contact Aninda Bose (aninda.bose@springer.com).

Cezary Biele · Janusz Kacprzyk · Wiesław Kope´c · Jan W. Owsi ´nski · Andrzej Romanowski · Marcin Sikorski Editors

## Digital Interaction and Machine Intelligence

Proceedings of MIDI'2022 – 10th Machine Intelligence and Digital Interaction – Conference, December 12–15, 2022, Warsaw, Poland (Online)

*Editors* Cezary Biele National Information Processing Institute Warsaw, Poland

Wiesław Kope´c Polish-Japanese Academy of Information Technology Warsaw, Poland

Andrzej Romanowski Institute of Applied Computer Science Łód˙z University of Technology Łód´z, Poland

Janusz Kacprzyk Polish Academy of Sciences Systems Research Institute Warsaw, Poland

Jan W. Owsi ´nski Polish Academy of Sciences Systems Research Institute Warsaw, Poland

Marcin Sikorski Department of Informatics in Management, Faculty of Management and Economics Gda´nsk University of Technology Gda´nsk, Poland

ISSN 2367-3370 ISSN 2367-3389 (electronic) Lecture Notes in Networks and Systems ISBN 978-3-031-37648-1 ISBN 978-3-031-37649-8 (eBook) https://doi.org/10.1007/978-3-031-37649-8

This work was supported by NiPI PL525 000 91 40.

© The Editor(s) (if applicable) and The Author(s) 2023. This book is an open access publication.

**Open Access** This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

## **Foreword**

Artificial intelligence (AI) is rapidly affecting more aspects of our lives thanks to significant advancements in its research and the widespread usage of interactive goods. This is leading to the birth of several new social phenomena.

Many nations have been working to comprehend these phenomena and discover solutions for moving artificial intelligence development in the proper direction to benefit individuals and communities at large. These initiatives necessitate multidisciplinary approaches that span not only the scientific fields involved in the creation of artificial intelligence and human-computer interaction but also strong collaboration between researchers and practitioners.

Because of this, the major objective of the MIDI conference is to combine two, up until recently distinct, areas of computer science research: artificial intelligence and human-technology interaction. Beginning in 2020, topics discussed at the MIDI conference will include artificial intelligence-related challenges in addition to interface design and user experience.

There is no denying that society is becoming more and more conscious of issues associated to the use of artificial intelligence in solutions. However, the development of artificial intelligence technology is advancing far more quickly than the search for solutions to the moral, social, and economic problems of the present. Both social research and AI technology expertise are required for the discussion of them to provide acceptable answers. As the conference's organizers, we are certain that the expanded format will make it an even more engaging platform for experts in the fields of artificial intelligence and human-technology interaction to exchange experiences.

As the conference hosted two Special Sessions, in 2022 the volume has been divided into four parts: (1) Machine Intelligence, (2) Digital Interaction, (3) Special Session: Advances in Collaborative Robotics, and (4) Special Session: Interacting with Virtual Reality Applications. As a result, the chapters deal with such topics as reinforcement learning, music prediction, medical usage of AI (prostate MRI analysis, chest scans analysis), co-designing immersive environments, gestural interaction with 3D data, collaborative robotics (i.e., electronic skin), and interaction with VR apps.

We believe that all readers interested in emerging trends as well as creators of enduser IT products and services will find inspiration and useful theoretical and practical information in this book. The eventual success or failure of a newly generated product depends on the focus on meeting the requirements of people in a future where technical solutions based on artificial intelligence are developed by people for people. No matter how cutting-edge a technology solution may be, if it is not sufficiently connected to the lifestyle or other variables affecting the target users' social behaviors, they will reject it. Underestimating the value of technology is a mistake, as evidenced by the history of technical progress and examples of technological solutions developed even by the biggest tycoons in the world.

#### vi Foreword

During this year's edition of the conference, two papers were awarded Best Paper Award in memory of professor Krzysztof Marasek. The professor was one of the initiators of the first MIDI conference focused on the user's perspective in computer science and the co-organizer of subsequent editions. Being an outstanding scientist and engineer, an excellent specialist in the field of linguistics, voice user interfaces, and voice-based interaction, he understood the importance of research on technological solutions conducted from the user's perspective. We hope that this year's and subsequent editions of the MIDI conference will step up to the mark and continue the work of professor Krzysztof Marasek.

> Cezary Biele Janusz Kacprzyk Wiesław Kope´c Jan W. Owsi ´nski Andrzej Romanowski Marcin Sikorski

## **Contents**

viii Contents

#### x Contents

## **Machine Intelligence**

## **Light Fixtures Position Detection Using a Camera**

Vojtˇech Leischner(B)

Czech Technical University in Prague, Technick´a 2, 160 00 Prague 6, Czech Republic<EMAIL_ADDRESS>https://dcgi.fel.cvut.cz

**Abstract.** Interdisciplinary research combines computer vision with stage light design to automatically detect light fixtures' positions to create light animations. Multiple programmable light fixtures are often used in theaters, the event industry, and interactive installations. When creating complex animations such as a wave traveling from one side to the other through multiple light fixtures array, all lights' positions must be known beforehand. Traditionally the position of the light is marked in the technical plan. However, technicians make mistakes during the installation and sometimes install the light in a different position. In such a case, time-consuming troubleshooting is needed to determine which light is misplaced and either correct the position in the software or manually move the light to the correct position. Our system saves time during installation and produces a light id and position pairs that users can use in various lighting control software. As a result, users can improvise and change the light positions more intuitively without needing a technical plan. Our system saves installation costs and enables rapid prototyping of light shows to create previously impossible organic designs. We verified the system in a controlled experiment and measured the influence of camera resolution on accuracy.

**Keywords:** DMX512 *·* light fixture *·* computer vision *·* position detection *·* camera

## **1 Introduction**

### **1.1 Use Case**

We focus on large-scale dynamic light installations. Consider the light installation at 131 South Dearborn, Chicago, the USA [22] as an example of an ideal use case. The installation consists of 925 glass bubbles, and each bubble has its light source. Install 925 individually programmable light sources and ensure that the wiring is right pose a considerable challenge and is prone to mistakes during installation. We aim to automate the light position mapping process with the proposed system.

Another use case we tested was mapping multiple programmable led rings and strips. Instead of manually marking each fixture's position, we have automated the process. As a result, the setup can be changed frequently to help designers find the ideal configuration and immediately test it. Please watch the experiment video [15] documenting the process.

### **1.2 Programming Light Show**

Without knowing the light positions, we can create only simple light shows, such as changing the light parameters uniformly or creating animations based on noise. To create a more complex light show, we need to distinguish individual lights and know their position.

We can then create a simulation with light sources represented as points in space. See Figure 1. We add virtual objects and animate their position. We can achieve various light effects by detecting the collision of light sources' positions with a volume of moving virtual objects. We turn on the light source when the light is inside the virtual object and turn it off when it is outside. We can also use multiple virtual objects to create more elaborate animations or map them interactively based on sensor inputs. For example, one can map people's movement to light intensity in different sectors.

**Fig. 1.** Schema of light scene animation: Light sources represented as points in space collide with the animated virtual 3D object. Lights inside the object will be turned on.

#### **1.3 Light Network Control**

Light fixtures are often controlled using DMX512 [10] protocol. Traditionally DMX address (a unique id of the light) is selected using a hardware DIP switch directly on the light fixture. Another option is to use Remote Device Management (RDM [9]) commands from the control software [6]. However, RDM is only available with some DMX light fixtures, so we often rely on manual selection, making changing DMX addresses difficult. According to the technical plan, each light with the appropriate DMX address must be installed in the correct position.

To control the lights, most often, we use the Artnet [2,22] or sACN [1,11] protocol - an extension of DMX standard that enables us to control more lights and use network topology and devices such as Ethernet switches to send DMX packets over the network.

In essence, we can control individual lights from a single networked computer. We need an ethernet adapter and Artnet/sACN node to convert the signal to DMX. DMX signal is then sent to DMX driver that maps the DMX values to voltage and current to dim individual lights. See Figure 2.

**Fig. 2.** Artnet light network diagram (https://www.ledlightinghut.com/artnetdmx512-converter.html).

### **2 State of the Art**

Various systems using a camera to control lights exist. Most of them deal with finding a user or controlling the light to track the user. In most cases, available solutions rely on knowing the light fixtures' position in space. We are trying to solve a different problem, localizing light sources relative to the camera. Still, some principles can be used for both problems.

Luxapose [14] system used a mobile phone as a camera and modified light fixtures to track a mobile phone's position. Light sources are modulated to produce pulses of light with encoded position and ID information that can be captured in a single frame exploiting the rolling shutter effect. While we are trying to determine the position of the light fixture, Luxapose is tracking a user. Theoretically, we could repurpose the system and use the known camera position to determine the light position. Unfortunately, the Luxapose system requires light source modification which is not practical in our use case. Moreover, the camera position relative to the light source would have to be known. While feasible, it would require precise measurement on site.

Similar to the Luxapose, a paper by Hossan and Tanvir [13] uses multiple known light fixtures to localize the camera position. We can not use triangulation methods to determine the camera position when using a single camera in our system. In practice finding the precise camera position relative to the light fixture would have to be repeated for all lights as we can not guarantee their position relative to each other. Such manual measurements would completely negate any benefits of automatization. The authors use found blob pixel count to measure the distance from the camera to the light fixture. A similar approach could be adopted while moving the camera back and forth to determine the distance from the camera to the light source. Other approaches [24] use a photo-diode as a sensor installed in the light fixture to track objects based on changes in light propagation. Multiple light fixtures have to be taken into account, or a quadrant photo-diode is used [7]. The camera is no longer needed in these cases, but the light fixtures have to be modified to include the appropriate sensor and communication unit. For example, BlackTrax [8], a commercial solution for motorized lights, uses multiple cameras and an infrared beacon to track the moving target. Craig Hiller developed a system to detect light's positions and even classify them as a fluorescent bulb, tube, or LED light [12]. The project relies on a Tango tablet [17] aligned with a DSLR camera and IMU in one package. The user walks under the lights to map their position and create a spatial map. Tracking is only possible with sensors being perpendicular to the light fixture, which is fine when detecting ceiling-mounted lights but would fail with vertical or organically shaped installations. The reported error is also not suitable for the light show scenario, with some lights being false positives and others missed during detection.

## **3 Software**

#### **3.1 Programming Environment**

The main program is written in Java Processing framework [4]. To acquire the camera stream, we used Gstreamer [23] based library [11]. To perform computer vision tasks, we have used the Java wrapper of OpenCV library [5].

### **3.2 Light Fixture Detection**

To find the light position, we create a frame difference between a frame with all lights turned off and a frame with a single light turned on. Then we threshold the resulting absolute difference to a binary image. We find the contours of the light blobs. Finally, we sort found blobs based on their area. We select the blob with the biggest area and find its centroid. The resulting x and y coordinates are saved and associated with the light DMX address. When we have processed all the lights, we normalize their coordinates and save all the information into a JSON file. Later this file can be loaded into light control software such as Touchdesigner [21], Madrix [3], OpenFrameWorks [20], Processing [4], VVVV [18], and similar software used to create light shows.

#### **3.3 GUI and User Input**

We have also developed a GUI for easier usage. We provide multiple options, such as selecting a target IP, setting custom binary thresholds, selecting from multiple cameras, and setting minimal and maximal blob size. Users can also select whether to cycle through all lights automatically or one by one by clicking the button. The user can manually adjust every detected light position with a mouse if needed. Users can also create custom masks to select the area in the camera image that should be ignored during detection. Furthermore, we have also enabled perspective corner pin transformation to be applied to the camera image. Acquiring undistorted camera images is essential to measure uniform distances between light fixtures correctly.

## **4 Experiment**

**Fig. 3.** Experiment setup - LEDs mounted on the wooden plate

## **4.1 Lights Setup**

We have tested our system on a setup with 30 individually programmable LED light sources controlled via the ArtNet network. Lights were installed on 250cm by 125cm wooden plate, all facing the same direction. See Figure 3. Each light is 3.5cm in diameter with 6 LEDs controlled as a single symmetrical light source.

## **4.2 Camera**

We have used the Logitech C922 camera, a widely available and affordable standard web camera with a USB interface. The camera was positioned perpendicular to the wooden plate 272cm away. The camera's Diagonal Field of View (FOV) is 78◦, horizontal FOV is 70.42◦, vertical FOV is 43.3◦, and Focal Length is 3.67mm. We tested the setup, once using 640\*480px camera resolution and once 1920\*1080px, so we could determine if the used resolution correlates with accuracy.

**Table 1.** Mean error in the position detection and standard deviation

## **5 Results**

Experiment data is available at Zenodo 6814223 [16]. All installed lights were detected correctly. See the results in Table 1. In the case of 640\*480px camera resolution, 1 pixel corresponded to 4.53mm in the wooden plate plane, and the average position error on the horizontal axis was 2.912mm and 2.426mm on the vertical axis. The maximum error on the horizontal axis was 2 pixels, and 1 pixel on the vertical axis with standard deviation 0.678px and 0.507px respectively.

In the case of 1920\*1080px camera resolution, 1 pixel corresponded to 2.02mm in the wooden plate plane, and the average position error on the horizontal axis was 2.885mm and 1.947mm on the vertical axis. The maximum error on the horizontal axis was 4 pixels, and 6 pixels on the vertical axis with standard deviation 1.069px and 1.318px respectively.

## **6 Discussion**

We assumed that the higher resolution of the camera and shorter distance to the lights would produce more accurate results as 1cm would be represented by more pixels in the camera image. Resolution does not play a crucial role, as we have not observed a significant accuracy increase when using 1920\* 1080px over 640\* 480px.

We need to know if particular light is left or right relative to the other light to propagate the animated wave through them correctly. We have successfully obtained correct relative relationships between lights irrespective of the used camera resolution. High absolute accuracy can be beneficial but unnecessary for the light-show creation.

The light source shines directly at the camera and produces a lens flare. Lens flare does not worsen the detection as long as it is uniform in all directions, so we can assume that the light source is in the center.

## **7 Conclusion**

We offer a robust way to automatically detect installed light fixtures positions with a single monocular camera. Furthermore, we save the information with the respective light DMX address in both machine and human-readable formats to be used in various light control software. Our proposed automatic light detection method can be thought of as a proof of concept with clear time and cost-saving benefits. More importantly, it opens the way for new stage light design possibilities. More testing in real-world scenarios is needed to further verify the presented system's viability.

## **8 Limitations**

An unobstructed view of all light sources is needed to detect their position correctly. Position mapping is possible only if we find a view in which the light fixtures do not overlap. For example, it would be challenging to automatically get the positions of light fixtures organized in a 3D helix shape.

The best use case for our system is the light sources positioned in a single layer. For example, lights are hanging from the ceiling.

The problem occurs when lights are organized in multiple layers. Such as multiple lights on a single string or rod beside each other. It might not be practical to use our method in such a case.

Reflections cause another limitation. When reflective surfaces surround the light sources, it can create several false-positive hot spots in-camera images that might be hard to distinguish from the actual light source. For example, a chromeplated ceiling has high reflectivity. In this case, reflections have to be eliminated before detection.

### **9 Future Research**

We can improve the tool to enable 3D position mapping. We can use the binocular stereoscopic camera to calculate the distance from the camera to the light. Alternatively, we can reposition a single monocular camera to acquire two points of view to achieve the same result. We could further improve the usability by merging multiple cameras to cover a larger area to map large-scale installations. Another approach would be to enable sequential mapping. After mapping one place, the user would physically move the camera to cover another neighboring area. The relative position change of camera origin could be calculated by the standard SLAM method [19].

### **10 Declarations**

**Acknowledgements.** The research was consulted with doc. Ing. Zdenˇek M´ıkovec, Ph.D.

**Funding and Competing interests.** The author works commercially in the interactive installations industry that can benefit from the proposed automatic light position detection system. This research has been supported by the project funded by a grant SGS22/172/OHK3/3T/13 and by RCI (CZ.02.1.01/0.0/0.0/16 019/0000765).

**Availability of data and materials.** Experiment data is available at DOI: 10.5281/zenodo.6814223 [16].

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Improved Vehicle Logo Detection and Recognition for Complex Traffic Environments Using Deep Learning Based Unwarping of Extracted Logo Regions in Varying Angles**

Zamra Sultan, Muhammad Umar Farooq(B) , and Rana Hammad Raza

Pakistan Navy Engineering College (PNEC), National University of Sciences and Technology (NUST), Karachi, Pakistan

<EMAIL_ADDRESS>{umar.farooq<EMAIL_ADDRESS>
**Abstract.** Intelligent Traffic Monitoring and Management System (TMMS) is a growing research area as cities infrastructure continues to evolve. Traffic situation is demanding innovative solutions for effective monitoring and management given the complex nature of the urban scenario. A major focus of this research domain is fine-grained vehicles classification that requires detection and recognition of distinct features of vehicles. Some of these features are semantic based while others are appearance based. One such appearance-based feature of a vehicle is its logo. Logo detection helps with identification of a vehicle's make during fine-grained classification process. There are various deep learning methods which give good performance for such object detection tasks. However, it is challenging to exploit these methods due to smaller size of logo especially in a surveillance environment. This work firstly presents a deep learning-based approach for detection of vehicles' logos in camera video feeds. Due to small size of logos, a unique pipeline using three different deep learning models is designed. Firstly, a modified Improved Warped Planar Object Detection Network (IWPOD-NET) selects a Region of Interest (ROI) and adjusts the orientation of vehicle logo. Then YOLO (You Only Look Once) v5 is used to detect the logo part in the selected ROI and finally, EfficientNet is used to further classify logo into different classes. This pipeline is tested on four surveillance environments namely toll control, law enforcement, dashcam, and parking lot access control. Comparative analysis shows accuracy improvement with this proposed approach in each testing case. A pose variance analysis is also performed to determine the orientation limits to which this approach can work. Secondly, a custom dataset, VL-10 (Vehicle Logos) is presented which provided further insights into the challenges w.r.t local environment settings. The whole approach improved the overall performance of the logo detection and recognition system.

**Keywords:** Video surveillance · Vehicle detection · Vehicle logo detection · Fine-grained vehicle logo classification · Deep learning · IWPOD-NET · EfficientNet · YOLOv5 · Image unwarping

### **1 Introduction**

Intelligent Traffic Monitoring and Management System (TMMS) has an increasing significance as the traffic infrastructure is advancing. With the advent of technology, intelligent TMMS has taken over conventional TMMS and has become an emerging research area. It includes detection, recognition and fine-grained classification of vehicles, automatic license plate recognition and various kinds of traffic analytics. Its applications vary from commercial domains to national security levels e.g., surveillance, security system, traffic congestion avoidance, accident prevention, advanced driver assistance systems, access control systems, intelligent parking systems, electronic toll collection, etc. These applications necessitate the development of more robust techniques for Intelligent TMMS.

The fine-grained classification of vehicles is the process of categorizing a vehicle into its make and model, generally known as Vehicles Make and Model Recognition (VMMR). Many techniques are used for this classification; some by considering the whole vehicle or others through a certain area of vehicles (e.g., frontal view, grill, or logo). A vehicle logo is a distinct appearance-based feature which is an important part of this whole process. Vehicle logo detection/recognition is required for many applications of TMMS. It can be used for estimation of brand reputation and traffic monitoring etc. Its recognition also plays an important role in the scenarios where the authenticity of vehicle's make is doubtful. For example, the scenario where a stolen vehicle's license plate is replaced with another one, or when a license plate is obscured or illegible, which affects the readability process in any license plate recognition application. The logo detection can be used as an authenticity validation step for this purpose. However, identification through vehicle logo is a challenging task due to its smaller size. Detection and recognition of small objects is a fundamental challenge in Computer Vision and is a growing research area these days. Many researchers have achieved high performance for small objects detection using Convolutional Neural Networks (CNNs). CNN is able to learn various image-based features from large-scale data without the need of any human intervention. Compared to different kinds of small objects, the vehicle logo has more complex rich content information at times due to their unusual designs. These small sized logos are also affected by background clutter and noise. Similarly, most of the times, logo is blurred and not properly illuminated, and even not visible due to rainy or snowy weather which causes difficulties in features identification. An overview of some challenges associated with this logo detection and recognition task are provided below:

• *Background Clutter and Noise*. Logo is a comparatively smaller part of vehicle due to which there are many sources of background interference including radiator grille, bumper and other small objects. These objects distract the feature extraction and results in increase of false detections.

This research has modified a state-of-the-art license plate detection model IWPOD-NET, to extract the desired ROI and then further unwarp it to fix the distortions in varying angles [32]. The used surveillance recordings are low in resolution and cause the accuracy to decrease. Therefore, a combination of different CNN models are used for detection and recognition purposes. The results show a significant improvement in overall detection and recognition of vehicle logos. The main contributions of this paper are highlighted as followed.

The rest of the paper is organized as followed: Sect. 2 provides a concise overview of the related research work. Section 3 provides a detailed working methodology of the proposed approaches and gives an overview of the proposed dataset. Section 4 discusses the test cases used to conduct this research and presents a comparative analysis of the results obtained from the experimentation. Finally, Sect. 5 concludes the papers and provides a future direction of this research.

## **2 Related Works**

Work in different domains of intelligent traffic monitoring and management system has been contributed by many researchers. Setchell discussed monitoring and management of road traffic using computer vision in details [1]. The paper presented two vision-based traffic monitoring systems including license plate recognition system and a road traffic monitoring system to track vehicles. Different ideas for vision-based traffic monitoring and management systems have been proposed [2-6]. VMMR is also an important part of traffic monitoring and management systems.

There have been many related approaches using conventional algorithms such as Scale-invariant Feature Transform (SIFT), Histogram of Oriented Gradients (HOG) and Sequential Pattern Mining (SPM) etc. [7-9]. Usually, logo is detected through the reference of license plate [10-12]. After the detection of license plate, different techniques are then implemented to segment the logo [13-15]. Psyllos *et al.* introduced an enhanced SIFT-based features extraction mechanism for vehicle logo recognition. The experiment results showed a significant improvement of recognition accuracy compared to conventional SIFT algorithm [16]. Mao *et al.* proposed a novel vehicle logo detection method which highlighted logo regions using direction filters and saliency map. Information entropy was then used to choose the binary image and for precise location of logo region [17]. Yu *et al.* proposed a system for vehicle logo recognition based on Bag-of-Words (BoW) [18]. The vehicle logo images were represented as histograms of visual words and were classified by SVM in three steps. These steps involved SIFT feature extraction, quantization of features into visual words and histograms of visual words with spatial information. Wang *et al.* presented a vehicle logo detection/recognition system which uses edge features to find logo in the rough logo region detected through prior knowledge [19]. Then, a combination of template matching and HOG were used to identify the category of logo.

These conventional techniques are usually dependent on hand-crafted features, which lack robustness in certain conditions. These conventional algorithms are relatively easier to implement, however, these algorithms are computationally expensive. Compared to these conventional approaches, various deep learning-based methods are also explored by researchers. The deep learning-based algorithms have significantly better performances in detection of objects under different complex environments as discussed in Sect. 1. Tong. K *et al.* presented an in-depth analysis of different deep learning-based small objects detection approaches [20]. The authors showed with experimentation that techniques such as multi-scale feature learning, data augmentation and different training strategies can significantly improve the small objects detection. Similarly, these stateof-the-art techniques have also been tried and tested for the task of small objects such as vehicle logo.

Yang *et al.* proposed a modified YOLOv3 model for vehicle logo detection in complex scenes [21]. The approach showed good accuracy on the proposed vehicle logo detection dataset. Zhang *et al.* proposed a real-time lightweight vehicle logo detection system based on deep convolutional networks [22]. The experimentation showed a significant improvement in accuracy of vehicle logo detection task. Jiang *et al.* proposed an improved YOLOv4 model which enhanced the backbone feature extraction network for efficient extraction of small vehicle logos [23]. It further used convolutional transformer to reduce influence of complex backgrounds. Yang *et al.* proposed an improved YOLOv2 network which provided fast and accurate vehicle logo detection [24]. The experimental results show that this approach significantly improved accuracy and speed of logo detection.

The surveillance recordings usually have low resolutions which makes the logo detection challenging, particularly at varying angles. Various techniques are used to improve the quality of video prior to the detection of objects. Some researchers used Super Resolution algorithms to overcome these limitations and to convert these lowresolution images to high-resolution ones [25-29]. Others used near-field and far-field video enhancement techniques [30]. Some researchers suggested the use of ROI rate control scheme for surveillance videos [31]. However, this proposed research work adjusts the orientation of logos at varying angles, which improves the overall vehicle detection and recognition task.

## **3 Working Methodology**

## **3.1 Selected Models**

A unique pipeline using a combination of different deep learning models is designed in this study for improved vehicle detection and recognition. The specific details of the selected models are as follows:

*Unwarping Model (IWPOD-NET).* Improved Warped Planar Object Detection Network (IWPOD-NET) [32] is able to detect four corners of vehicle license plate in a variety of unconstrained scenarios. It then unwarps the license plate to a fronto-parallel view and eliminates perspective related issues. This network was introduced to tackle license plate detection with varying viewpoints. However, this research modified the model to serve as basis for ROI localization and orientation adjustment of logos. The localized area above the license plate was extended by a variable (assuming that the logo placement is always on the top of license plate) to include the logo part as well in complete ROI, which was then unwarped. For example, if the height of license plate is 3 units and variable is set to be 2 units, then the ROI will be extended by 2 × 3 units vertically upwards. Workflows for both original and modified network are shown in the following Fig. 1. The characters on the license plate of the vehicles utilized in this study have been concealed in order to maintain the privacy of the vehicle owners.

**Fig. 1.** Workflows for both the Original IWPOD-NET (a) and Modified IWPOD-NET (b) models

*Logo Detection Model (YOLOv5m).* Advent of YOLO models improved the real-time object detection. Many versions of YOLO have been introduced so far, which have significantly improved object detection. This research utilizes YOLOv5, which has different architecture series including P5 and P6. These series are further divided into nano (n), small (s), medium (m), large (l) and extra-large (x) that vary on the basis of parameter sizes and provide optimizations for particular image sizes. For example, P5 is optimized for training images of sizes 640 × 640. For the desired task, YOLO-v5-P5 medium sized model is used in this system of logo detection. Due to low classification accuracies, the model was used to detect the presence of logo only. For a comparative analysis, results are also presented in Table 2 for the approach when YOLO was used for both detection and classification purpose. The unwarped image from previous step is then fed to YOLO-v5m model for the detection of logo. The workflow is shown in following Fig. 2. As shown in the figure, logo is successfully extracted from the complete ROI.

**Fig. 2.** Logo Detection using YOLO-v5m

*Logo Classification Model (EfficientNet).* EfficientNet [29] introduced by Google is a commonly used CNN architecture for image classification purposes. The model achieves state-of-the-art performance, in addition to being faster and lightweight. This research work trained EfficientNet on logos in the proposed dataset. Firstly, all images were cropped to the logo part, based on dimensions from the annotated images that were fed to the object detection model. Then these cropped logo images were used for training the classification model. In testing phase, once the logo is detected in ROI, it is then fed to the classification model as an input. It then classifies the logo into its respective class. The addition of EfficientNet has significantly improved the results as discussed in Sect. 4.

#### **3.2 Dataset**

A high-quality dataset is essentially required for tackling various computer vision tasks involving object detection and recognition etc. The lack of a local dataset can become a huge challenge when performing these tasks for any particular region with a specific complex traffic environment. To the best of our knowledge, there is no standard local dataset available. Therefore, a local dataset of vehicles, VL-10, is developed where logos are either prominently visible or partly visible due to varying angles as in surveillance environment as shown in Fig. 3.

#### 18 Z. Sultan et al.

All the logos are self-collected from various local and online sources (e.g., different car selling websites with publicly available images). The logos are categorized into ten different vehicle classes, including Honda, Suzuki, Toyota, Kia, Hyundai, Mitsubishi, Daihatsu, Faw, Hino and Nissan. A total of 500 and 50 images per class were selected to form the training and validation sets, respectively. The data augmentation was then carried out by adding blur and noise to each image. Therefore, the total images per class became 1500 and 150 for training and validation sets, respectively.

**Fig. 3.** Some Images from VL-10 Dataset

## **4 Results and Discussions**

#### **4.1 Test Cases**

Different surveillance scenarios have been chosen for test/evaluation of the proposed model pipeline. These include video streams from toll control (i.e., CCTV video from height), law enforcement (road-level cameras), parking lot access validation (cameras installed at various entry/exit points), and dashboard cameras. The highlights of each test case are provided in Table 1. Similarly, example frames from each test case are shown in Fig. 4.

#### **4.2 Comparative Analysis**

Three different approaches were tested, and their comparative analysis is accordingly presented in this section.

*Approach 1 - Without IWPOD-NET Using YOLOv5 for Both Detection and Classification.* The YOLOv5 model was trained on VL-10 dataset for both object detection and classification tasks. All test scenarios were first passed to this model to get baseline accuracies. The mAP for detection and classification is presented in Table 2 for each test case. IWPOD-NET was not used in this approach so that results could be compared, and the impact of IWPOD-NET could be observed.

*Approach 2 - With IWPOD-NET Using YOLOv5 for Both Detection and Classification.* A modified IWPOD-NET was introduced. The detected vehicles were passed to IWPOD-NET for the ROI extraction and unwarping. After extraction of ROI, it is then passed to the YOLOv5 model trained on VL-10 dataset. Here, YOLOv5 detects and classifies the logos into their respective classes. Results obtained in the form of mAP are shown in Table 2. It can be seen that results were improved to a certain level. The highest improvement shown is in the Law enforcement test scenario. This is due to the fact that this test set has side camera viewpoints and IWPOD-NET helped to unwarp these side views really well. On the other hand, the lowest improvement is shown by parking lot access validation test set, which is apparently because camera viewpoint was already frontal, and the license plates were not very oriented in this case.

*Approach 3 - With IWPOD-NET Using YOLOv5 for Detection and EfficientNet for Classification.* In the third approach, while using IWPOD-NET and YOLOv5, another model, EfficientNet was introduced. YOLOv5 was trained to detect the presence of logo only. All the classes of VL-10 dataset were trained on EfficientNet which was then used for logo classification. Firstly, each test dataset was passed to modified IWPOD-NET for the extraction and unwarping of ROI. The ROI was then passed to YOLOv5 model for the detection of logo. After the logo was detected, it was then passed to the EfficientNet for classification of the detected logo. The mAP for the detection by YOLOv5 and the accuracy of classification by EfficientNet is shown in Table 2. Considerable improvements in the results were observed. The highest improvement in the detection were observed in the Dashcam testing case. The margin left was due to the motion blur present in this case since both the objects (vehicle/logo) and the camera were in motion. Highest classification accuracy was produced in the case of parking lot access validation. The margin left in this case was due to the inclusion of night images in dataset where the logos were not perfectly visible, and the use of streetlights create the issue of reflection. Approach 3 stood out from the other two approaches. Therefore, this pipeline could be used as an offline tool for logo detection and classification in various surveillance environments.

#### **4.3 Pose Variation Calculation**

Pose variation analysis was conducted to determine the capacity of the proposed model to detect logos accurately at varying angles. For this purpose, some images were captured in the daylight similar to the settings in parking lot access validation dataset. Different angles of each vehicle were captured as shown in Fig. 5(a). Poses were categorized by comparing the oriented bounding boxes and rectangular bounding boxes of license plate as shown in Fig. 5(b). IWPOD-NET detected the four corners of license plate (green dots). It then made an oriented bounding box joining these four corner points (blue bounding box). Another algorithm was used to make rectangular bounding boxes (red bounding boxes). The upper lines for both bounding boxes were used to identify the poses. This was done by calculating the angle "θ" subtended between both lines.

20 Z. Sultan et al.

According to the observations, the proposed module was able to detect logo in the range of 0–30° while it was mostly undetectable beyond this range. In the range 0– 10°, the module's logo detection mean Average Precision (mAP) was 0.91 while the classification accuracy was 0.97. In the range 10–20°, the module's logo detection mAP was 0.85 while the classification accuracy was 0.90. On the other hand, in the range 20– 30°, the module's logo detection mAP was 0.80 while the classification accuracy dropped to 0.75. Angles from 0–40° are depicted in Fig. 6 along with the visual appearance of logos.

**Table 1.** Highlights of video data for different test cases

**Table 2.** Results for each testcase using all the three approaches

**Fig. 4.** (a) Single frame from each test case; (b)–(d) Individual vehicles in respective datasets of each test case along with their original logo containing ROIs and their respective unwarped ROIs

**Fig. 5.** (a): Photos capturing from different angles; (b): angle calculation for pose variance analysis

**Fig. 6.** Pose variance analysis on different angles along with respective logo appearance

## **5 Conclusion and Future Work**

Intelligent TMMS is grasping researchers' attention due to the evolving infrastructure of cities. This is not only becoming complex day-by-day but also demanding more innovative solutions. Vehicle classification is now shifting towards more fine-grained classification. For this, all such features of vehicles are being considered which are distinct and specific. One such feature is the vehicle's logo. Though, it stands out as a unique feature depicting the information regarding vehicle make/model, yet its small size is a complex challenge for its detection and recognition. The detection of small sized objects like a logo in a surveillance environment where vehicles themselves appear to be very small is nearly impossible. Video enhancement techniques increase computational cost and time. This paper specifically targeted surveillance environment and presented a novel method of detection and recognition of vehicle's logo. A modified IWPOD-NET was introduced to overcome the orientation issues. Then YOLOv5 was used to detect logos and EfficientNet was used to classify logos. This pipeline was tested on four surveillance environments namely toll control, law enforcement, dashcam, and parking lot access control. Comparative analysis showed accuracy improvement with this proposed approach in each testing case. Pose variance analysis was also performed to determine the orientation limits to which this approach can work. It was observed that the proposed approach worked best in the 0–30° range, while it was mostly undetectable beyond this range. Secondly, a new dataset, VL-10 was presented which focused on local environment settings. The proposed technique in this research work improved the results for logo detection and can therefore be used in offline surveillance environment.

As part of a future work, the deep learning architecture can be optimized to detect and recognize vehicle logos in real-time. Similarly, a completely new deep learning architecture can also be designed particularly for this vehicle logo task.

**Acknowledgement.** We acknowledge support from National Center of Big Data and Cloud Computing (NCBC) and Higher Education Commission (HEC) of Pakistan for conducting this research.

**Data Availability.** The custom dataset utilized in this study is available on request from the corresponding author.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Predicting Music Using Machine Learning**

Aishwarya Asesh(B)

University of Utah, Salt Lake City, USA<EMAIL_ADDRESS>
**Abstract.** The intricate temporally prolonged sequences seen in music make it a perfect environment for the study of prediction. Melody, harmony, and rhythm are three examples of the structural elements found in music. This research incorporates music excerpts prediction by understanding structural details using Markov chain and LSTM models. The novel approach compares to state-of-the-art algorithms by predicting how a musical excerpt would continue after being given as input. To compare the variations in prediction and learning, different learning models with different input feature representations were utilized. This algorithm envisions multitude of usage including next generation music recommendation system using intra-sequence matching, pitch-tone correction, amongst others by integrating with recent advances in deep learning, computer vision, and speech techniques.

**Keywords:** music excerpt · markov chain · LSTM · sequence prediction

## **1 Introduction**

Even though the western music scale only has twelve notes per octave, it encompasses the vast majority of the music one is familiar with, from Bach and Beethoven to Elton John and Beyonce. The past ten years have seen an increase in interest and a diversity of methods for algorithmically creating music using a variety of learning models [1]. Can deep learning models be used to study and utilize the genius of Mozart or Beethoven? How simple is it to recreate particular styles of music that the generation is accustomed to hearing?

While music has attracted a lot of attention, research on music prediction has received less attention. In this research, using a musical snippet, predictions are made for following N musical occurrences (for instance, 10 quarter-note beats). The actual musical occurrences are then contrasted with the predicted music, and the prediction is graded using a scoring system. Experiments leaves one wondering on whether music prediction or music generation is trickier.

## **2 Data**

The Patterns for Prediction Development Dataset, was used. This data was produced by analysing a randomly chosen portion of the Lakh MIDI Dataset<sup>1</sup>, a

c The Author(s) 2023 C. Biele et al. (Eds.): MIDI 2022, LNNS 710, pp. 26–34, 2023. https://doi.org/10.1007/978-3-031-37649-8\_3

<sup>1</sup> https://www.music-ir.org/mirex/wiki/2020:Patterns for Prediction.

dataset made up of one million popular music tracks. For both monophonic and polyphonic songs, the symbolic MIDI format was applied. Each input, or prime, corresponds to about 35 s of music, and each output, or continuation, includes the following 10 quarter-note beats.

## **3 Feature Representation**

#### **3.1 Basic Music Notation**

The fundamental component of music and the foundation from which all chords and melodies are built are notes. Each note has a pitch and a duration. Pitch is essentially the note's sound frequency. The MIDI music file system can store 128 different pitches. Quarter-note lengths are used to measure duration, which is the length of the note. When several notes are played simultaneously, they form chords. Since only one note is played at a time, monophonic music lacks chord structure. Polyphonic music has chords.

#### **3.2 Note and Chord Representation**

Each note was encoded as an integer ranging from 0 to 128 due to the fact that notes have 128 pitches in the MIDI file system. The number 129 was used to symbolize a rest, or the amount of time between notes. In string format, a chord was shown as a collection of pitches. For instance, "68.70.75" might be used to denote the chord with the pitches 68, 70, and 75. The training set's chords were all sampled, and a dictionary was built that converted the uncommon pairings into integer values. This corresponded to 101,039 distinct combinations and consequently input data dimensions for the medium-sized dataset.

How to handle inputs (chords) that are present in the testing set but absent from the training set is one of the difficulties in music prediction. This issue is avoided while creating music since the model only generates chords that are present in the training set. There will invariably be note combinations in the testing data that the model hasn't "seen" previously while making a forecast. The initial strategy was replacing the unfamiliar chord with the most prevalent chord in the practice set.

A more sophisticated approach is to substitute the most frequent chord found inside the prime sequence for the unknown chord as the next sequence item. Combinatorial proliferation of chord possibilities in polyphonic music makes the process a bit challenging.

**Duration Representation:** The representation of the durations is a quarternote length. The number of durations may be insurmountable, in principle. The duration length is typically spread among a limited set of choices. All of the training data's durations were sampled, and a dictionary was built that converted each distinct duration length to an integer. The dictionary's length varied depending on the dataset, although it was often less than 100 words long.

**Combination Representation:** The combination of note pitch and durations as a tuple was represented initially as a Pre-process step. For instance, the tuple was used to represent a note with a pitch of 60 and a duration of 1 quarter-note length (60, 1). Each distinct tuple was mapped to an integer value, just how the chords and durations were done.

**Interval Representation:** Note intervals were used in the final input representation that was assessed. Pitch difference between two nodes is an interval. Each interval denotes a semitone of difference between the pitches of two notes that are right adjacent to each other. For instance, the pitch difference between the notes 60 and 65 is 5 semitones. Negative intervals are used to illustrate drops in pitch. There are 256 total possible interval values because there are 128 total pitches. Instead of using pitch sequences for training and prediction, interval sequences were used for this representation. For instance, the sequence of intervals for a prime consisting of the four notes 65, 70, 45, and 60 would be 5, -25, 15. To find the note predictions using the list of interval changes, the last pitch of the prime was preserved in a list and utilized as the beginning point.

## **4 Methods and Modeling**

## **4.1 Markov Chain Model**

One of the earliest models utilized for music production was the Markov chain. A first-order and a second-order Markov chain was trained and evaluated. Separate training sessions were held for notes and durations. This model could be utilized with inference methods like restricted inference and beam search [2–4].

## **4.2 LSTM Model**

A Forest of Long-Short-Term Memory (LSTM) network was used for next set of experiments. A single network with multivariate input made up one model (both notes and durations). For the notes and durations, respectively, there were two different LSTM networks in the other model. It was established that notes and durations have a significant relationship. Input layer, 64-dimensional embedding layer, 512-hidden unit LSTM layer, batch normalization layer, and dropout layer with a dropout rate of 0.3 were the layers present in both models. The multivariate model then has two dense layers employing a categorical cross-entropy loss and an rmsprop optimizer for gradient descent after concatenating the two inputs. The batch size was 64, and there were typically 100 epochs. Experiments were performed using more epochs, but the outcomes didn't change, over-fitting was also observed in some cases. Figure 1 below displays the multivariate LSTM model's graph.

**Fig. 1.** LSTM Multivariate Model

#### **4.3 LSTM Encoder-Decoder Model**

Two different models for the notes and durations, as well as a single multivariate model, were trained. The parameters matched those of the LSTM model aforementioned. The longest prime sequence in the training set served as the input sequence length, and the longest target sequence in the training set served as the output sequence length. The graph of the multivariate encoder-decoder model is shown in Fig. 2.

#### **4.4 Inference Modeling**

Prediction was performed using the common beam search algorithm with beam sizes ranging from k = 2 to k = 4. Only the notes from the prime part were used for the constrained inference [5]. The following note with the highest score, for instance, would be the new option for prediction of song's prime sequence.

#### **4.5 Other Techniques**

Transposition was a method used to increase the uniformity of the input data. Transposition in music refers to the process of raising or lowering notes and chords by a fixed distance. The spaces between succeeding notes remain constant. Six major scales and six minor scales make up the 12 different scales in Western music. One of these scales is used in the majority of harmonic music. For instance, changing a song from the key of C major to the key of E major would require adding four semitones or four intervals to each pitch. Depending on whether the

**Fig. 2.** LSTM Encode-Decoder Multivariate Model

music was originally in a major or minor key, each input song was transposed to either C major or C minor. The training then took place using the transposed pitches. The prediction notes were created by transposing the prime sequence into either C major or C minor before testing. The anticipated target notes were then recomposed back into their original key before being contrasted with the actual target notes.

## **5 Results and Observations**

Both the training timeframes and the outcomes were adversely affected by the exponential growth of the input data for the polyphonic music. Scoring was done based on cardinality score and pitch score.

**Cardinality Score:** Minimizing a general supermodular monotone nonincreasing function *f*() with respect to a cardinality constraint. Let *x*<sup>∗</sup> <sup>1</sup> ∈*<sup>x</sup> f*(∅) <sup>−</sup> *f*(*x*) and use the greedy algorithm on the set function

$$g(S) := f(x\_1^\*) - f(x\_1^\* \cup S),$$

which is a monotone non-decreasing submodular set function with *g*(∅) = 0. Thus, the greedy algorithm maximizes the function with respect to a cardinality constraint within an approximation factor of 1 <sup>−</sup> <sup>1</sup>*/e*. However, there are two caveats. First, the greedy algorithm uses a budget of *k* <sup>−</sup> 1 instead of *k* (budget of one is spent on identifying *x*<sup>∗</sup> <sup>1</sup>). Second, and most importantly, observe that the approximation factor is obtained on the function *g*(*S*) and not *f*(*S*), they get a set *S* of size *k* <sup>−</sup> 1 such that

$$f(x\_1^\*) - f(x\_1^\* \cup S) \ge 1 - \frac{k}{(k-1)e} \cdot (f(x\_1^\*) - f(x\_1^\* \cup S^\*)),$$

where *S*<sup>∗</sup> is the optimal set of size *k* <sup>−</sup> 1 to be added to *x*<sup>∗</sup> <sup>1</sup> with the goal of minimizing *f*().

**The Pitch Score.** The right "shape" or separation from each note is rewarded by the cardinality of continuations. However, it rewards a continuation that is transposed one semitone lower in most cases. The pitches are additionally scored because this isn't entirely accurate. Two normalized histograms of the pitches from the prediction and the ground truth are made to achieve this. The amount by which the histograms are overlapped determines the score. The operation is carried out while ignoring octaves as well. The cardinality score and the pitch score are combined linearly to produce the final score.

**Baseline.** The first baseline estimate included selecting notes from the training set at random. The second technique involved selecting notes from the training set based on their frequency. The last approach used the same steps as the first, but added restrictions required that the sampled note be in the song's expected prime sequence. The third method generated the highest baseline for both the polyphonic and monophonic music sets.

**Discussions.** Polyphonic and monophonic datasets were evaulated individually. 90% of the songs were used for training and 10% for testing in the first division of the datasets into training and testing groups. Figure 3 and Fig. 4 displays the outcomes of numerous training and testing runs for the monophonic and polyphonic dataset. The LSTM model produced the greatest results when it used constrained inference, transposition on the input data, and a longer sequence length of 64. Compared to the best baseline score, the model's accuracy increased by over two times. The accuracy of the LSTM model was improved by transposing the input and lengthening the sequence. Transposition improved all of the models' accuracy, suggesting that more reliable input data improves learning. There were no appreciable improvements in accuracy brought about by the beam search method of inference. Prediction accuracy was also improved by constrained inference, particularly for the Markov model.

The notes and duration feature representations for the input data were far more accurate than the interval feature representation. This suggests that compared to pitches, intervals may have less structure and greater unpredictability. Unsurprisingly, the results were significantly worse for the input with larger dimensions (i.e., the tuple combinations of pitches and durations). There were some minor differences between employing a multivariate LSTM model and training two distinct models for notes and durations. When all other variables remained constant, the multivariate model consistently beat the two independent models by a little margin. This suggests that there is some relationship between the anticipated notes and durations. In addition to having a slightly higher accuracy, the multivariate models also required less time to train than two individual models.

**Fig. 3.** Results - Model Predictions for Polyphonic Music Data

Across all models, the pitch score was far more variable than the cardinality score. When the findings were examined more closely, it became clear that many of the projected sequences only had two or three different pitches. As a result, the model could determine which pitch or pitches appeared in the target sequence the most frequently, but not which order they were played. Additionally, no predictions were ever made for notes that in the target sequence seemed to be more "random" (i.e., not present in the prime sequence). While music might have a strong structural element, it also has randomness or surprises that provide appeal.

Overall, the LSTM model performed better than every other model. The Markov models had the highest variance of all the models, indicating that they would benefit from further restrictions or domain expertise [6–8].

The baseline values for the polyphonic music were significantly higher than those for the monophonic music, neither set of results should be used to judge the other. Due to the method used to determine the score, the baseline scores for the two forms of music are different.

**Fig. 4.** Results - Model Predictions on Monophonic Datasets

### **6 Conclusion and Future Work**

Because music has both structure and randomness, it presents an intriguing and difficult prediction problem. Although feature representation is significant, feature size is even more crucial. Training is substantially more challenging for inputs with very high dimensions. Constraints can significantly shrink the search space and improve model accuracy when used in inference. Meaningful learning and inference depend on limiting the input space and the search space in ways that nonetheless capture pertinent relationships and significant features.

Random pop tracks were employed in this research. Pop music is just one of many different musical styles. Jazz music is more free-form than classical music, which has a lot more structure. A fascinating expansion of this research would be to compare the results of other musical genres. Models, such as Hidden Markov Models and LSTM with attention mechanisms, leave room to wonder [9,10].

**Acknowledgments.** All that I am, or ever hope to be, I owe to my angel mother.

### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **A Novel Process of Shoe Pairing Using Computer Vision and Deep Learning Methods**

Marek Kozlowski1(B), Przemyslaw Buczkowski<sup>1</sup>, and Piotr Brzezinski<sup>2</sup>

<sup>1</sup> National Information Processing Institute, Warsaw, Poland<EMAIL_ADDRESS><sup>2</sup> Vive Textile Recycling, Warsaw, Poland

**Abstract.** The industrialisation of the footwear recycling processes is a major issue in the European Union—particularly in view of the fact that at least 90% of shoes consumed in western economies are ultimately sent to landfill. This requires new AI-empowered technologies that enable detection, classification, pairing, and quality assessment in a viable automatic process. This article discusses automatic shoe pairing, which comprises two sequential stages: a) deep multiview shoe embedding (compact representation of multiview data); and b) clustering of shoes' embeddings with a fixed similarity threshold to return sets of possible pairs. Each shoe in our pipeline is represented by multiple images that are collected in industrial darkrooms. We present various approaches to shoe pairing from fully unsupervised ones based on image descriptors to supervised ones that rely on deep neural networks—to identify the most effective one for this highly specific industrial task. The article also explains how the selected method can be improved by hyperparameter tuning, massive increases in training data, and data augmentation.

**Keywords:** clustering *·* deep learning *·* multiview embedding *·* representation learning

## **1 Introduction**

The increased availability of affordable mass-produced goods, coupled with rapidly changing consumer fashion trends, has resulted in a sharp increase in the consumption of products in many industrial sectors. The production of textiles for clothing and footwear is expected to increase by 2.7% each year until 2030<sup>1</sup>. In [8], the authors claim that approximately 5% of the twenty billion pairs of shoes produced worldwide every year are recycled or reused. In the European Union, it is estimated that the waste that results from postconsumer shoes could reach several million tonnes per year. Politicians and members of civil society

<sup>1</sup> https://www.businessoffashion.com/reports/news-analysis/the-state-of-fashion-2021-industry-report-bof-mckinsey/.

c The Author(s) 2023

C. Biele et al. (Eds.): MIDI 2022, LNNS 710, pp. 35–44, 2023. https://doi.org/10.1007/978-3-031-37649-8\_4

are starting to implement the Zero Waste to Landfill policy to address one of the major challenges of the twenty-first century for the footwear sector. This ambitious goal requires extensive support from intelligent information systems.

The problem considered in this article results from practical need. VIVE Textile Recycling is the leader of the textile recycling industry in Poland and Europe. The firm is regularly required to process massive deliveries of shoes. A conveyor belt transports the shoes to a darkroom, where their profile photos are taken (of the vamp, sole, and heel). Next, a management system, ShoeSelector consumes information from many external intelligent systems, and decides, for example, where the shoes should be thrown off of the conveyor belt, and whether they should be paired with their counterparts or remain single shoes. Shoe pairing is crucial for the shoes to be resold.

The primary goal of this article is to present a novel shoe-pairing method that can be used effectively in the process of analysing large quantities of shoes that are transported on a conveyor belt. We have verified various well-known classical approaches, including image descriptors like Central/Hu moments, HOG, and SSIM-based approaches, as well as a novel deep-learning-based method that outperforms others.

In our deep learning approach to shoe pairing, we introduced two sequential stages: a) multi-image shoe embedding that uses a deep neural network; and b) clustering of the shoes' embeddings with a fixed similarity threshold to return pairs (clusters in which size = 2) and singles (unpaired shoes). Each shoe in our pipeline is represented by three images (vamp, sole, and heel) that are collected in the darkrooms. Sample images are presented in Fig. 1. Our approach is evaluated on diverse custom industrial datasets, as provided by Vive Textile Recycling.

**Fig. 1.** Sample shoe images. The top row presents all three images (sole, vamp, and heel) for three distinct shoes. The bottom row presents the corresponding shoes found by the system.

### **2 Related Work**

Shoe pairing can be modelled as clustering of shoes' compact representations within a cluster (maximum size = 2) and a minimum similarity threshold to discriminate correct pairs (two-element clusters) against single shoes without counterparts.

Clustering is a fundamental task in the machine learning paradigm whose objective is to divide subjects into a number of groups in such a way that subjects in the same groups are more similar to each other and less similar to those in other groups. Traditional methods cluster subjects on the basis of a single set of features per subject [11].

Multiview clustering (MVC) is a variant of clustering in which each subject is represented by multiple sets of features [2]. MVC has been applied successfully to various applications, including computer vision, natural language processing, healthcare, and social media. Given that our project involves only images assigned to shoes, our focus lies in computer vision.

MVC has been used widely in image clustering [7] and motion segmentation [5] tasks. Chi et al. [3] conducted MVC for web image retrieval ranking. Xin et al. [14] successfully applied MVC for person reidentification. Typically, several feature types, such as HOG [4], LBP [10], and SIFT [9], can be extracted from images prior to cluster analysis.

Since 2012, deep learning has proved outstandingly efficient in a variety of applications, such as image classification, speech recognition, object detection, and natural language processing. Multiview deep clustering methods demonstrated better performance than traditional multiview shallow clustering methods. Most of the literature presents multiview deep clustering as the process of clustering on the representations obtained from multiview representation models built in a supervised manner [2].

Multiview representation learning (MVRL) has recently become popular by its exploitation of complementary information of multiple features or modalities. Recently, due to the remarkable performance of deep models, deep MVRL has been adopted in many domains, including computer vision and signal processing. One article [15] presents a comprehensive review of deep MVRL in two perspectives: (1) deep MVRL extensions of traditional methods; (2) MVRL methods in full deep learning scope. The first group of methods introduce the advancements of deep learning models into traditional MVRL methods, such as multiview canonical correlation analysis and matrix factorisation. The second group represents pure deep learning MVRL methods, such as multiview autoencoders, conventional neural networks, and deep belief networks [15].

The deep neural networks used to learn deep representations adopted in multiview clustering methods have superior expression ability to reflect multiview data comprehensively. Multiview deep representation learning is a method of transforming a collection of inputs (in our case, profile images) into compact, fixed-size, floating-point representations that exhibit the desired properties. Such embedding can be subject to further processing, such as clustering. However, the authors of [2] claim that the separate processes of representation learning and clustering entails limitations, such as representation learning being unaware of clustering's goal. Our approach resolves this problem by using learning representations that are tailored to the purpose of our clustering, which is pairing.

The outstanding efficiency of multiview deep-based clustering and its ability to make representation learning aware of the clustering's goal inspired us to develop a novel shoe-pairing process based on the clustering of representations obtained from a trained deep multiview representation model (hereinafter referred to as deep multiview embedding-based clustering, DMVEC).

## **3 The Proposed Approach**

The solution is a web service that is capable of responding to a variety of shoerelated requests via REST API. This article describes only the pairing aspect of the sytem. Answering pairing-related questions requires the system to perform a series of processing steps—some of which are shared with other tasks. The first (1) is input preprocessing (e.g. image decoding from base64 encoding, image scaling, and normalisation). Later (2), shoe detection is performed (empty hangers occasionally get photographed). Next, (3) deep multiview embedding of a shoe is performed (this step assumes that the embedding model is already trained and available for inference). Last (4), clustering of the shoes' embeddings into pairs can be requested. In the two sections below, we describe (3) and (4) in greater detail.

## **3.1 Deep Multiview Representation Learning**

The deep multiview embedding approach is a supervised method of changing the representation of a shoe into a single fixed-size vector based on its three images (shoe embedding). The process is based on a deep neural network, which is responsible for transforming the three images of a shoe into a compact vector of floating numbers. This transformation is trained in such a way that each shoe in a pair should be located close to the other, but away from unrelated shoes. The Euclidean distances between the embeddings can be used by the clustering method as a distance metric.

This method is a multibranch combination of convolutional layers and dense layers. The input comprises three photographs of a shoe (vamp, sole, and heel). The output is a single 64-dimensional unit floating vector. In the initial part of the proposed deep neural network, we used the convolutional base of the VGG-16 network [12], pretrained on the ImageNet dataset<sup>2</sup>. The results of the convolutional base are globally pooled by channels. The use of the convolutional base of the VGG-16 network in our method is an example of transfer learning. The nonconvolutional part of the network is trained from scratch by freezing the VGG-16 weights. The vector obtained as a result of global average pooling of the output of the convolutional base is processed using a fully connected layer. Such

<sup>2</sup> https://image-net.org.

**Fig. 2.** The deep multiview shoe embedding model. vgg preproc – a preprocessing layer in which images are converted from RGB to BGR, before each colour channel is zerocentered relative to the ImageNet dataset; vgg – the convolutional base of the original VGG-16 network pretrained on ImageNet; gap – global average pooling, the layer used for dimensionality reduction; dense - a dense or otherwise fully-connected layer; relu – an activation layer in which all negative values are replaced with zero and the remaining values are passed to the output; sum – a layer for aggregating the output by adding tensors from multiple layers; norm – a layer that normalises the input vector to a unit vector.

a network fragment, called a branch, is repeated between one and three times, depending on the number of image types desired. Then, all branches are aggregated over the corresponding indices (element-wise) and transformed once more using the fully connected layer. Last, the 64-dimensional output is normalised so that it lies on a multidimensional sphere with a unit radius. Figure 2 presents the entire architecture of the deep neural network. To train our embedding model, we used triplet learning (namely, the TripletSemiHardLoss and TripletHardLoss loss functions), in which each reference (anchor) shoe requires a positive example (a corresponding shoe from a pair) and a negative example (a nonsimilar shoe). The deep neural network is trained in a regime that forces similarity (e.g. Euclidean-distance-based similarity) between the representations of the reference and the compatible shoe, while reducing the similarity of the representation of the reference example and the negative example.

## **3.2 Clustering**

Given the trained deep multiview representation model (the shoe embedding model), a visually represented batch of shoes can be transformed into a set of embeddings. The most similar embeddings can then be identified to pair the shoes. The desired pairing should connect the elements so that the distance and, therefore, the differences in appearance—is as short as possible (or alternatively, similarity is as high as possible).

Although we evaluated many methods, considering the heavy deep multiview embedding model and specific nonfunctional requirements (such as the number of shoes in the clustered population being limited by the capacity of the conveyor belt, which is around 1,000–2,000 hangers), the most robust and comprehensive was the greedy method, which is based on agglomeration clustering with an additional termination condition. It works as follows:

## **4 Results**

In the initial phase of our experiments, we had limited access to labelled data. Approximately 1,000 hand-labelled pairs of shoes were split into training (80%) and test (20%) sets. Using these relatively small datasets, we evaluated various methods to identify the most promising one. At this point, we suspected that we using too little data to fully unlock the benefits of deep learning. We opted to use transfer learning, which is helpful in cases of scarce data.

We verified four unsupervised classical methods: Central and Hu moments, HOG, and SSIM. We used a special evaluation measure called pairing accuracy, which is the percentage of shoes that are correctly paired (or unpaired). We define pairing accuracy as the number of shoes with correct assignment (properly paired or properly unpaired) divided by the number of shoes considered.

**Table 1.** The quality evaluation of the shoe-pairing methods on the test set measured by pairing accuracy; VAMP/SOLE/BOTH indicates from which images the shoe-pairing model was inferring.

Table 1 presents the results of the shoe-pairing experiments on our initial test set. The first row presents the proposed DMVEC approach (deep multiview embedding-based clustering). The subsequent rows present the results obtained by our unsupervised classical baselines:

1. Central moments – the shoe images, separately or joined horizontally in one image, are described by image descriptors called central moments [6] before clustering is applied;

**Fig. 3.** Adjustment of the similarity threshold is a crucial task in shoe pairing. The image on the right shows the similarity between shoes in proper and incorrect pairs in a set of 34,000 pairs returned by a prototype system and next labelled by human taggers. The image on the left demonstrates how we tuned the hyper-param-similarity threshold using the 1,000-shoe validation set. We counted partial errors and accuracy (acc) for each threshold setting. The legend presents the errors as a ratio of TP/FP/TN/FN to all shoes in the paired set. *T hr* is represented by the vertical line that corresponds to the threshold value that reports the highest accuracy on the 1,000-shoe validation set.

The initial experiment revealed that DMVEC obtained the best results; however, the test set consisted solely of pairs i.e. all shoes in the test set had their counterparts within the test set. This situation is a borderline one; usually, the ratio of paired shoes in a real-world population of shoes under pairing is between 50% and 90%. The appearance of singles in a population demands the introduction of a threshold, which prevents further pairing and returns the remaining shoes as singles. We received a 1,000-shoe validation set from Vive Textile Recycling that represented the most common distribution of pairs in a population. In Fig. 3, we present how we tuned the similarity threshold value based on the validation set.

The next phases of the project involved conducting experiments on diverse test populations that contained significant proportions of singles. The results of the experiments revealed that accuracy reported a huge drop from an initial 99% to around 80–90% (lower scores when more singles were in the population;

**Fig. 4.** Training and validation (on the test set) metrics concerning our best pairing model. The model was trained using the maximum number of true pairs collected over two years of labelling efforts. We accumulated 108,000 shoes in pairs, and validated/tested after each epoch using 2,000 shoes, of which 1,000 were paired and 1,000 were singles (the x-axis represents epochs; the y-axis represents accuracy - left axis and loss score - right axis).

80% was reported when we had equal numbers of paired shoes and singles). After investigating the drop in pairing accuracy, we reached two conclusions. (1) Finding pairs in a population that does not contain singles is a significantly easier task than pairing a general population. The threshold is challenging to discover; embeddings are typically not separable by any single threshold and this introduces a tradeoff between precision and recall. (2) The difficulty of shoe pairing depends inherently on the size of the population. This differs from other tasks, like classification, in which scores are independent of validation set size. This happens because in pairing, examples in the dataset are not independent of each other. This can be also seen from the perspective of a solution space, in which a combinatorial explosion appears in the number of possible pairings.

We decided that our training dataset was insufficiently representative for deep multiview representation learning, and that a more robust embedding model was demanded. We manually labelled tens of thousands of pairs returned by a prototype system in a preindustrial environment. After more than a year of labelling, we had collected approximately 54,000 proper pairs. In Fig. 4, we present the training and validation process using our largest training dataset and data augmentation (cropping, saturation, and hue disturbance). The trained model is highly robust and reports high accuracy scores during clustering on the test set—even when it was tested on a very difficult set (1,000 shoes in pairs and 1,000 singles), it presents accuracy above 97% after the 10th epoch.

### **5 Conclusions**

This article presents a novel approach to shoe pairing, a dual-stage approach that comprises deep multiview shoe embedding and clustering. We evaluated different approaches to shoe pairing, from classical unsupervised ones based on image descriptors to the proposed supervised one that applies deep neural networks. The best-performing model in this task is the proposed supervised method, DMVEC. It reports almost 100% accuracy on the test sets that exclusively comprise shoes in pairs, and at least 97% when singles cover almost half of the test population. Over a broad number of tests, we evaluated different test sets (with different size and various distributions of singles). We also demonstrated how our selected method can be improved by hyperparameter tuning (similarity threshold tuning), massive increases in training data, or data augmentation.

In the future, we plan to conduct research on further optimisations for DMVEC. We suspect that multiple factors can impact the pairing accuracy of our model, such as the margin hyperparameter in triplet learning, the schedule of TripletHardLoss and TripletSemiHardLoss during training, or the GAP layer discarding too much information. There are also other necessary tasks apart from pairing, which will be addressed soon.

### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Representation of Observations in Reinforcement Learning for Playing Arcade Fighting Game**

Huaiyu Du<sup>1</sup> and Rafal J´o´zwiak1,2(B)

<sup>1</sup> Faculty of Mathematics and Information Science, Warsaw University of Technology, Warsaw, Poland<EMAIL_ADDRESS><sup>2</sup> Applied Artificial Intelligence Laboratory, National Information Processing Institute, Warsaw, Poland

**Abstract.** Reinforcement learning (RL) is one of three basic machine learning paradigms, alongside supervised learning and unsupervised learning. Reinforcement learning algorithms have become very popular in simple computer games and games like chess and GO. However, playing classical arcade fighting games would be challenging because of the complexity of the command system (the character makes moves according to the sequence of input) and combo system. In this paper, a creation of a game environment of The King of Fighters '97 (KOF '97), which implements the open gym env interface, is described. Based on the characteristics of the game, an innovative approach to represent the observations from the last few steps has been proposed, which guarantees the preservation of Markov's property. The observations are coded using the "one-hot encoding" technique to form a binary vector, while the sequence of stacked vectors from successive steps creates a binary image. This image encodes the character's input and behavioural pattern, which are then retrieved and recognized by the CNN network. A network structure based on the Advantage Actor-Critic network was proposed. In the experimental verification, the RL agent performing basic combos and complex moves (including the so-called "desperation moves") was able to defeat characters using the highest level of AI built into the game.

**Keywords:** Reinforcement learning · KOF '97 · Arcade fighting game · Python · pattern recognition · Artificial Intelligence · Neural Network · CNN · graphical representation · agent's reward system · transfer learning

## **1 Introduction**

Since DeepMind's AlphaGo defeated the world champion in the ancient GO game in 2015, with the rise of deep learning, more and more researchers have been dedicated to studying reinforcement learning (RL). One of the favorite RL study field is playing video games. Since "Street Fighter 2" was published on the arcade platform in 1991, fighting games have taken the world by storm, and the fighting genre has become an essential category in video games. Many researchers have studied applying RL techniques to fighting games such as the 2D game "Super Smash Bros" [1], 2.5D game "Little Fighter 2" [3], and 3D game "Blade & Soul" [5]. In [1], researchers used information such as player's location, velocity, and action state read from game memory as observations instead of raw pixels. They used two main classes of model-free RL algorithms: Q-learning and policy gradients, and both approaches were successful even though the two algorithms found qualitatively different policies from each other. After training the agent to fight against in-game AI, a self-play curriculum was utilized to boost the agent's performance, making the agent competitive against human professionals. In [3], since in 2.5D games, 3D objects in a scene are orthographically projected to the 2D screen, the main challenge is the ambiguity between distance and height. To tackle this problem, in addition to an array of 4-stacked, gray-scaled successive screenshots, the location information of both the agent and the opponent read from the environment was used as an observation. A CNN was used to extract visual features from the screenshots, and other information features were concatenated with the CNN's output. They also utilized LSTM in the network in order to enable the agent to learn proper game related and action-order related features. In [5], a novel self-play curriculum was utilized. By reward shaping, three different styles (Aggressive, Balanced, and Defensive) of agents were created and trained by competing against each other for robust performance. By introducing a range of different battle styles, diversity in the agent's strategies were enforced, and the agents were capable of handling a variety of opponents.

Compared with classical fighting games on the arched platform, such as the "Street Fighter" series and "The Kind of Fighters" series, the command system of games mentioned above is relatively simple: special moves are triggered either by combinations of direction keys and action keys or by dedicated keys. While in the game of KOF '97, the movement system is more complex. Except for basic moves such as a punch or a kick, special and desperation moves' commands consist of a sequence of joystick movements and button presses. A desperation moves' command is longer than special moves' and using a desperation move consumes a power stock indicated by a green flashing orb at the bottom of the screen. The movement and combo systems require from the RL agent to master the ability to complete relatively long sequences of accurate inputs.

To the best of our knowledge, till now, there is no RL research dedicated to arcade fighting games like KOF '97, which typically feature special moves and desperation moves that are triggered using rapid sequences of carefully timed button presses and joystick movements. The aim of the study described in this paper was to develop a RL agent which can not only win the game KOF '97 but also masters a complex game movement system. In order to achieve this goal, special attention was paid to the effective representation of observations in the RL environment.

## **2 Environment Setup for KOF '97**

#### **2.1 Interaction with Arched Emulator**

An arched game emulator called MAME is used to set up the python game environment. It is possible to drive MAME via Lua scripts externally, and the event hook allows interaction with the game at every step. An open source Python library called MAMEToolkit [4] encapsulates Lua scripts operation into Python classes. The KOF '97 game environment, which implements the Gym env interface, is built based on this python lib. Figure 1 shows a simplified communication process between the game environment and MAME emulator. Whole source code, including game environment, training, and evaluating, is available on GitHub<sup>1</sup>

**Fig. 1.** Simplified communication process

#### **2.2 Action Space**

There are eight basic input signals: UP, DOWN, LEFT, RIGHT, A, B, C, and D. The action space has 49 discrete actions, including all meaningful combinations of the basic input signals. One of the problem is that direction inputs in the commands are based on the character's orientation, i.e., if the character changes its orientation, all direction inputs in the commands should be flipped horizontally. Since the agent operates 1P's (Player 1) character, most of the time, the character is facing to the right side. It makes the training samples unbalanced. Moreover, the experience learned from the default orientation could not be applied to the other side. A trick of action flipping is used to tackle these problems. In the action space, FORWARD and BACKWARD commands are used instead of LEFT and RIGHT, and the environment will transfer FOR-WARD and BACKWARD to the LEFT or RIGHT according to the character's orientation. With the help of this trick, the agent's win rate increased to over 90% from 70% after 30 M timesteps' training.

<sup>1</sup> Source code address: https://github.com/duhuaiyu/AIbotForKof97.

#### **2.3 Observation**

The observation vector from a single step consists of several parts. The first part contains basic information (such as characters' HP, location, POW state, and combo count). All scalar features are normalized. The second part is the characters' ACT code, which represents the character's action. ACT is a concept in the game implementation mechanism. The character's ACT code and corresponding action can be checked in debug mode of the game. A character's move consists of one or more ACTs, and each ACT consists of several animation frames. As Figs. 2 shows, the character's "100 Shiki Oni Yaki" Move consists of 3 ACTs (Codes are 129, 131, and 133, respectively) which denote the start, middle, and end of the move. Each character can have a maximum of 512 ACTs. Although some of the ACT codes are not used, for compatibility, a 512 bits one-hot encoding vector is used to represent the ACT code feature. The basic information and characters' ACT code can be read from the game memory. The last part is the agent's input, which is also a one-hot encoding vector.

**Fig. 2.** "100 Shiki Oni Yaki" Move's ACTs

#### **2.4 Graphical Representation of the Input and ACT Sequences**

For readers can have a better understanding of the approach proposed below, it is worth to mention about the command judgment system of the game. As mentioned, to make a special move or desperation move, the player needs to finish a sequence of input in a short period. The game system would determine which move the character is going to do according to the input history. As human input varies in rhythm and precision, the judgment of input sequence has some flexibility. Take the move "hopping back" for example. The command is "← ←". As Fig. 3 shows, as long as the player can finish such an input pattern in 8 frames, the character can make the move successfully. In case 1 and case 2, although activated keys last for a different time, the input pattern "← ←" is finished in 8 frames, so the character can make the hopping back move. In case 3, such an input pattern is not finished in the time window, so the character failed to make the move.

The observation features from a single step are not enough to capture all important information since, on the one hand, the character's move highly

**Fig. 3.** Hop back command's input judgment

depends on the recent input history; on the other hand, according to the sequence of actions the characters have already made and for how long these actions have lasted, the player can make more accurate predictions and make an optimal response. In order to preserve the Markov's property, it is necessary to maintain the character's input sequence and ACT sequence within a specific time window. But simply concatenating features from several steps is not a good idea. Firstly, the feature vector for each step is long (over 1000 per step), and at least 30 frames' feature is needed since desperation move's commands are pretty long. It means more neurons for the network are needed, and training such a network would be slow and difficult. Besides, because of the command judgment system's flexibility, the input sequence can vary for the same move, and it is hard for an MPL network to find such a pattern. Our idea is to stack the 1P's input sequence, 1P's ACT sequence, and 2P's ACT sequence as three binary images. Figure 4 shows our approach to the representation of the feature sequences in the form of binary images, and these images encode the character's input and behavioural pattern, which are then retrieved and recognized by the CNN network. Although input sequences can differ for the same move, the input sequences will form a similar pattern on the graphic representation, and CNN is good at image pattern recognition.

#### **2.5 Reward System**

Based on the goal, the following reward system inspired by the reward system described in [1] is proposed:

$$R^{total} = R^{1P.damage} - P^{2P.damage} - P^{distance} - P^{time} - P^{POW}$$

where

$$\begin{aligned} R^{1P.damage} &= d^{1P} \ast c^{1P} \ast \gamma^{1P} \\ P^{2P.damage} &= d^{2P} \ast c^{2P} \ast \gamma^{2P} \\ P^{distance} &= Max(0, g - \beta) \ast \gamma^{distance} \\ P^{time} &= 1 \ast \gamma^{time} \\ P^{POW} &= \begin{cases} 5 & \text{if a POW orb consumed} \\ 0 & \text{otherwise} \end{cases} \end{aligned}$$

**Fig. 4.** Graphical representation of the input sequence and ACT sequences

The first two terms are a damage reward and a damage penalty (R1*<sup>P</sup> damage*, P2*<sup>P</sup> damage*). The priority is to win the game, so it is quite natural that if the character deals damage (d1*<sup>P</sup>* ), the agent gets a reward. In contrast, if the opponent deals damage (d2*<sup>P</sup>* ), the agent gets a penalty. In order to encourage the agent to perform more combo, the damage reward and penalty will be amplified linearly with the combo numbers(c1*<sup>P</sup>* , c2*<sup>P</sup>* ). On top of that, two factors: γ1*<sup>P</sup>* , and γ2*<sup>P</sup>* , are introduced to balance the reward and penalty caused by damage. To prevent the agent from behaving too aggressively, c2*<sup>P</sup>* is set to 1.3, and c1*<sup>P</sup>* is set to 1.

Except for the damage reward and penalty, some other terms are also introduced to adjust the agent's behavior. The distance penalty term P*distance* is used to make the agent keep a reasonable distance between the two characters. If the distance exceeds a certain threshold β, the agent will get a small penalty controlled by the factor γ*distance*. The term P*time* gives the agent a small penalty at every step. It is used to encourage the agent to deal damage and win the game as soon as possible. The last term in the formula is P *POW* . It gives a moderate penalty (specifically 5) when the POW orb is consumed, and if the desperation move can hit the opponent later, the reward will counter-weigh this penalty. Otherwise, the agent will be punished for using the POW orb for nothing.

#### **2.6 Proposed Network Structure**

Figure 5 shows the overall structure of our proposed network. The network's input consists of four parts: basic features as a vector, stacked input sequence (last 40 frames), stacked 1P characters' ACTs (last 36 steps), and stacked 2P characters' ACTs (last 36 steps). There are three CNNs to extract features from these images, and the results of these CNN extractors are concatenated with the basic features as the input of our Action Network and Value Network. Figure 5b shows the structure of CNN network. The proposed network is referred to as Multi-CNN in the following content.

**Fig. 5.** Proposed network structure

## **3 Experiment**

#### **3.1 Training Process**

The game's difficulty level (Built-in AI level) was set to 8, which was the hardest setting. Built-in AI refers to a programmed action strategy that is used to fight against players in this game. Even for intermediate-level gamers, winning a level 8 built-in AI is quite challenging. On top of that, to simplify the training process, the game model was changed from team-play to single-play. The chosen RL Algorithm was Proximal Policy Optimization [6]. The model's training was carried out on a personal computer with a 12-cores CUP and an RTX3060 GPU with 12 environments running in parallel.

### **3.2 Results**

**Multi-CNN Model and Transfer Learning.** Each model was evaluated for 50 episodes. The first model's agent operated the character Iori to fight against Kyo Kusanagi. As Table 1 shows, after 54 h of training, the first Multi-CNN model achieved a 100% win rate. A random agent was made as a negative baseline, and the win rate of the random agent is only 18%. Based on the first Multi-CNN model, using transfer learning, three more models were trained to fight with another three arbitrary chosen characters: Goro Daiman, Mai Shiranui, and Billy. After 30 M timesteps' training, as Table 1 shows, the transfer learning models also achieved similar performance. As Fig. 6 shows, the transform learning models' performance increases more rapidly than the one learned from scratch, meaning that the experience learned from the previous model can be transferred to new models. The agents can defeat their counterparts in a short time and have lots of HP remaining. Besides, our agents are able to perform more combos and desperation moves than the random agent. Overall the Multi-CNN models are more than capable of winning the game by a considerable margin. Sample videos of the Multi-CNN model agents are available on Youtube<sup>2</sup>.

**Table 1.** Test Result

**Fig. 6.** Mean rewards of Multi-CNN models and LSTM model

<sup>2</sup> Sample videos: https://youtu.be/v22Le-c9Uak.

**LSTM Network for Comparison.** The LSTM network is well-suited to tackle tasks based on time series data since the cells can remember values over arbitrary time intervals. In order to illustrate our proposed network's validity, an LSTM network was constructed for comparison. As Fig. 7 shows, the structure is similar to the Multi-CNN's. The only difference is that the multiple CNN extractors are replaced with an LSTM module. As Table 1 shows, given the same budget (50 M timesteps), the LSTM model can only achieve a 46% win rate. As Fig. 6 shows (the purple line), at the first 5 M steps of training, the performance increases steadily, then it begins to fluctuate without apparent improvement. Whereas in the Multi-CNN model, the reward gradually increases as the step increases. Figure 8 shows that more LSTM models with different numbers of hidden layers and hidden units were trained for 10 M timesteps, but none of these models was promising. Another observation is that the training process dramatically slowed as the number of hidden layers and units increased.

**Fig. 7.** LSTM network structure

**Fig. 8.** Mean reward over timesteps of more LSTM models

#### **3.3 Discussion**

Even though our agent can perform some simple combos and defeat the builtin AI by a large margin, the agent failed to learn advanced combo skills. It is because performing advanced combos needs a long sequence of accurate inputs at the right time. Such opportunities are rare unless the player intentionally creates them. Furthermore, at each step, our agent has many actions to choose from, so purely by random exploration, it is almost impossible for our agents to get such an experience. Some studies, such as [2,7], combined supervised learning and reinforcement learning to make the training process more effective. In the first stage, they used supervised learning to make the model imitate human operation, and in the second stage, they used RL to enhance their model. It would be a promising direction for further improvement.

Atari games are usually used as a testbed for RL algorithms, and the operation skill of such games is relatively low. Whereas in reality, manipulating things is often more complex and involves a sequence of operations with some patterns. The Recurrent Neural Network (RNN) is commonly used to process time series data. However, in such models, more recent data usually impact the results more, so it is not quite efficient for detecting behavioral patterns with some flexibility. The proposed graphical representation of sequences and network structure provides a new idea for detecting and simulating complex action patterns that are required by more practical tasks.

## **4 Conclusions**

In this paper, a lot of valuable experience from the previous research is applied to our task: playing the arched fighting game KOF '97. Based on the features of this game and our goal, a novel graphical representation of the input sequence and characters' action sequence is proposed. The input pattern and characters' behavior pattern are well extracted by the proposed Multi-CNN network. Besides, a contribution to the RL community has been made by adding a new game environment of KOF '97, which is one of the most iconic arcade fighting game.

The experiments show that the agent can not only win the game by a large margin but also learns to perform some combos and desperation moves according to the situation. They also show that the experience learned from fighting against one character can be transferred to fighting against other characters via transfer learning.

## **References**

1. Firoiu, V., Whitney, W.F., Tenenbaum, J.B.: Beating the world's best at super smash bros. with deep reinforcement learning. arXiv preprint arXiv:1702.06230 (2017)

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## AI4U: Modular Framework for AI Application Design

Kamil Wołoszyn<sup>1</sup> , K. Turchan<sup>1</sup> , M. Rąpała<sup>2</sup> , and K. Piotrowski1(B)

<sup>1</sup> IHP - Leibniz-Institut für innovative Mikroelektronik, Frankfurt (Oder), Germany<EMAIL_ADDRESS><sup>2</sup> CBK PAN - Centrum Badan Kosmicznych PAN, Zielona Gora, Poland<EMAIL_ADDRESS>https://www.ihp-microelectronics.com , https://cbkpan.pl

Abstract. This paper presents the concepts of a universal, modular framework that shall enable rapid development of AI applications. The goal of the conceptual tool is the versatility in terms of changing the environment, as well as integration of different types of sensors to perform a specific task. The framework follows the Sens4U approach that will facilitate the entire process of building the AI applications and simplify the testing process.

Keywords: AI *·* Artificial Intelligence *·* AI Application *·* Framework

## 1 Motivation

Artificial intelligence (AI) has grown in popularity in recent years. AI has been applied in many scenarios, from intelligent assistants to advanced real-time vision systems. AI enables devices to make decisions based on data collected from cameras or various sensors. This data is processed and the device responds correspondingly. Artificial intelligence (AI) is a part of computer science domain which builds systems or machines that in specific tasks can imitate human intelligence. They are also able to constantly improve their performance according to collected data. Artificial intelligence is often associated with humanoid robots that are believed to replace humans, but their real purpose is to increase human capabilities instead [1]. Artificial intelligence is divided by the mechanism of complexity.

Artificial Weak/ Narrow Intelligence is the type we are using nowadays. Most of current AI systems can be classified as narrow artificial intelligence. All software using technologies such as natural language processing, Machine learning, pattern recognition, data mining is considered as narrow artificial intelligence. Narrow artificial intelligence focuses mainly on tasks in which human beings can be outperformed.

Artificial Strong/General Intelligence is the concept of an intelligent system with a great deal of knowledge and cognitive ability, which will be able to independently perform tasks even those not yet known. It should think similarly to a person and with comparable or even better efficiency. At present, there is no machine that can understand the world as a human being can.

Artificial Super Intelligence is the most popular term circulating in the computer science world in recent times. An artificial super intelligence will hypothetically be able to interpret and to understand human behaviour and/or intelligence. It will be able to surpass human intelligence in mathematical tasks, science, medicine or applying solutions to different types of problems. What is more, artificial super intelligence is assumed to be able to evoke understanding, emotions or to be capable of having its own desires.

Artificial intelligence is an idea that integrates human intelligence with machines, as shown in Fig. 1. An issue related to artificial intelligence is ML (Machine Learning), which is one of the subcategories, that belongs to the area of artificial intelligence. Its main goal is to allow computers to learn using specific algorithms.

Fig. 1. Artificial intelligence structure.

This paper presents the concept of a universal, modular tool that will enable rapid construction of AI applications. The goal of this conceptual tool is its versatility in changing environment and different sensor types integration needed to perform a specific task. The concept of the proposed tool using the Sens4U approach [2], will improve the construction process, testability and reliability of the application. The idea of Sens4U is based on accelerating the application construction for non-experts in the field by using the modularity of available libraries.

Currently, there are many tools available for training deep neural networks, and many examples of software codes that support the sensors needed to create an intelligent robot or implementation of trained models. However, putting these together and making it fully function requires a large time investment.

When working on the SpaceRegion project, the initial focus was to build a stationary robot with no mobility capability. The robot is equipped with video cameras and servos to control the cameras, and its task was to observe and track detected objects. In the course of the work, it was decided that a mobile robot would be additionally built for the project. Initially the mobile robot was supposed to have the same functionality as the stationary robot, but during the code implementation to the mobile robot it turned out that most of the software code was the same, and only the code responsible for controlling the robot chassis has changed significantly.

This gave rise to the idea of building a universal tool that would speed up the process of developing AI applications. This approach can be also used to move one application from a particular scenario to another, for example a consumer solution to a space solution, without re-implementing the application from scratch. A universal tool must define application areas and requirements that simplify the migration process between scenarios. According to the Sens4U approach, the application area of the proposed tool will be expandable, by adding support for new sensors or updating those already implemented in the conceptual tool.

This paper is structured as follows. Section 2 discusses the related work, while Sect. 3 describes the proposed approach, followed by the example scenarios given in Sect. 4. Section 5 concludes the paper.

## 2 Related Work

At present, one can find systems that provide users with libraries and tools to help building robot applications, such as Robot Operating System (ROS) [3] or CubeSystem [4]. ROS is a meta operating system focused on building robots, which main goal is to support users in recycling of previously written code for robotics research or development purposes. ROS has a Gazebo [5] robot simulator that allows applications to be developed virtually, prior to the real robot, saving time and cost of modifications that might arise during construction. Another advantage of the ROS system is language versatility, thanks to which software can be written in such programming languages as Python [6], C++ [7], Lisp [8]. The big disadvantage of this system is the lack of multi-platform The stable version work only on Unix [9] systems, but there are also experimental versions for Windows system.

ROS is open source software which allows the community to create new solutions. CubeSytem on the other hand is a collection of hardware and programmable components that enable rapid prototyping of robots. CubeSystem in combination with the RobLib library [10] allows customization and linking to a distinct set of libraries that are tailored to a specific application. For example, the RobLib library contains general functions for controlling robots based on a two-wheeled differential drive. It is generally difficult to find systems or sets of libraries that have standardized functionality for sensors of the same type. The operation complexity level of the application is raised by a number of additional software, libraries and an adequate operation system that have to be installed. Fulling of such requirements can be problematic for a user unfamiliar with the field. Given the existing software and libraries, it seems reasonable to present the concept of a tool that is functional, modular, and cross-platform to support the construction of a complete application.

## 3 Proposed Approach

The concept of the tool for fast and modular construction of the application is shown in Fig. 2. The fact is almost every available sensor on the market has readymade hardware drivers as well as dedicated software. The aim of the proposed tool is to create software, which will be a set of dedicated libraries, but any introduced modification in the conceptual tool will be applicable on different sensors. As a result, it will always return the same output, assuming that all the requirements specified during the construction of the conceptual tool are met.

The idea for the proposed tool emerged during working on the development of a computer vision application. The programming language used to build the application is Python version 3. Python is a high-level programming language, and thanks to its simplicity and versatility, it has become an ideal tool for building AI systems.

Fig. 2. Concept of the proposed tool.

The base AI application that was created for the project had to observe the environment, recognise people in the same room and check from time to time whether the person is still in the same place. Based on the people present in the room, the robot estimated in which office it was located. The basic application was initially built on a stationary robot, which is the QBO ONE [11] produced by Thecorpora as shown in Fig. 3.

Fig. 3. Robot QBO.

Further, the plan was also to investigate the possibility of a smooth transfer of the application samples to more demanding environments (like space). In order to support that, the application should fulfil its tasks when the original sensors and actors are replaced by devices that fit the new environmental conditions. The application should thus use generic application programming interface (API) to apply the functionalities of the devices and for instance adapt to their physical features (video resolution of a camera module or the resolution of a motor turn).

Replacing a device with another one does not change the application implementation although it can change its internal behaviour and features. The API equips the application with the control over the device and provides information about its features. The application developer who uses the conceptual tool will be able to obtain a list of available functions for a given sensor. When building the tool, it would be necessary for the user to define the communication interface and to enter the parameters so it could work correctly.

For example, a user is building a robot with built-in DC motors. Using the API functionality of the conceptual tool, the user would be able to call, for example, the getFeaturesDCmotor() method. This method will return information to the user about parameters that should be entered and about available functions for DC motors e.g., turnForward(), turnBackward(), stop().

Another functionality of the tool concept is its modularity, which aims to support the seamless modification of the project. As an example lets take the modification of the mentioned robot. The user replaces the standard DC motors with stepper motors, which brings a number of changes to the created main application. However, when using the proposed conceptual tool, the only modification that the user should provide, would be possibly a name change of the used motors. The basic functionality would be consistent for both motors and the program operation after the motors replacement would not be disturbed.

What's more, after modifications, calling the getFeaturesSmotor () method for stepper motors will display additional functions, besides turnForward (), turnBackward (), stop() stepper motors can also have functions like turnForwardAboutAngle (x\_angle), turnBackwardAboutAngle (x\_angle) and turnLeft-AboutAngle (x\_angle), turnRight-AboutAngle (x\_angle), setStartPosition() while maintaining the requirements related to the motor mounting direction specified during the tool construction.

The proposed tool will allow expansion with new libraries depending on the users' needs. The main advantage of the conceptual tool would be reliability to environmental changes, and unification of functionality for sensors of the same type, what would result in rapid modifications in the design. During the construction of the tool, it will be possible to extend its operation to support different types of sensors. Moreover, sample functionalities such as object detection on the provided video stream or real-time object tracking from trained deep learning models can be implemented in the tool for artificial intelligence purposes.

For devices with low processing power, the proposed conceptual tool will be able to implement functionality to support AI accelerators. The AI accelerators will enable the satisfactory performance of artificial neural networks combined with computer vision. AI accelerator is, for example, the Intel Neural Compute Stick 2 [12]. One advantage of the proposed tool will be its cross-platform capability, which would allow the tool to run on both Microsoft Windows and Linux platforms.

## 4 The Application Area

This section presents examples of scenarios from which requirements for the correct operation of the proposed tool may arise. Some of the scenarios have been implemented, some are in the process of being implemented and some may be created in the future for development, experimental or research purposes during the development of the proposed conceptual tool. While creating the application, scenarios based on terrestrial solutions were used for testing reasons and finding available solutions. However, the conceptual tool also envisions applications in extraterrestrial scenarios.

## 4.1 Watchman Scenario

The first developed scenario is the Watchman scenario. The main task of this scenario is to monitor a specific room and control people authorized to be in it. The system based on optical sensor and machine learning must be able to detect, recognize and perform verification of a person present in the room. The next step that the created system must take is to react during the detection of a possible intruder. During the detection of an intruder, the task of the system is to take a picture of a person who is not authorized to be in the room and record the event in the database for further verification by the administrator. The system must be adapted to expand the database of people authorized to stay in the room.

## 4.2 Parkmonitor Scenario

The second scenario is the Parkmonitor scenario. The main task of this scenario is to monitor a specific area and count free and occupied parking spaces. System using optical sensor and machine learning must be able to detect and return information about number of free and occupied parking spaces.

## 4.3 Tracker Scenario

The third scenario is the Tracker scenario. The main task in this scenario is to track a specific object by an unmanned craft (drone). For example, such an object can be a ball, a cyclist, a person or a car. In general, any model of object or objects that can be trained using neural networks suit this scenario. The system, based on the optical sensor and machine learning, after activating the function, must be able to follow a specific object until it disappears from the frame or the function is disabled. The drone operator must be able to activate and deactivate the tracking function.

## 4.4 Mapping the Environment Scenario

The fourth scenario is the Mapping the Environment scenario. The main task in this scenario is to build a map of the area on which the mobile robot moves. Created system has to enable real preview of the built map and must have the ability to export the created map of space/area. Moreover, the system using an optical sensor and machine learning has to be able to mark certain objects on the built map. Additionally, the system must have the possibility to choose the option of controlling the vehicle. The first option is to drive autonomously to build a map of the room. The second option after building the map is to drive to a specific point on the map and the last required option is manual control of the vehicle. A prototype of the mobile vehicle under development is shown in Fig. 4.

Fig. 4. Mobile vehicle prototype.

## 4.5 Vehicle Counting

The fifth is the scenario of Vehicle counting. The main task of this scenario is to build a system capable of counting vehicles. Vehicles that will be counted are wheeled vehicles such as: cars, trucks and motorcycles. Counting of vehicles has to be done separately in two directions for vehicles approaching and moving away from the device. Additionally, the device on which the created system will operate must be resistant to external weather conditions and it should be as small as possible. Prototype of vehicle counting device under development is shown in Fig. 5. After installing and testing the device, the system will transmit the vehicle counting results via a network to the middleware. Vehicle counting device prototype consists of Raspberry Pi 4, Raspberry Pi High Quality Camera, 6mm 3MP Wide-Angle Lens and Intel Neural Compute Stick 2.

## 4.6 Space Surveyer

The sixth scenario is the space scenario of the Space Surveyer. The main task in this scenario is for the mobile robot to explore the terrain and determine the landing site of the lander. The created system must allow communication

Fig. 5. Vehicle counting device prototype.