**4. Advantages of g-C3N4 materials in photocatalyst**

The advantages of photocatalysts with varied morphologies are obvious. Twodimensional photocatalysts made in a specific way, for example, have a bigger specific surface area and more reactive sites. Other decorative components introduced throughout the construction process can also be used. However, there are still certain issues, such as a tight reaction temperature requirement and uneven particle distribution.

#### **4.1 Doping of elements**

The use of element doping to change the band structure of photocatalysts and enlarge the light absorption zone is an effective method [9]. Non-metallic (B, O, S, I) doped g-C3N4 has been widely utilized to alter the optical absorption of internal bandgap structure and to adjust the photocatalytic performance of g-C3N4 to promote the redox potential in visible light [10–14]. The light absorption range of nanosheets, on the other hand, reduces due to the quantum confinement effect, which increases the band

gap [15, 16]. As a result, we can apply elemental doping to change the band structure of g-C3N4, which improves both light absorption and charge separation [17]. Oxygen doping, boron doping, phosphorus doping, and halogen doping will all be investigated.


From all results of comparison between expected and reported emotions, we can see that the reported emotions are not similar to the expected emotions, which are the annotated emotions of the video stimuli. There are several possible causes; for instance, some subjects might be more sensitive to feel sad when they saw some stories. That is, some subjects can feel very sad, whereas other subjects can feel little sad, not sad and not happy (neutral), or feel happy when they saw the negative movie. This phenomenon is normal since different people have different emotion perception. This explanation is also valid for positive movie. Although the annotated emotion of this movie is positive, some subjects can feel happy while other subjects feel sad. Another possible reason is that, sometimes, the subjects cannot fully understand the contents of the movies because they watched the movies and walked at the same time. Therefore, subjects need to concentrate on walking in addition to watching movies. This makes some subjects cannot completely understand the content of the movies, so the reported emotion is opposite from the desired emotion we want them to feel. Other possible explanation is that some components or stories in the positive movie can make some subjects feel sad. For example, some music or video scenes might be very intense and some subjects are sensitive to these contents. This explanation is still related to the previous one. Additionally, individual preferences are also importance issues that should be considered. For example, if some subjects do not like sci-fi movie that we used as the positive stimuli, that movie can make them feel sad because this movie is the kind they do not like. Music soundtrack of the movies is also the reason that makes the perceived emotions different from the emotion we expected. If the subjects like the music, they can feel happy even the movie is negative movie, and if the subjects do not like the music, they can feel sad even the movie is positive movie. Lastly, if the subjects did not feel well when they watch the movie during walking, for example, some subjects feel motion sickness, or some subjects feel bored, the perceived emotions will be inaccurate and different from the emotions we expected. Because of this reason, we asked the subjects whether they can walk naturally or not after they finished walking.

In the first dataset, we did not have this information. But for the second dataset, seven subjects answered that they can walk naturally, eight subjects answered that they cannot walk naturally, and eight subjects answered they are unsure. If we

consider their explanation, the subjects who answered they can walk naturally have positive feedbacks. The followings are some examples.

• 映像に集中していたから

Because I was concentrating on the video

• 歩きにくさを感じなかったから

Because I did not find it difficult to walk

• 飽きなかったから!

I did not get tired of it!

Unfortunately, the subjects who answered that cannot walk naturally have negative feedbacks with walking while watching videos on HoloLens 2. These are some examples.

• 音や映像に気を取られたから

Because I was distracted by the sound and image

• 途中でフラついたり、まっすぐ歩けなかったりしたからです。

Because I was fluttering on the way and I could not walk straight

• 映像に集中していて、たまに枠線を超えそうになったから。

Because I was concentrating on the video and sometimes I almost crossed the border.

• 歩く範囲が小さいため

Because the walking range is small

Even some subjects answered that they feel unsure, some feedbacks are also negative.

• 時々眠たかったから

Sometimes I wanted to sleep

• 映像を見ながら歩くのが少し難しかった

It was a little difficult to walk while watching the video

• 枠外には出なかったが、円状を単調に歩いていたので、目がくらんで、不自然 に歩いていたかもしれないから。

I did not go out of the frame, but I was walking monotonously in a circle, so I might have been dazzled and walked unnaturally.

There are some positive feedbacks from unsure subjects. These answers show that they are walk unconsciously so we can collect the very natural walking styles.

• 自分がどう歩いていたか気にしていなかった

I did not care how I was walking

• 何も考えていなかったため、自然だったかはわからないです

I did not think about anything, so I do not know if it was natural

From these answers, we can see that some subjects have difficulties with watching videos on HoloLens 2 while walking at the same time. These are very reasonable explanations why emotion induction is not effective, and the reported emotions are much different from expected emotions. In the first dataset, we did not collect these data, hence, we cannot know that the subjects can walk naturally or not, and we cannot know how they feel after watching videos on HoloLens 2 while walking, but for the second dataset, we have these data, and they are very useful.

By using Microsoft HoloLens 2 for displaying emotional videos, there are several things we found and worth for consideration.

	- The subjects need to pay too much attention with the movie, so some subjects cannot walk naturally, some subjects have motion sickness, and some subjects feel bored because the movies are too long.
	- If subjects decide to give priority to walking more than watching movies, some subjects cannot fully understand the movie content.
	- Showing short movies instead of full movie or using some animation as mixed reality agents (VR/AR) should be better ideas to induce real-time emotion of the subjects.

#### **5. Conclusion**

To summarize, this study extends our previously proposed emotion induction and data collection method [34]. In conventional emotion induction, emotional videos are shown to the subjects before walking using a computer screen or television. In our method, emotional videos are shown on Microsoft HoloLens 2, which is the latest smart glasses technology. We found that displaying emotional videos on HoloLens 2 while walking can make the subjects express emotions on their gaits unconsciously while walking. Subjects can also see the room environment and the stimuli contents on HoloLens 2 at the same time. Some subjects think it is easy to walk while watching videos on HoloLens 2, while some subjects say that it is difficult to focus on waking and paying attention with video contents on HoloLens 2 at the same time. Our goal of this study is to simulate a real-time emotion while walking, however using full-length movies may not be a good idea because there are some negative feedbacks from our participants. For the walking direction, using a non-straight walking path will make the data collection become more realistic since capturing human gaits in reality is difficult to capture only straight and clean walking data. Therefore, if the emotion recognition system was developed and tested using non-straight walking gait data, opportunities to deploy this system in real-world scenario are increased. Additionally, expected emotions, which are the annotated emotion of the stimuli, should not be used to tag a walking trial since the stimuli emotions can be opposite to the emotions that the subjects perceived. Asking the subjects to report their actual feelings after walking is the best way we can do for now. In this study, OptiTrack motion capturing system was used for capturing gait data, but it is not mandatory to use marker-based systems such as OptiTrack or Vicon. Using marker-less motion capturing device, for example, Microsoft Kinect is also possible. Even a standard video camera or mobile phone camera with pose-estimation software such as OpenPose can also be used to capture body movement data for emotion recognition by gait analysis. In summary, this study investigates the possibility of performing emotion recognition and analysis by using smart glasses to induce emotions of subjects. The results show that emotion recognition from human gaits can be performed and is very useful in many circumstances. Since emotion recognition is an example of tangible applications of intelligence video surveillance, methods for inducing human emotions should be also considered. To develop an effective emotion recognition system as a part of intelligence video surveillance, obtaining a highquality dataset is an important factor that should be focused on.

#### **6. Acknowledgment**

The author would like to thank all participants who joined our both experiments. Additionally, we appreciated the help and support from all members of Kaoru Sumi Laboratory at Future University Hakodate, who supported and assisted in both experiments such as experiment venue setup, equipment setup, experimental design, translation of all documents, and interpretation between Japanese and English during the entire experiments processes.

#### **Funding statement**

This work was supported by JST Moonshot R&D Grant Number JPMJMS2011.

*Methods for Real-time Emotional Gait Data Collection Induced by Smart Glasses in a Non… DOI: http://dx.doi.org/10.5772/intechopen.107410*

### **Author details**

Nitchan Jianwattanapaisarn1,2, Kaoru Sumi1 \* and Akira Utsumi2

1 School of Systems Information Science, Future University Hakodate, Hokkaido, Japan

2 Interaction Science Laboratories, Advanced Telecommunications Research Institute International, Kyoto, Japan

\*Address all correspondence to<EMAIL_ADDRESS>
© 2022 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

### **References**

[1] Picard RW. Affective Computing. MIT press; 2000

[2] Tiam-Lee TJ and Sumi K. Analysis and prediction of student emotions while doing programming exercises. In: International Conference on Intelligent Tutoring Systems. Springer. 2019. pp. 24–33

[3] Bouchrika I. A survey of using biometrics for smart visual surveillance: Gait recognition. In Surveillance in Action. Cham: Springer; 2018. pp. 3-23. DOI: 10.1007/978-3-319-68533-5\_1

[4] Anderez DO, Kanjo E, Amnwar A, Johnson S, Lucy D. The rise of technology in crime prevention: Opportunities, challenges and practitioners perspectives. 2021. arXiv preprint arXiv:2102.04204

[5] Montepare JM, Goldstein SB, Clausen A. The identification of emotions from gait information. Journal of Nonverbal Behavior. 1987;**11**(1):33-42

[6] Khamsemanan N, Nattee C, Jianwattanapaisarn N. Human identification from freestyle walks using posture-based gait feature. IEEE Transactions on Information Forensics and Security. 2017;**13**(1):119-128

[7] Limcharoen P, Khamsemanan N, Nattee C. View-independent gait recognition using joint replacement coordinates (jrcs) and convolutional neural network. IEEE Transactions on Information Forensics and Security. 2020;**15**:3430-3442

[8] Limcharoen P, Khamsemanan N, Nattee C. Gait recognition and reidentification based on regional lstm for 2-second walks. IEEE Access. 2021;**9**:<PHONE_NUMBER>68

[9] Kitchat K, Khamsemanan N, Nattee C. Gender classification from gait

silhouette using observation angle-based geis. In: 2019 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM). IEEE. 2019. pp. 485–490

[10] Isaac ER, Elias S, Rajagopalan S, Easwarakumar K. Multiview gait-based gender classification through pose-based voting. Pattern Recognition Letters. 2019;**126**:41-50

[11] Janssen D, Schöllhorn WI, Lubienetzki J, Fölling K, Kokenge H, Davids K. Recognition of emotions in gait patterns by means of artificial neural nets. Journal of Nonverbal Behavior. 2008;**32**(2):79-92

[12] Roether CL, Omlor L, Christensen A, Giese MA. Critical features for the perception of emotion from gait. Journal of Vision. 2009;**9**(6):15-15

[13] Karg M, Kühnlenz K, Buss M. Recognition of affect based on gait patterns. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics). 2010;**40**(4):1050-1061

[14] Barliya A, Omlor L, Giese MA, Berthoz A, Flash T. Expression of emotion in the kinematics of locomotion. Experimental Brain Research. 2013;**225** (2):159-176

[15] Venture G, Kadone H, Zhang T, Grèzes J, Berthoz A, Hicheur H. Recognizing emotions conveyed by human gait. International Journal of Social Robotics. 2014;**6**(4):621-632

[16] Li B, Zhu C, Li S, Zhu T. Identifying emotions from non-contact gaits information based on microsoft kinects.

*Methods for Real-time Emotional Gait Data Collection Induced by Smart Glasses in a Non… DOI: http://dx.doi.org/10.5772/intechopen.107410*

IEEE Transactions on Affective Computing. 2016;**9**(4):585-591

[17] Li S, Cui L, Zhu C, Li B, Zhao N, Zhu T. Emotion recognition using kinect motion capture data of human gaits. PeerJ. 2016;**4**:e2364

[18] Zhang Z, Song Y, Cui L, Liu X, Zhu T. Emotion recognition based on customized smart bracelet with built-in accelerometer. PeerJ. 2016;**4**:e2258

[19] Chiu M, Shu J, Hui P. Emotion recognition through gait on mobile devices. In: 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). IEEE. 2018. pp. 800–805

[20] Quiroz JC, Geangu E, Yong MH. Emotion recognition using smart watch sensor data: Mixed-design study. JMIR Mental Health. 2018;**5**(3):e10153

[21] Xu S, Fang J, Hu X, Ngai E, Guo Y, Leung V, et al. Emotion recognition from gait analyses: Current research and future directions. arXiv preprint arXiv: 2003.11461. 2020.

[22] Lemke MR,Wendorff T, Mieth B, Buhl K, Linnemann M. Spatiotemporal gait patterns during over ground locomotion in major depression compared with healthy controls. Journal of Psychiatric Research. 2000;**34**(4–5):277-283

[23] Michalak J, Troje NF, Fischer J, Vollmar P, Heidenreich T, Schulte D. Embodiment of sadness and depression gait patterns associated with dysphoric mood. Psychosomatic Medicine. 2009; **71**(5):580-587

[24] Gross MM, Crane EA, Fredrickson BL. Effort-shape and kinematic assessment of bodily expression of emotion during gait. Human Movement Science. 2012;**31**(1):202-221

[25] Destephe M, Maruyama T, Zecca M, Hashimoto K, Takanishi A. The influences of emotional intensity for happiness and sadness on walking. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE. 2013. pp. 7452-7455

[26] Sun B, Zhang Z, Liu X, Hu B, Zhu T. Self-esteem recognition based on gait pattern using kinect. Gait & Posture. 2017;**58**:428-432

[27] Halovic S, Kroos C. Not all is noticed: Kinematic cues of emotion-specific gait. Human Movement Science. 2018;**57**: 478-488

[28] Sadeghi H, Allard P, Duhaime M. Functional gait asymmetry in ablebodied subjects. Human Movement Science. 1997;**16**(2-3):243-258

[29] Kang GE, Gross MM. Emotional influences on sit-to-walk in healthy young adults. Human Movement Science. 2015;**40**:341-351

[30] Kang GE, Gross MM. The effect of emotion on movement smoothness during gait in healthy young adults. Journal of Biomechanics. 2016;**49**(16): 4022-4027

[31] Kim S, Nussbaum MA, Ulman S. Impacts of using a head-worn display on gait performance during level walking and obstacle crossing. Journal of Electromyography and Kinesiology. 2018;**39**:142-148

[32] Sedighi A, Ulman SM, Nussbaum MA. Information presentation through a head-worn display ("smart glasses") has a smaller influence on the temporal structure of gait variability during dualtask gait compared to handheld displays (paper-based system and smartphone). PLoS One. 2018;**13**(4):e0195106

[33] Sedighi A, Rashedi E, Nussbaum MA. A head-worn display ("smart glasses") has adverse impacts on the dynamics of lateral position control during gait. Gait & Posture. 2020;**81**: 126-130

[34] Jianwattanapaisarn N, Sumi K. Investigation of real-time emotional data collection of human gaits using smart glasses. Journal of Robotics, Networking and Artificial Life. 2022;**9**(2):159-170. DOI: 10.57417/jrnal.9.2\_159

[35] Baveye Y, Dellandréa E, Chamaret C, Chen L. Deep learning vs. kernel methods: Performance for emotion prediction in videos. In: 2015 International Conference on Affective Computing and Intelligent Interaction (acii). IEEE. 2015. pp. 77–83

#### **Chapter 9**

## Combining Supervisory Control and Data Acquisition (SCADA) with Artificial Intelligence (AI) as a Video Management System

*Muhammad H. El-Saba*

#### **Abstract**

The latest Video management systems (VMS) software relies on CCTV surveillance systems that can monitor a larger number of cameras and sites more efficiently. In this paper, we study the utilization of SCADA to control a network of surveillance IP cameras. Therefore, the video data are acquired from IP cameras, stored and processed, and then transmitted and remotely controlled via SCADA. Such SCADA application will be very useful in VMS in general and in large integrated security networks in particular. In fact, modern VMS are progressively doped with artificial intelligence (AI) and machine learning (ML) algorithms, to improve their performance and detestability in a wide range of control and security applications. In this chapter, we have discussed the utilization of existing SCADA cores, to implement highly efficient VMS systems, with minimum development time. We have shown that such SCADA-based VMS programs can easily incubate AI and deep ML algorithms. We have also shown that the harmonic utilization of neural networks algorithms (NNA) in the software core will lead to an unprecedented performance in terms of motion detection speed and other smart analytics as well as system availability.

**Keywords:** SCADA, distributed control systems (DCS), security, artificial intelligence (AI), face recognition, machine learning, video surveillance video analytics

#### **1. Introduction**

The Supervisory Control and Data Acquisition (SCADA) is a software overlay application, which is used on top of intelligent control networks. The control nodes are traditionally smart sensors and programmable logic controllers (PLCs) [1–3]. In fact, the SCADA industry started due to the need for a user-friendly front-end to control systems containing smart devices, devices, and PLCs. However, SCADA systems evolved rapidly and are now penetrating the reliable operation of modern infrastructures [2, 3] of smart cities. SCADA systems have made substantial progress over the recent years to increase their functionality, scalability, performance, and openness. As shown in **Figure 1**, the main components of a SCADA system are as follows:

Indeed, it is possible to purchase a SCADA system from a single supplier or tailor a SCADA system from different manufacturers, such as Siemens and Allen-Bradley PLCs. This chapter presents a method based on neural networks (NN) for monitoring and operating video surveillance systems (VMS), like those in traffic control networks and electronic plaza sites. The method suggests that the thresholds used for generating alarms can be adapted to each surveillance device (e.g., IP Camera). The intelligent SCADA method was actually utilized in other application fields, for example, in electrical power control and renewable energy systems [3].

In this chapter, we show how to exploit such existing SCADA programs to implement a wide-area video management systems (VMS), which incorporate state-ofthe-art AI technologies, such as access control, intrusion detection, face recognition, license plate recognition, crowd detection, and city surveillance. These technologies have been implemented in our emerging VMS, Xanado [4], which is expected to have a unique value in identifying criminals and terrorists, patrolling highways, and in aiding forensics.

Such SCADA solutions are multi-tasking and are based upon a real-time database that is located on dedicated servers of the system. Such SCADA servers are responsible for data acquisition and handling (e.g., data polling, alarm checking, logging verification, and data archiving) on the basis of a set of chosen parameters.

#### **Figure 1.**

*Conventional Architecture of a SCADA system, The Master Station refers to the servers and software responsible for communicating with Remote Terminal Units (RTUs), such as PLCs.*

*Combining Supervisory Control and Data Acquisition (SCADA) with Artificial Intelligence… DOI: http://dx.doi.org/10.5772/intechopen.104766*

#### **2. SCADA-based video surveillance solutions**

One of the main advantages of SCADA systems is that it allows operators to visualize, in real time, what is happening in any particular industrial process, react to alarms, control processes, change configurations, and track information in real time. However, SCADA systems differ from distributed control systems (DCSs) that are generally found in industrial plant sites. While a DCS covers a plant site, a SCADA system covers much larger areas. Similarly, wide-area video monitoring requires large-scale monitoring systems, like those of SCADA networks. For operations that span several sites, it is important to have a central monitoring station that acts as eyes and ears across all sites. Central monitoring stations use diverse types of cameras and sets of technology to monitor and protect people and property, especially when personnel cannot be on site. It is important that these technologies work together to create a holistic monitoring system. Fortunately, SCADA architecture supports TCP/IP (Internet Protocol), UDP, or other IP-based communications protocols, which makes it ideal for video surveillance control with a network of IP cameras. In fact, SCADA systems have traditionally used combinations of direct serial buses or Ethernet or Wi-Fi connections to meet communication requirements, as well as IP over SONET (Synchronous Optical Network) at large sites. **Figure 2** depicts the architecture of SCADA-based VMS programs, which are employing neural network algorithms (NNA). The employed NNA aims to improve the control of the system by using an iterative supervised process. The objective is to determine and optimize the SCADA-VMS control parameters, for specific sites with specific surveillance devices. The chosen parameters, such as the favorite angles of PTZ cameras, the detection speed (of motion anomalies), and false arm causes, will help to increase the surveillance performance and system availability. In addition, the optimized system will minimize false alarms in a continuous adaptive manner, according to each site-specific equipment. Actually, the NNA is based on finding differences in the behavior of the surveillance system over time. The iterative process starts with a database of the stored SCADA-VMS database, as shown in **Figure 2**.

The SCADA-based VMS programs, with NNA, can help in this context and can easily incorporate the following features and intelligent analytics:

#### **2.1 Object detection and tracking**

The process of identifying objects in an image and finding their position is known as object detection. **Figure 3** depicts the object detection and identification tasks. This activity benefitted a lot from the field of computer vision assisted by AI. As shown, the trained model using deep learning must be evaluated for its performance on some data called as test dataset' [5–9].

#### **2.2 Face recognition**

Face identifiers offer advantages in access control, safety, security, retail stores, and traffic control. Facie recognition is actually an analytic program that identifies persons from their facial features in an image or video surveillance.

Face recognition programs are usually utilizing AI to quickly identify and interpret complex figures. When comparing a face image to a database of previously stored images of known faces, the AI algorithm can then determine the best match. In fact, the face recognition and analysis algorithms have enabled security systems to capture many wanted criminals and stop many crimes.

As facial recognition increases in efficacy, the number of other applications will also increase (e.g., in banking, retail stores, and means of transportation). Thanks to deep learning-based AI algorithms, face analysis is not only able to identify with high accuracy, but also provides extraordinary analytic capabilities. For instance, it can now detect the criminal behavior, from his/her mood and gests. Thanks to 3D/4D digital signal processing (DSP), with AI, face recognition technology will be expanding in identifying terrorists and criminals' actions, by incorporating motion detection algorithms.

#### **2.3 Automatic number plate recognition (ANPR)**

The ANPR is analytic software that reads vehicle plates and automatically matches them to recognized vehicle license plates, without the need for human

**Figure 3.**

*Block diagram of object detection and identification y artificial intelligence.*

*Combining Supervisory Control and Data Acquisition (SCADA) with Artificial Intelligence… DOI: http://dx.doi.org/10.5772/intechopen.104766*

intervention. Therefore, ANPR offers accurate identification and safety of vehicle access and traffic control. ANBR is usually implemented using optical character recognition (OCR) and convolutional neural networks (CNN). Actually, CNN is a widely used neural network architecture for computer vision tasks. The CNN automatically extracts important features on images. More details about CNN are provided below in Section 4.

#### **3. Artificial intelligence in video surveillance and video analytics**

Artificial intelligence (AI) is nowadays revolutionizing video management systems (VMS) and the ways of securing smart premises and smart cities by video surveillance and control. In fact, AI-based technology can promote security and surveillance equipment by enhancing object detection and motion interpretation as well as providing analytics with increasingly reliable data. For instance, the reduction of false alarms in security systems is one of the major benefits of AI-based tools and algorithms. For instance, AI-CCTV cameras are networked IP cameras that deliver advanced analytical functions, such as face recognition, vehicle classification, car counting, license plate recognition (LPR), and other traffic analytics. Advanced video analytics software is built into the camera and recorder, which then enables artificial intelligence functions. Some AI algorithms are rule-based and others are self-learning. Like typical CCTV cameras, AI-CCTV stores information so any incidents can be reviewed. However, AI CCTV can detect and send alerts in real time. In particular, SCADA legacy can help a lot in large-sites and wide-area VMS systems.

The artificial intelligence (AI) tools are heavily dependent on neural networks (NN) and computer vision [5]. As shown in **Figure 4**, a neural network (NN) is a system of software or hardware, which mimics the operation of human brain neurons. Therefore, an NN is simply a group of interconnected layers of a perceptron. Note that an NN has multiple hidden layers, and each layer has multiple nodes. The neural network takes the training data in the input layer and forwards it through hidden layers, on the basis of specific weights at each node [10]. Therefore, it returns an output value to the output layer. The inputs to nodes in a single layer have adaptable weights that affect the final output prediction.

There are a lot of different kinds of neural networks that are used in machine learning projects. There are recurrent neural networks, feed-forward neural networks, and convolutional neural networks (CNNs). It can take some time to properly tune a neural network to get consistent, reliable results. Testing and training your NN is important before deciding which parameters (input features of a face image) are important in your recognition model.

**Figure 4.** *Schematic diagram of a neural network (NN) and how it works.*

**Figure 5.**

*Schematic diagram of the layers of a convolutional neural network (CNN) showing its classification sequence of a handwritten character (in ASCII).*

#### **4. Deep learning-based video surveillance solutions**

In deep learning, a convolutional neural network (CNN) is a sort of NNs; which is particularly useful in video surveillance projects. The advantage of deep learning- (DL)-based algorithms with respect to legacy computer vision algorithms is that DL systems can be continuously trained and improved with updated datasets.

In DL, a convolutional neural network (CNN) is a type of NN, commonly used in image recognition and processing, with emphasis on machine vision of images and video. As shown in **Figure 5**, the layers of a CNN consist of an input layer, an output layer, and a hidden layer that includes multiple convolutional layers, pooling layers, fully connected layers, and normalization layers.

Deep learning systems have shown a remarkable ability to detect undefined or unexpected events. This feature has the true potential of significantly reducing false alarm events that happen in many security video analytics systems. Many applications have shown that deep learning systems can "learn" to achieve 99.9% accuracy for certain tasks, in contrast to rigid computer algorithms where it is very difficult to improve a system past 95% accuracy [4].

#### **5. Xanado program**

Xanado is a video management system (VMS) software, designed for large-scale and high-security installations. It is built as a client-server DCS to ensure end-to-end protection of video integrity and boost the overall performance of existing hardware. In addition to central management of all data servers, IP cameras, and users in a multi-site setup, Xanado includes an integrated video wall size for operators demanding overall awareness of any event. The software supports failover recording servers making it the perfect choice for mission-critical installations that require continued access to video recording in case of a server failure. Xanado is ideal for installations with 24/7 operation requirements and can run on high-speed recording engines (NVR), making it suitable for monitoring airports, banks, traffic control, as well as smart city surveillance.

The general system architecture is shown in **Figure 6**. As shown in Figure, the management data server lies in the center of the VMS. It holds the main application and handles the system configuration. Note that the recording server is responsible *Combining Supervisory Control and Data Acquisition (SCADA) with Artificial Intelligence… DOI: http://dx.doi.org/10.5772/intechopen.104766*

**Figure 6.** *Xanado general system architecture.*

for all communication, recording, and event handling related to devices, such as cameras and I/O modules. The system stores the video in a customized database. The management server, event server, and log server use an SQL server to store configuration, alarms, and log events. As the VMS is designed for a large-scale operation, the Management Client may run locally or remotely, for centralized administration.

The smart client nodes (SCN) are working as follows. SCN connects to the management server and attempts to log in. The management server tries to authenticate the user and the user-specific configuration is retrieved from the SQL database. Therefore, the login is granted and the configuration is sent to the SCN. Live video streams are then retrieved from cameras by the recording server. The recording server sends a multicast stream to the multicast-enabled network. This requires that all switches handling the data traffic between the SCN and the recording server must be configured for multicast.

We adopt the ONVIF standard [11] for full video interoperability in multi-vendor installations to ensure information exchange by a common protocol. The ONVIF protocol profiles are collections of specifications for interoperability between ONVIF compliant devices, such as cameras and NVR.

Of course, Xanado VMS has many add-on modules and can be tailored to several specific applications. In the following subsection, we describe an application of our VMS to the case of Electronic Toll Plaza control, which is utilized nationwide in highway traffic control.

#### **6. Case study: toll plaza control**

Toll plazas are utilized on traffic highways to collect fees from passing vehicles. The toll plaza consists of six zones—an approach zone, queue area, toll lanes, the toll island, departure zone, a bailout lane, and, in some cases, a terminal supervisor. The so-called Electronic Toll Collection (ETC) enables toll collection without delay or total stop of vehicles. This section deals with the equipment and software to be

**Figure 7.** *Schematic of a booth in an electronic toll office, which communicates with a passing vehicle.*

installed on the roadside of toll plaza networks. We shortly depict the basic concepts for installing the system of electronic toll collection. **Figure 7** is a schematic of a single booth and a lane of an electronic toll plaza.

#### **6.1 Main specifications of a smart toll-plaza control system**

A smart toll collection system has the following four components. The first three components are usually installed at the toll booths. The later backend is installed in the control room and is usually connected with manages the complete toll collection process [12].

#### **6.2 Toll plaza components equipment**

The toll system comprises of Lane System and Plaza System, integrated into an architecture that facilitates easy and accurate toll collection. The following **Figures 8** and **9** depict the lane and plaza equipment. The OHLS (Over Head Lane Signal) is often required. The so-called AVCC (auto vehicle classification system ) is needed to determine the different fares of different vehicles if lanes have no signage with the type of passing vehicle. AVCC Systems may be Treadle systems that use a combination of vehicle magnetic loop, height sensors, and piezo sensors. Alternatively, AVCC may be IR-based.

*Combining Supervisory Control and Data Acquisition (SCADA) with Artificial Intelligence… DOI: http://dx.doi.org/10.5772/intechopen.104766*

**Figure 8.** *Schematic of toll-plaza lane and booth equipment.*

#### **Figure 9.**

*Schematic of plaza equipment.*

Also, the PBX telephone, which may be required inside each booth, is not shown. The WIM (Weigh-In-Motion) platform, which senses and records the vehicle weight, is dedicated to truck traffic control. Additional cameras should be installed inside each booth. In fact, a major challenge faced by any concessionaire in operating toll roads is the prevention of revenue leakage that goes very high like 20% of the daily collection in the situation of un-monitored systems.

#### *6.2.1 Booth & Lane equipment list*

a-Booth Equipment: 1. Toll Lane Controller; 2. Operator Terminal Screen (POS); 3. Receipt Printer; 4. Barcode Reader; 5. Barrier Controller; 6. Alarm; 7. Document Viewer; 8. WIM Indicator.

b-Lane Equipment: 9. OHLS; 10. Fare Display; 11. Incident Capture Outdoor Camera; 12. Barrier Gate; 13. Traffic Light; 14. AVCC; 15. Vehicle Entry Loop; 16. Vehicle Exit Loop; 17. Outdoor ANPR Camera; 18. Smart Card or RFID Reader; 19. WIM Platform.

#### *6.2.2 Plaza equipments*

The toll Plaza equipment consists of:

1. Database Server; 2. CCTV Display Screen; 3. POS Workstation; 4. CCTV (extra security) Cameras; 5. Report Printer; 6. IP Phone; 7. Network Switches; 8. IP phone Master (PABX) Unit; 9. UPS Power Supply.

#### **6.3 Plaza network installation procedure**

The first step, before the installation of any security system, is to ensure the presence of detailed drawings and documentation, a bill of materials with their specification. The installation of the ETC system uses many devices like vehicle-mounted electronic tags, toll point of sale(POS), RFID readers, and switches.

#### **Figure 10.**

*Connection scenario #1 of a toll plaza using a ring fiber between booths and plaza control room.*

**Figure 11.** *Illustrating example showing the internet.*

*Combining Supervisory Control and Data Acquisition (SCADA) with Artificial Intelligence… DOI: http://dx.doi.org/10.5772/intechopen.104766*

#### **Figure 12.**

*Example of a VLAN configuration of a toll plaza network.*

Actually, there exist several scenarios to connect the plaza network using Ethernet copper cables and/or optical fibers. The network topology may have several choices. For instance, you can use a bus topology, rings, μ-rings, or a hybrid bus/ring topology. One of these scenarios is depicted below in **Figure 10**.

If we have a sufficiently long multi-core optical fiber cable, you can also install a large ring between all switches in the main booth and the control room.

Note that the design of the toll plaza data network should be secured and isolated from direct exposure to other internet users. In particular, the sensitive video signal (from IP cameras) should be routed indirectly to the internet, via NVR, followed by a separate NIC card to the plaza server. The plaza server can be then connected to the internet by another NIC card. This is illustrated in the following **Figure 11**. In all cases, a virtual private network (VPN) should be installed before routing the plaza signals, to any external WAN, such as the internet.

In all cases, the data switches are dividing the core network into small subnets; called VLANs. For instance, by sorting node devices by functions or positions (camera, POS, etc., VLANs are often associated with IP subnets. Hence, networks with different VLANs will not be visible to each other. On the physical layer, the network remains the same as shown in **Figure 12**.

#### **6.4 IP planning**

The only way for someone to access the CCTV system is to know the IP address, username, and password (**Table 1**).

#### **6.5 Port forwarding and accessing the internet**

To remotely view a security CCTV system, you had to allow it to communicate to the internet. To allow access to your system from the internet, you have to configure the firewall inside the router to flow through the NVR. This process is called port forwarding and requires some advanced knowledge. There exist many guides on port forwarding if you feel confident configuring remote viewing. Each router will have its own method for port forwarding and I recommend checking PortForward.com for your specific model.

#### **Table 1.**

*Example of IP planning of the overall data network.*

#### **Figure 13.**

*Modular architecture of Xanado VMS, with application to traffic surveillance and traffic control at a toll plaza network.*

*Combining Supervisory Control and Data Acquisition (SCADA) with Artificial Intelligence… DOI: http://dx.doi.org/10.5772/intechopen.104766*

#### **6.6 Remote viewing using a smartphone**

There exist Android and iPhone network operating systems (NoS) Apps that work with CCTV systems, such as IPTecno Pegaso [12]. This can connect to the system and display live video feeds. It also allows for one-way and two-way audio interaction, PTZ camera control, and motorized camera control. To learn how to connect your system, please follow any guide on how to view security cameras from iPhone or Android, such as this: https://www.cctvcameraworld.com/ how-to-view-security-cameras-from-phone/.

#### **6.7 Plaza database server and toll management system**

The Smart Toll Collection System can greatly expedite the time taken by each vehicle to pay the toll fees. There are existing solutions that are deployed and are practically well-suited. A distributed database architecture platform should be installed to enable operations of ticket issues and validation. In the control system, the failure of the LAN connectivity should not impact the lane operation.

#### **6.8 ETC management system**

The following **Figure 13** depicts the modular architecture of Xanado, with emphasis on traffic surveillance control at toll plazas. Such software provides comprehensive capabilities to manage toll collection operations (24/7). The Central Administration Module facilitates the entire operation of monitoring and collection across toll plazas as one centralized unit. The Plaza Module can configure, manage all plazas toll collection operations, report toll collection in an audited manner. The Lanes Module is running under the plaza module, lanes module facilitates accurate toll collection for the vehicle passing from the toll plaza.

#### **7. Conclusions**

Public concern over security in recent years has driven the demand for video surveillance. Security and video surveillance need continuous change with time, to cope with the new hardware capabilities of IP cameras and video storage equipment. In addition, the increasing need for video analytics is required in modern VMS software, to perform their job of monitoring activities and protecting humans and their properties. Governments and police departments worldwide are constantly looking for new CCTV surveillance features that will help prevent crime. The latest VMS software relies on CCTV surveillance systems that can monitor a larger number of cameras and sites more efficiently. Therefore, combining SCADA features with VMS is significant.

This chapter presents a method based on neural networks (NN) for monitoring and operating video surveillance systems (VMS), like those in traffic control networks of electronic plaza sites. The method suggests that the thresholds used for generating alarms can be adapted to each surveillance device (e.g., IP Camera).

The industry needs to do more research on hybrid systems that combine the best of SCADA and AI algorithms together with VMS software.

*Intelligent Video Surveillance - New Perspectives*

### **Author details**

Muhammad H. El-Saba Professor at the Dept of Electronics and Communication Engineering, Ain-Shams University, Engineering College, Cairo, Egypt

\*Address all correspondence to<EMAIL_ADDRESS>
© 2022 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

*Combining Supervisory Control and Data Acquisition (SCADA) with Artificial Intelligence… DOI: http://dx.doi.org/10.5772/intechopen.104766*

#### **References**

[1] M. H. El-Saba, Supervisory Control And Data Acquisition (SCADA). 2017. Measurement & Instrumentation Systems, Publisher: Hakim. Available from: https://www.researchgate.net/ publication/353142664\_Supervisory\_ Control\_And\_Data\_Acquisition\_SCADA

[2] Mariana Hentea. 2008. Improving Security for SCADA Control Systems. Available from: https://www. researchgate.net/publication/253290135

[3] Marugán AP, Márquez FG. SCADA and Artificial Neural Networks for Maintenance Management. In: International Conf. on Management Science and Engineering Management. Cham: Springer; 2017. pp. 912-919

[4] El-Saba MH. Xanado, Video Management System, to be published

[5] Whittaker D. Why AI CCTV is the Future of Security and Surveillance in Public Spaces. Security magazine, USA, securitymagazine.com, 2021

[6] Bharadwaj HS, Biswas S, Ramakrishnan KR. A large scale dataset for classification of vehicles in urban traffic scenes. In: Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image (ICGIP). Published in Association for Computing Machinery (ACM), New York, NY, USA. 2016

[7] Mohana et al. Design and implementation of object detection, tracking, counting and classification algorithms using artificial intelligence for automated video surveillance applications. In: Advanced Computing and Communication Society (ACCS), 24th annual International Conference on Advanced Computing and Communications (ADCOM-2018). Bangalore: IIITB; 2018

[8] Kain Z et al. Detecting abnormal events in university areas. In: 2018 International conference on Computer and Applications Beirut, Lebanon, IEEE. 2018

[9] Mohana, Aradhy HVR. Design object detection and tracking using deep learning and artificial intelligence for video surveillance applications. International Journal of Advanced Computer Science and Applications. 2019;**10**(12)

[10] Turaga SC, Murray JF, Jain V, Roth F, Helmstaedter M, Briggman K, et al. Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Computation. 2010;**22**(2):511-538

[11] Balamurugan Rajagopal BGSA, Parasuram K. A robust framework to detect moving vehicles in different road conditions in India. Journal of Theoretical and Applied Information Technology. 2019;**96**(1):1-14

[12] Wang P, Li L, Jin Y,Wang G. Detection of unwanted traffic congestion based on existing surveillance system using in freeway via a CNN-architecture trafficnet, 13th IEEE Conference on Industrial Electronics and Applications (ICIEA), Wuhan IEEE. 2018. pp. 1134-1139

### *Edited by Pier Luigi Mazzeo*

The development of new technologies based on artificial intelligence and computer vision, together with the possibility of connecting different devices together in realtime, has enabled the development and progress of intelligent video surveillance. Thanks to IoT technologies, high-resolution cameras can be networked to monitor their territory, collect recordings and have them analyzed by artificial intelligence systems trained to identify critical situations. This book discusses new achievements in intelligent video surveillance solutions and their future prospects.

Published in London, UK © 2023 IntechOpen © scyther5 / iStock

Intelligent Video Surveillance - New Perspectives

Intelligent Video Surveillance

New Perspectives

*Edited by Pier Luigi Mazzeo*