We are IntechOpen, the world's leading publisher of Open Access books Built by scientists, for scientists

4,000+

Open access books available

116,000+

International authors and editors

120M+

Downloads

Our authors are among the

Top 1%

most cited scientists

12.2%

Contributors from top 500 universities

Selection of our books indexed in the Book Citation Index in Web of Science™ Core Collection (BKCI)

## Interested in publishing with us? Contact<EMAIL_ADDRESS>
Numbers displayed above are based on latest data collected. For more information visit www.intechopen.com

## **Meet the editor**

Prof. António J. R. Neves received the Ph.D. in Electrical Engineering from the University of Aveiro, in 2007. Since 2002 he is a researcher at Institute of Electronics and Informatics Engineering of Aveiro and since 2007 an Assistant Professor at the Department of Electronics, Telecommunications and Informatics of the University of Aveiro. He is an IEEE Senior Member and member

of several other research organizations worldwide. His main research interests are computer vision, robotics and image and video processing. He participated or coordinated several research projects and published 145 publications, including books, book chapters, journal articles and conference papers. He has also a vast experience as a reviewer of several journals and conferences. As a professor, he supervised several Ph.D. students and Master students in the referred areas. Moreover, he has been head Professor for several courses in the areas of computer science, signal processing, image processing and robotics. Over the last months I have been coordinating the restructuring of the Master's Program in Electronics and Telecommunications Engineering, as he has been recently nominated as director.

Contents

**Preface VII**

**Section 1 Video Surveillance Systems 1**

Ying-Jen Chen

**Smart Cities 19**

and Richard Jiang

**Future Aspect 37**

**Section 2 Human Activity Recognition 57**

**Descriptor and DNN 59**

Fazalul Rahiman

Kumar Yadav

Chapter 1 **Particle-Filter-Based Intelligent Video Surveillance System 3**

Chapter 2 **Video Surveillance-Based Intelligent Traffic Management in**

Chapter 3 **Advance Intelligent Video Surveillance System (AIVSS): A**

Chapter 4 **Real-Time Action Recognition Using Multi-level Action**

Chapter 5 **Human Activity Recognition without Vision Tracking 77**

Chapter 6 **Device-Free Localization for Human Activity Monitoring 95**

Fozia Mehboob, Muhammad Abbas, Abdul Rauf, Shoab A. Khan

Mritunjay Rai, Agha Asim Husain, Tanmoy Maity and Ravindra

Cheng-Bin Jin, Trung Dung Do, Mingjie Liu and Hakil Kim

Carlos Alberto Flores Vázquez, Joan Aranda, Daniel Icaza, Santiago Pulla, Marcelo Flores-Vázquez and Nelson Federico Cordova

Shaufikah Shukri, Latifah Munirah Kamarudin and Mohd Hafiz

## Contents

#### **Section 3 Face and Expressions Recognition 121**

Chapter 7 **Access Control in the Wild Using Face Verification 123** Ricardo Ribeiro, Daniel Lopes and António Neves

#### Chapter 8 **Detecting Micro-Expressions in Real Time Using High-Speed Video Sequences 141** Radu Danescu, Diana Borza and Razvan Itu

Preface

Intelligent video surveillance is a multidisciplinary field related to computer vision, pattern

The goal is to efficiently extract useful information from a considerable number of videos collected by surveillance cameras by automatically detecting, tracking and recognizing ob‐ jects of interest, and understanding and analyzing their activities. The widespread availabili‐ ty of digital cameras and processing equipment, together with a growing need for public safety have shifted the attention of researchers in video surveillance, becoming one of the

Video surveillance has a huge amount of applications, from public to private places, such as homeland security, crime prevention, traffic control, accident prediction and detection, mon‐ itoring patients, elderly or children in different environments. These applications require monitoring indoor and outdoor scenes of airports, train stations, highways, parking lots, stores, shopping malls, offices, just to name a few. Nowadays, there are a considerable num‐ ber of digital cameras in the referred places collecting a huge amount of data on a daily ba‐ sis. Researchers are urged to develop intelligent systems to efficiently extract and visualize

The exponential effort on the development of new algorithms and systems for video surveil‐ lance is confirmed by the amount of effort invested in projects and companies, the creation on new startups worldwide and, not less important, in the quantity and quality of the manuscripts published in a considerable number of journals and conferences worldwide. This book is an outcome of research done by several researchers and professionals who have highly contributed to the field of video surveillance. The main goal is to present recent ad‐ vances in this hot topic. I would like to thank all the authors for their excellent contributions

This book contains of eight chapters divided into three sections. Section 1 consists of three chapters focusing on the development of systems and algorithms for video surveillance. Sec‐ tion 2 presents three chapters focusing on people, namely human activity and action recog‐ nition. Section 3 consists on a couple of chapters related to algorithms for face and

Finally, I hope that all the readers of this book will find it interesting and informative, con‐

**Prof. António J. R. Neves, Ph.D.** Department of Electronics

University of Aveiro

Portugal

Telecommunications and Informatics

recognition, image processing, networks, embedded systems and image sensors.

most active research areas in computer vision.

useful information from this big data source.

and to all those people who helped in this project.

sidering it as a good tool for their research or project.

expressions recognition.

## Preface

**Section 3 Face and Expressions Recognition 121**

**VI** Contents

**Video Sequences 141**

Chapter 7 **Access Control in the Wild Using Face Verification 123** Ricardo Ribeiro, Daniel Lopes and António Neves

Radu Danescu, Diana Borza and Razvan Itu

Chapter 8 **Detecting Micro-Expressions in Real Time Using High-Speed**

Intelligent video surveillance is a multidisciplinary field related to computer vision, pattern recognition, image processing, networks, embedded systems and image sensors.

The goal is to efficiently extract useful information from a considerable number of videos collected by surveillance cameras by automatically detecting, tracking and recognizing ob‐ jects of interest, and understanding and analyzing their activities. The widespread availabili‐ ty of digital cameras and processing equipment, together with a growing need for public safety have shifted the attention of researchers in video surveillance, becoming one of the most active research areas in computer vision.

Video surveillance has a huge amount of applications, from public to private places, such as homeland security, crime prevention, traffic control, accident prediction and detection, mon‐ itoring patients, elderly or children in different environments. These applications require monitoring indoor and outdoor scenes of airports, train stations, highways, parking lots, stores, shopping malls, offices, just to name a few. Nowadays, there are a considerable num‐ ber of digital cameras in the referred places collecting a huge amount of data on a daily ba‐ sis. Researchers are urged to develop intelligent systems to efficiently extract and visualize useful information from this big data source.

The exponential effort on the development of new algorithms and systems for video surveil‐ lance is confirmed by the amount of effort invested in projects and companies, the creation on new startups worldwide and, not less important, in the quantity and quality of the manuscripts published in a considerable number of journals and conferences worldwide.

This book is an outcome of research done by several researchers and professionals who have highly contributed to the field of video surveillance. The main goal is to present recent ad‐ vances in this hot topic. I would like to thank all the authors for their excellent contributions and to all those people who helped in this project.

This book contains of eight chapters divided into three sections. Section 1 consists of three chapters focusing on the development of systems and algorithms for video surveillance. Sec‐ tion 2 presents three chapters focusing on people, namely human activity and action recog‐ nition. Section 3 consists on a couple of chapters related to algorithms for face and expressions recognition.

Finally, I hope that all the readers of this book will find it interesting and informative, con‐ sidering it as a good tool for their research or project.

> **Prof. António J. R. Neves, Ph.D.** Department of Electronics Telecommunications and Informatics University of Aveiro Portugal

**Section 1**

**Video Surveillance Systems**

**Video Surveillance Systems**

**Chapter 1**

Provisional chapter

**Particle-Filter-Based Intelligent Video Surveillance**

DOI: 10.5772/intechopen.76960

In this study, an intelligent video surveillance (IVS) system is designed based on the particle filter. The designed IVS system can gather the information of the number of persons in the area and hot spots of the area. At first, the Gaussian mixture background model is utilized to detect moving objects by background subtraction. The moving object appearing in the margin of the video frame is considered as a new person. Then, a new particle filter is assigned to track the new person when it is detected.  Moreover, the Kalman filter is utilized to estimate the position of the person when the person is occluded. Information of the number of persons in the area and hot spots is gathered by tracking persons in the video frame. Finally, a user interface is designed to feedback the gathered information to users of the IVS system. By applying the proposed IVS system, the load of security guards can be reduced. Moreover, by hot spot analysis, the business operator can understand customer habits to plan the traffic flow and adjust the product

Keywords: intelligent video surveillance (IVS), Gaussian mixture model, particle filter,

Video surveillance systems are often utilized at some specific places such as exits, entrances, parking lots, convenient stores, etc. for management. Traditionally, security guards watch screens of monitored places for surveillance. However, according to statistics, one security guard can only watch four monitors at the same time, and the concentration can last only for 10 minutes such that more than 50% of key information is lost. Moreover, it is not economically

> © 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and eproduction in any medium, provided the original work is properly cited.

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

Particle-Filter-Based Intelligent Video Surveillance

**System**

System

Ying-Jen Chen

Abstract

Kalman filter

1. Introduction

Ying-Jen Chen

Additional information is available at the end of the chapter

placement for improving customer experience.

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.76960

#### **Particle-Filter-Based Intelligent Video Surveillance System** Particle-Filter-Based Intelligent Video Surveillance System

DOI: 10.5772/intechopen.76960

#### Ying-Jen Chen Ying-Jen Chen

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.76960

#### Abstract

In this study, an intelligent video surveillance (IVS) system is designed based on the particle filter. The designed IVS system can gather the information of the number of persons in the area and hot spots of the area. At first, the Gaussian mixture background model is utilized to detect moving objects by background subtraction. The moving object appearing in the margin of the video frame is considered as a new person. Then, a new particle filter is assigned to track the new person when it is detected.  Moreover, the Kalman filter is utilized to estimate the position of the person when the person is occluded. Information of the number of persons in the area and hot spots is gathered by tracking persons in the video frame. Finally, a user interface is designed to feedback the gathered information to users of the IVS system. By applying the proposed IVS system, the load of security guards can be reduced. Moreover, by hot spot analysis, the business operator can understand customer habits to plan the traffic flow and adjust the product placement for improving customer experience.

Keywords: intelligent video surveillance (IVS), Gaussian mixture model, particle filter, Kalman filter

## 1. Introduction

Video surveillance systems are often utilized at some specific places such as exits, entrances, parking lots, convenient stores, etc. for management. Traditionally, security guards watch screens of monitored places for surveillance. However, according to statistics, one security guard can only watch four monitors at the same time, and the concentration can last only for 10 minutes such that more than 50% of key information is lost. Moreover, it is not economically

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and eproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

worthwhile to hire multiple security guards to monitor one video surveillance system. Therefore, intelligent video surveillance (IVS) systems have become more and more important in commercial sector and have attracted a lot of attention in research area as well [1–3].

2.1. Adaptive Gaussian mixture model

The recent history of each pixel, f g X1; X2; ⋯Xt , is modeled by a mixture of k Gaussian distri-

where k is the number of distributions, ωi,t is an estimate of the weight of the ith Gaussian distribution in the mixture at time t, μi,t is the mean value of the ith Gaussian distribution in the mixture at time t, Σi,t is the covariance matrix of the ith Gaussian distribution in the

> 2πj j Σ 1 2 e �1 2ð Þ Xt�μ T

Xt � μi,t � �T

The adaptive Gaussian mixture model has been applied for different kinds of applications, such as automatic speech emotion recognition [15], tracking targets on long-range radar

The key idea of particle filtering is to approximate the probability distribution by a weighted

� �

Each sample consists of an element s which represents the hypothetical state of the object and a

sample set is calculated by propagating each sample according to a system model. Each element of the set is then weighted in terms of the observations, and N samples are drawn

n¼1

n ¼ 1; ⋯; N

S ¼ sð Þ n ; πð Þ n � ��

with replacement. The mean state of the object is estimated at each time step by

Eð Þ¼ S X N

The updating rules for the parameters of the adaptive Gaussian mixture model can be found in [5]. After the Gaussian mixture model is established, the foreground pixels (representing the

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi

ωi,tη Xt; μi,t

;Σi,t

Σ�1

Σi,t Xt � μi,t

r � �: (3)

n o: (4)

� � (1)

Particle-Filter-Based Intelligent Video Surveillance System

http://dx.doi.org/10.5772/intechopen.76960

5

ð Þ Xt�μ : (2)

n¼1 πð Þ n ¼ 1. The evolution of the

πð Þ n sð Þ n : (5)

butions. The probability of observing the current pixel value is given as Eq. [4]:

X k

i¼1

P Xð Þ¼ t

mixture at time t, and η is a Gaussian probability density function:

η Xt; μ;Σ � � ¼ 1

moving objects) can be obtained by applying the Mahalanobis distance:

Dið Þ¼ Xt

systems [16], fast sampling-based motion planning [17], etc.

corresponding discrete sampling probability π where PN

Particle filter provides a robust tracking framework.

2.2. Particle filter

sample set [18]:

IVS system can be defined as the real-time monitoring of both persistent and transient objects within a specific environment. IVS is also referred to as video analytics (VA) which involves the use of software to automatically detect the objects of specific interest and analyze their behaviors. For finding the objects of interest, it is usually done by detecting the movements or changes in the image that can be achieved by background subtraction technique. To do background subtraction, an effective way to build up the background is the Gaussian mixture model [4, 5]. After the object of interest is detected, the goal is to analyze their behavior that sometimes can be done by tracking them. Usually, particle filters and Kalman filters are employed for the purpose of tracking objects in IVS systems [6–10].

IVS systems have been applied for different kinds of purposes. Ref. [11] presents an architecture for a perimeter security system dedicated to critical transport infrastructure protection. Ref. [12] addresses a framework for event decision of vision-based intelligent surveillance system based on the fuzzy model. A shape-perceived algorithm using the building block-based matching method is presented in [13] for object tracking of intelligent surveillance applications. A way of unification of flame and smoke detection algorithms by merging the common steps into a single processing flow is proposed in [14] for IVS systems.

The aim of this study is to design a IVS system based on the particle filter. The designed IVS system can gather the information of the number of persons being in the area, the number of persons having been in the area, and hot spots (places of more than usual interest, activity, or popularity) of the area. The Gaussian mixture background model is utilized to detect moving objects by background subtraction in the designed IVS system. The moving object appearing in the margin of the video frame is considered as a new unit (person). When a new person is detected, a new particle filter is established and assigned to track the new person. For saving the computational load, the particle filter is terminated when the corresponding tracked person leaves the video frame. Moreover, the Kalman filters is utilized to estimate the position of the person when the person is occluded. Information of the number of persons in the area (having been in the area) and hot spots is gathered by tracking persons in the video frame. Finally, a user interface is designed to feedback the gathered information to users of the IVS system. By applying the proposed IVS system, the load of security guards can be reduced. Moreover, by hot spot analysis, the business operator can understand customer habits to plan the traffic flow and adjust the product placement for improving customer experience.

## 2. Preliminary

In this section, three well-known techniques, i.e. adaptive Gaussian mixture model, particle filter, and Kalman filter, are presented for constructing the IVS system.

#### 2.1. Adaptive Gaussian mixture model

worthwhile to hire multiple security guards to monitor one video surveillance system. Therefore, intelligent video surveillance (IVS) systems have become more and more important in

IVS system can be defined as the real-time monitoring of both persistent and transient objects within a specific environment. IVS is also referred to as video analytics (VA) which involves the use of software to automatically detect the objects of specific interest and analyze their behaviors. For finding the objects of interest, it is usually done by detecting the movements or changes in the image that can be achieved by background subtraction technique. To do background subtraction, an effective way to build up the background is the Gaussian mixture model [4, 5]. After the object of interest is detected, the goal is to analyze their behavior that sometimes can be done by tracking them. Usually, particle filters and Kalman filters are

IVS systems have been applied for different kinds of purposes. Ref. [11] presents an architecture for a perimeter security system dedicated to critical transport infrastructure protection. Ref. [12] addresses a framework for event decision of vision-based intelligent surveillance system based on the fuzzy model. A shape-perceived algorithm using the building block-based matching method is presented in [13] for object tracking of intelligent surveillance applications. A way of unification of flame and smoke detection algorithms by merging the common steps into a single processing flow is proposed in [14] for

The aim of this study is to design a IVS system based on the particle filter. The designed IVS system can gather the information of the number of persons being in the area, the number of persons having been in the area, and hot spots (places of more than usual interest, activity, or popularity) of the area. The Gaussian mixture background model is utilized to detect moving objects by background subtraction in the designed IVS system. The moving object appearing in the margin of the video frame is considered as a new unit (person). When a new person is detected, a new particle filter is established and assigned to track the new person. For saving the computational load, the particle filter is terminated when the corresponding tracked person leaves the video frame. Moreover, the Kalman filters is utilized to estimate the position of the person when the person is occluded. Information of the number of persons in the area (having been in the area) and hot spots is gathered by tracking persons in the video frame. Finally, a user interface is designed to feedback the gathered information to users of the IVS system. By applying the proposed IVS system, the load of security guards can be reduced. Moreover, by hot spot analysis, the business operator can understand customer habits to plan the traffic flow and adjust the product placement for

In this section, three well-known techniques, i.e. adaptive Gaussian mixture model, particle

filter, and Kalman filter, are presented for constructing the IVS system.

commercial sector and have attracted a lot of attention in research area as well [1–3].

employed for the purpose of tracking objects in IVS systems [6–10].

IVS systems.

4 Intelligent Video Surveillance

improving customer experience.

2. Preliminary

The recent history of each pixel, f g X1; X2; ⋯Xt , is modeled by a mixture of k Gaussian distributions. The probability of observing the current pixel value is given as Eq. [4]:

$$P(\mathbf{X}\_t) = \sum\_{i=1}^k \omega\_{i,t} \eta \left(\mathbf{X}\_t, \mu\_{i,t}, \Sigma\_{i,t}\right) \tag{1}$$

where k is the number of distributions, ωi,t is an estimate of the weight of the ith Gaussian distribution in the mixture at time t, μi,t is the mean value of the ith Gaussian distribution in the mixture at time t, Σi,t is the covariance matrix of the ith Gaussian distribution in the mixture at time t, and η is a Gaussian probability density function:

η Xt; μ;Σ � � ¼ 1 2πj j Σ 1 2 e �1 2ð Þ Xt�μ T Σ�1 ð Þ Xt�μ : (2)

The updating rules for the parameters of the adaptive Gaussian mixture model can be found in [5]. After the Gaussian mixture model is established, the foreground pixels (representing the moving objects) can be obtained by applying the Mahalanobis distance:

$$D\_i(\mathbf{X}\_t) = \sqrt{\left(\mathbf{X}\_t - \boldsymbol{\mu}\_{i,t}\right)^T \boldsymbol{\Sigma}\_{i,t} \left(\mathbf{X}\_t - \boldsymbol{\mu}\_{i,t}\right)}.\tag{3}$$

The adaptive Gaussian mixture model has been applied for different kinds of applications, such as automatic speech emotion recognition [15], tracking targets on long-range radar systems [16], fast sampling-based motion planning [17], etc.

#### 2.2. Particle filter

The key idea of particle filtering is to approximate the probability distribution by a weighted sample set [18]:

$$\mathbf{S} = \left\{ \left( \mathbf{s}^{(n)}, \pi^{(n)} \right) \middle| n = 1, \cdots, N \right\}. \tag{4}$$

Each sample consists of an element s which represents the hypothetical state of the object and a corresponding discrete sampling probability π where PN n¼1 πð Þ n ¼ 1. The evolution of the sample set is calculated by propagating each sample according to a system model. Each element of the set is then weighted in terms of the observations, and N samples are drawn with replacement. The mean state of the object is estimated at each time step by

$$E(\mathbf{S}) = \sum\_{n=1}^{N} \pi^{(n)} \mathbf{s}^{(n)}.\tag{5}$$

Particle filter provides a robust tracking framework.

The particle filter has been successfully applied to many applications. An algorithm to track the vehicle with the adaptively changed scale based on particle filter is propose in [19]. The vehicle guidance with control action computed by a Rao-Blackwellized particle filter is proposed in [20]. The localization of indoor robot based on particle filter with EKF proposal distribution is proposed in [21].

#### 2.3. Kalman filter

The Kalman filter [22] addresses the general problem of trying to estimate the state of a discrete-time controlled process that is governed by the linear stochastic difference equation:

$$\mathbf{x}\_{t} = \mathbf{F}\mathbf{x}\_{t-1} + \mathbf{w}\_{t} \tag{6}$$

The measurement update equations are given as

the state-dependent model uncertainties.

Figure 1. The operation architecture of Kalman filter.

Figure 1 shows the operation architecture of Kalman filter.

Kt ¼ P�

bxt ¼ bx�

t HT HP�

Pt ¼ ð Þ I � KtH P�

The Kalman filter has been widely applied to time series analysis and statistical modeling problems. This study [23] improves the navigation performance, when refraction starlight is used to compute the position and velocity of a satellite in unscented Kalman filter. An antispoofing algorithm based on adaptive Kalman filter for high dynamic positioning in global positioning system is proposed in [24]. In this work [25], the robust Kalman filter is applied to the people occupancy estimation problem, and an iterative algorithm is developed to handle

t þ Kt φt � Hbx�

t

t HT þ R � ��1 (14)

� � (15)

Particle-Filter-Based Intelligent Video Surveillance System

http://dx.doi.org/10.5772/intechopen.76960

7

t : (16)

with measurement equation

$$\mathbf{q}\boldsymbol{\varrho}\_t = \mathbf{H}\mathbf{x}\_t + \mathbf{v}\_t \tag{7}$$

where xt is the state vector, F is the transition matrix, φt is the measurement output, H is the output matrix, wt is the process noise, and vt is the measurement noise. The process noise and measurement noise are assumed to be independent of each other, white, and with normal probability distributions:

$$p(\omega) \sim N(0, \ \mathbf{Q}) \tag{8}$$

$$p(\mathbf{v}) \sim \mathcal{N}(0, \ \mathbf{R}) \tag{9}$$

where Q is the process noise covariance matrix and R is measurement noise covariance matrix. Moreover, bx� t is defined to be the a priori state estimate at step t, and bxt is defined to be the a posteriori state estimate. Then, e� t � xt � bx� t is defined to be a priori estimate error, and et � xt � bxt is defined to be the a posteriori estimate error. The a priori estimate error covariance is then

$$\mathbf{P}\_t^- = E\left[\mathbf{e}\_t^- \left(\mathbf{e}\_t^-\right)^T\right] \tag{10}$$

and the a posteriori estimate error covariance is

$$\mathbf{P}\_t = E\left[\mathbf{e}\_t^- (\mathbf{e}\_t)^T\right].\tag{11}$$

The equations for the Kalman filter fall into two groups: time update (predictor) equations and measurement update (corrector) equations. The time update equations are given as

$$
\widehat{\mathbf{x}}\_t^- = \mathbf{F} \widehat{\mathbf{x}}\_{t-1} \tag{12}
$$

$$\mathbf{P}\_t^- = \mathbf{F}\mathbf{P}\_{t-1}\mathbf{F}^T + \mathbf{Q}.\tag{13}$$

The measurement update equations are given as

The particle filter has been successfully applied to many applications. An algorithm to track the vehicle with the adaptively changed scale based on particle filter is propose in [19]. The vehicle guidance with control action computed by a Rao-Blackwellized particle filter is proposed in [20]. The localization of indoor robot based on particle filter with EKF proposal

The Kalman filter [22] addresses the general problem of trying to estimate the state of a discrete-time controlled process that is governed by the linear stochastic difference equation:

where xt is the state vector, F is the transition matrix, φt is the measurement output, H is the output matrix, wt is the process noise, and vt is the measurement noise. The process noise and measurement noise are assumed to be independent of each other, white, and with normal

where Q is the process noise covariance matrix and R is measurement noise covariance matrix.

et � xt � bxt is defined to be the a posteriori estimate error. The a priori estimate error covari-

t e� t � �T h i

t ð Þ et T h i

The equations for the Kalman filter fall into two groups: time update (predictor) equations and

t � xt � bx�

P� t ¼ E e�

Pt ¼ E e�

measurement update (corrector) equations. The time update equations are given as

P�

bx�

t is defined to be the a priori state estimate at step t, and bxt is defined to be the a

xt ¼ Fxt�1 þ wt (6)

φt ¼ Hxt þ vt (7)

pð Þ� ω Nð Þ 0; Q (8)

pð Þ� v Nð Þ 0; R (9)

t is defined to be a priori estimate error, and

: (11)

t ¼ Fbxt�1 (12)

t ¼ FPt�1FT þ Q: (13)

(10)

distribution is proposed in [21].

with measurement equation

probability distributions:

posteriori state estimate. Then, e�

and the a posteriori estimate error covariance is

Moreover, bx�

ance is then

2.3. Kalman filter

6 Intelligent Video Surveillance

$$\mathbf{K}\_{l} = \mathbf{P}\_{t}^{-} \mathbf{H}^{T} \left( \mathbf{H} \mathbf{P}\_{t}^{-} \mathbf{H}^{T} + \mathbf{R} \right)^{-1} \tag{14}$$

$$
\widehat{\mathbf{x}}\_t = \widehat{\mathbf{x}}\_t^- + \mathbf{K}\_t \left( \mathbf{q}\_t - \mathbf{H} \widehat{\mathbf{x}}\_t^- \right) \tag{15}
$$

$$\mathbf{P}\_t = (\mathbf{I} - \mathbf{K}\_t \mathbf{H})\mathbf{P}\_t^{-}. \tag{16}$$

Figure 1 shows the operation architecture of Kalman filter.

The Kalman filter has been widely applied to time series analysis and statistical modeling problems. This study [23] improves the navigation performance, when refraction starlight is used to compute the position and velocity of a satellite in unscented Kalman filter. An antispoofing algorithm based on adaptive Kalman filter for high dynamic positioning in global positioning system is proposed in [24]. In this work [25], the robust Kalman filter is applied to the people occupancy estimation problem, and an iterative algorithm is developed to handle the state-dependent model uncertainties.

Figure 1. The operation architecture of Kalman filter.

## 3. IVS system design

The IVS system design is presented in this section. Figure 2 shows the operation process of the IVS system. Firstly, the adaptive Gaussian mixture model is applied for constructing the background model to detect moving objects in the video image frame. The moving object appearing in the margin of the video image frame is considered as a new unit (person), and then a new particle filter is established and designated to track the new person. Moreover, the Kalman filter is utilized to correct the position obtaining by the particle filter and to estimate the position during occlusion. After that, the information of the number of persons in the area, the number of persons having been in the area, and hot spots are obtained by analyzing the tracking paths.  Each block of the IVS system design shown in Figure 2 will be illustrated in the following subsections.

#### 3.1. Gaussian mixture model for detecting new units

By applying the adaptive Gaussian mixture model described in Subsection 2.1, the moving objects can be detected by using the Mahalanobis distance of Eq. (3). Figure 3 illustrates the foreground pixels representing the moving objects obtained by the adaptive Gaussian mixture model. Here, we assume that a new unit (person) will appear only from the border of the monitored place. Therefore, for a moving object detected in the margin of the monitoring video

Figure 3. Foreground pixels representing the moving objects obtained by the adaptive Gaussian mixture model: (a)

Particle-Filter-Based Intelligent Video Surveillance System

http://dx.doi.org/10.5772/intechopen.76960

9

Figure 4 shows the checking process for determining the object detected in the margin of the monitoring video frame as a new person or not. At first we need to check the size and ratio of the detected object to identify that the detected object is a person or not. If the size and ratio of the detected object are identified as a person, then we have to check that the detected person is new or not. In the case that there is no tracked person in the video frame, the detected person in the margin of the video frame is determined to be a new person. In the case that there has (have) been tracked person(s) in the video frame, we need to calculate the distance(s) between the detected person and tracked person(s) to check that the detected person is new or not. If the distance(s) is (are all) longer than a predefined threshold Td, the detected person is considered as a new person. If some distances are shorter than Td, we need to apply Eq. (18), which will be described in the following subsection, to calculate the similarities of color distribution between

Figure 4. The checking process for determining the object detected in the margin of the monitoring video frame as a new

frame, we need to determine that it is a new person or not.

background, (b) moving object, and (c) foreground pixels.

person or not.

Figure 2. The operation process of the IVS system.

3. IVS system design

8 Intelligent Video Surveillance

3.1. Gaussian mixture model for detecting new units

Figure 2. The operation process of the IVS system.

The IVS system design is presented in this section. Figure 2 shows the operation process of the IVS system. Firstly, the adaptive Gaussian mixture model is applied for constructing the background model to detect moving objects in the video image frame. The moving object appearing in the margin of the video image frame is considered as a new unit (person), and then a new particle filter is established and designated to track the new person. Moreover, the Kalman filter is utilized to correct the position obtaining by the particle filter and to estimate the position during occlusion. After that, the information of the number of persons in the area, the number of persons having been in the area, and hot spots are obtained by analyzing the tracking paths.  Each block of the IVS system design shown in Figure 2 will be illustrated in the following subsections.

By applying the adaptive Gaussian mixture model described in Subsection 2.1, the moving objects can be detected by using the Mahalanobis distance of Eq. (3). Figure 3 illustrates the foreground pixels representing the moving objects obtained by the adaptive Gaussian mixture model. Here, we assume that a new unit (person) will appear only from the border of the

Figure 3. Foreground pixels representing the moving objects obtained by the adaptive Gaussian mixture model: (a) background, (b) moving object, and (c) foreground pixels.

monitored place. Therefore, for a moving object detected in the margin of the monitoring video frame, we need to determine that it is a new person or not.

Figure 4 shows the checking process for determining the object detected in the margin of the monitoring video frame as a new person or not. At first we need to check the size and ratio of the detected object to identify that the detected object is a person or not. If the size and ratio of the detected object are identified as a person, then we have to check that the detected person is new or not. In the case that there is no tracked person in the video frame, the detected person in the margin of the video frame is determined to be a new person. In the case that there has (have) been tracked person(s) in the video frame, we need to calculate the distance(s) between the detected person and tracked person(s) to check that the detected person is new or not. If the distance(s) is (are all) longer than a predefined threshold Td, the detected person is considered as a new person. If some distances are shorter than Td, we need to apply Eq. (18), which will be described in the following subsection, to calculate the similarities of color distribution between

Figure 4. The checking process for determining the object detected in the margin of the monitoring video frame as a new person or not.

the detected object and the tracked units with distance shorter than Td. If all similarities between the detected object and the tracked units with distance shorter than Td are lower than a predefined threshold Ts, the detected object is determined as a new person.

It is noted that there are several people detection methods [26, 27]. However, in the designed IVS system, we assume that the only moving objects are persons. Hence, we choose a simple method, which is adaptive Gaussian mixture model, to detect that the persons appear from the border of the monitored place for reducing the computational load.

#### 3.2. Particle filter for tracking units

For a new detected person, a new particle filter is established and designated to track the new person. In the design of the particle filter, the target model of target region (the detected person) is the color distribution which is represented by histograms calculated in the HS (Hue, Saturation) space using 8 � 8 bins. A popular measure between two color distributions is the Bhattacharyya coefficient. Considering discrete densities such as two color histograms

$$p = \left\{ p^{(u)} \right\}\_{u=1\ldots m}, \; q = \left\{ q^{(u)} \right\}\_{u=1\ldots m} \tag{17}$$

the Bhattacharyya coefficient is defined as

$$\rho[p,q] = \sum\_{u=1}^{m} \sqrt{p^{(u)}q^{(u)}}.\tag{18}$$

The larger r is, the more similar the two distributions are. For two identical histograms, we obtain r ¼ 1 indicating a perfect match. The target region of the detected person is represented by a rectangle so that a sample is given as

$$\mathbf{s} = \begin{bmatrix} \mathbf{x} & \mathbf{y} & H\_{\mathbf{x}} & H\_{\mathbf{y}} \end{bmatrix}^{T} \tag{19}$$

Although the particle filter is a robust method for tracking objects, it cannot deal with some special cases. Since we use color distribution for the target model of the particle filter, it may lose tracking when the color of the background is similar to the color of the tracking object.

In the IVS system design, the Kalman filter is utilized to correct the position obtained by the particle filter and to estimate the position during occlusion. Here, the uniform linear move-

1 0

bxt�1 byt�1 \_ bxt�1 \_ byt�1

þ wt�1 (22)

Particle-Filter-Based Intelligent Video Surveillance System

http://dx.doi.org/10.5772/intechopen.76960

11

0 1

1 0

0 1

Moreover, if the tracking object is occluded, still the particle filter will lose tracking.

ment is considered. Hence, the linear stochastic difference equation is given as

3.3. Kalman filter for correcting and estimating positions

Figure 5. The target region of the detected person represented by a rectangle.

bxt byt \_ bxt \_ byt

and the measurement equation is given as

Figure 6. Persons tracked particle filter.

where x and y represent the center location of the rectangle and Hx and Hy are the width and length of the rectangle, respectively, as shown in Figure 5. The sample set is propagated through the application of a dynamic model:

$$\mathbf{s}\_{t} = \mathbf{A}\mathbf{s}\_{t-1} + \mathbf{w}\_{t-1} \tag{20}$$

where A is an 4�4 identity matrix and wt�1 is a random vector drawn from the noise distribution of the system. Assuming that the target histogram is q and the histogram of the sample sð Þ n is psð Þ n , the observation probability of each sample is given as

$$
\pi^{(n)} = \rho[p\_{\mathfrak{s}^{(n)}}, q]. \tag{21}
$$

The tracking result can be calculated by Eq. (5). During filtering, samples with a high weight may be chosen several times, leading to identical copies, while others with relatively low weights may not be chosen at all. Figure 6 illustrates persons tracked by particle filter.

Particle-Filter-Based Intelligent Video Surveillance System http://dx.doi.org/10.5772/intechopen.76960 11

Figure 5. The target region of the detected person represented by a rectangle.

Figure 6. Persons tracked particle filter.

the detected object and the tracked units with distance shorter than Td. If all similarities between the detected object and the tracked units with distance shorter than Td are lower than

It is noted that there are several people detection methods [26, 27]. However, in the designed IVS system, we assume that the only moving objects are persons. Hence, we choose a simple method, which is adaptive Gaussian mixture model, to detect that the persons appear from the

For a new detected person, a new particle filter is established and designated to track the new person. In the design of the particle filter, the target model of target region (the detected person) is the color distribution which is represented by histograms calculated in the HS (Hue, Saturation) space using 8 � 8 bins. A popular measure between two color distributions is the Bhattacharyya coefficient. Considering discrete densities such as two

u¼1…m ; q ¼ qð Þ u n o

ffiffiffiffiffiffiffiffiffiffiffiffiffiffi pð Þ u qð Þ u q

r½ �¼ p; q Xm

u¼1

The larger r is, the more similar the two distributions are. For two identical histograms, we obtain r ¼ 1 indicating a perfect match. The target region of the detected person is represented

s ¼ xyHx Hy

where x and y represent the center location of the rectangle and Hx and Hy are the width and length of the rectangle, respectively, as shown in Figure 5. The sample set is propagated through

where A is an 4�4 identity matrix and wt�1 is a random vector drawn from the noise distribution of the system. Assuming that the target histogram is q and the histogram of the sample sð Þ n

The tracking result can be calculated by Eq. (5). During filtering, samples with a high weight may be chosen several times, leading to identical copies, while others with relatively low

weights may not be chosen at all. Figure 6 illustrates persons tracked by particle filter.

u¼1…m (17)

: (18)

� �T (19)

st ¼ Ast�1 þ wt�1 (20)

πð Þ n ¼ r psð Þ n ; q � �: (21)

a predefined threshold Ts, the detected object is determined as a new person.

border of the monitored place for reducing the computational load.

p ¼ pð Þ u n o

3.2. Particle filter for tracking units

the Bhattacharyya coefficient is defined as

by a rectangle so that a sample is given as

the application of a dynamic model:

is psð Þ n , the observation probability of each sample is given as

color histograms

10 Intelligent Video Surveillance

Although the particle filter is a robust method for tracking objects, it cannot deal with some special cases. Since we use color distribution for the target model of the particle filter, it may lose tracking when the color of the background is similar to the color of the tracking object. Moreover, if the tracking object is occluded, still the particle filter will lose tracking.

#### 3.3. Kalman filter for correcting and estimating positions

In the IVS system design, the Kalman filter is utilized to correct the position obtained by the particle filter and to estimate the position during occlusion. Here, the uniform linear movement is considered. Hence, the linear stochastic difference equation is given as

$$
\begin{bmatrix}
\hat{\boldsymbol{x}}\_{t} \\
\hat{\boldsymbol{y}}\_{t} \\
\dot{\hat{\boldsymbol{x}}}\_{t} \\
\dot{\hat{\boldsymbol{y}}}\_{t}
\end{bmatrix} = \begin{bmatrix}
1 & 0 & 1 & 0 \\
0 & 1 & 0 & 1 \\
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 1
\end{bmatrix} \begin{bmatrix}
\hat{\boldsymbol{x}}\_{t-1} \\
\hat{\boldsymbol{y}}\_{t-1} \\
\dot{\hat{\boldsymbol{x}}}\_{t-1} \\
\dot{\hat{\boldsymbol{y}}}\_{t-1}
\end{bmatrix} + \boldsymbol{w}\_{t-1} \tag{22}
$$

and the measurement equation is given as

$$
\begin{bmatrix}
\widehat{\boldsymbol{x}}\_{t} \\
\widehat{\boldsymbol{y}}\_{t}
\end{bmatrix} = \begin{bmatrix}
\mathbf{1} & \mathbf{0} & \mathbf{0} & \mathbf{0} \\
\mathbf{0} & \mathbf{1} & \mathbf{0} & \mathbf{0}
\end{bmatrix} \begin{bmatrix}
\widehat{\boldsymbol{x}}\_{t} \\
\widehat{\boldsymbol{y}}\_{t} \\
\widehat{\boldsymbol{x}}\_{t} \\
\widehat{\boldsymbol{y}}\_{t}
\end{bmatrix} + \boldsymbol{\mathfrak{v}}\_{t} \tag{23}
$$

occluded. However, if the occluded object is stayed in the back of obstacle without moving, the

Particle-Filter-Based Intelligent Video Surveillance System

http://dx.doi.org/10.5772/intechopen.76960

13

Finally, by analyzing the tracking paths, the information of the tracking result, the number of persons in the area, the number of persons having been in the area, and hot spots are obtained and then fed back to the user through the user interface as shown in Figure 7. Furthermore, several parameters can be adjusted through the user interface for adapting

The experiment is done in the San Shia Campus of the National Taipei University, Taipei, Taiwan. Figure 8 illustrates the operation process of the IVS system. In Figure 8, the up-left frame is the original image; the up-right frame is the background subtraction binary image applying Gaussian mixture model; the left-down frame shows the tracking result by applying particle filter (green rectangle) and the result corrected by Kalman filter (white rectangle), and the right-down frame is the final tracking result. Figure 9 makes a comparison to show the

Kalman filter will still lose the tracking.

different environments.

4. Experiment results

Figure 8. Illustration of the IVS operation process.

3.4. User interface for feeding back information

where ð Þ bx; by is the estimating position and \_ bx and \_ by represent the estimating speeds on x and y directions, respectively. In the measurement update equation of Eq. (15), the measurement φt ¼ xt yt � �T is obtained from the tracking result of the particle filter.

For improving the tracking results of particle filter, after propagation by Eq. (20), the estimating speeds \_ bxt�1 and \_ byt�1 of the Kalman filter are added to the position of each sample of the particle filter such that

$$\mathbf{s}\_{t}^{(n)} = \left[ \mathbf{x}\_{t}^{(n)} + \dot{\hat{\mathbf{x}}}\_{t-1'}, \ y\_{t}^{(n)} + \dot{\hat{y}}\_{t-1'}, H\_{\mathbf{x}'} H\_{y} \right]. \tag{24}$$

When the tracked object is occluded, the Kalman filter is applied to estimate the position of the occluded object. Therefore, for the case that all the similarities between samples of particle filter and the target are lower than lower than a predefined threshold Tp, the Kalman filter doesn't use the measurement correction and only takes the filter prediction as object position. Moreover, all samples of the particle filter are uniformly distributed around the estimating position such that the particle filter can retrieve tracking after the object recovering from

Figure 7. User interface of the IVS system.

occluded. However, if the occluded object is stayed in the back of obstacle without moving, the Kalman filter will still lose the tracking.

#### 3.4. User interface for feeding back information

Finally, by analyzing the tracking paths, the information of the tracking result, the number of persons in the area, the number of persons having been in the area, and hot spots are obtained and then fed back to the user through the user interface as shown in Figure 7. Furthermore, several parameters can be adjusted through the user interface for adapting different environments.

## 4. Experiment results

bxt byt � �

where ð Þ bx; by is the estimating position and \_

s ð Þ n t ¼ x

bxt�1 and \_

Figure 7. User interface of the IVS system.

φt ¼ xt yt

12 Intelligent Video Surveillance

ing speeds \_

particle filter such that

¼ 1 0 0 1

� �T is obtained from the tracking result of the particle filter.

ð Þ n t þ \_ bx t�1

byt \_ bxt \_ byt

byt�1 of the Kalman filter are added to the position of each sample of the

byt�1, Hx, Hy

þ vt (23)

: (24)

by represent the estimating speeds on x and y

" # bxt

bx and \_

directions, respectively. In the measurement update equation of Eq. (15), the measurement

For improving the tracking results of particle filter, after propagation by Eq. (20), the estimat-

, yð Þ n t þ \_

When the tracked object is occluded, the Kalman filter is applied to estimate the position of the occluded object. Therefore, for the case that all the similarities between samples of particle filter and the target are lower than lower than a predefined threshold Tp, the Kalman filter doesn't use the measurement correction and only takes the filter prediction as object position. Moreover, all samples of the particle filter are uniformly distributed around the estimating position such that the particle filter can retrieve tracking after the object recovering from

h i

The experiment is done in the San Shia Campus of the National Taipei University, Taipei, Taiwan. Figure 8 illustrates the operation process of the IVS system. In Figure 8, the up-left frame is the original image; the up-right frame is the background subtraction binary image applying Gaussian mixture model; the left-down frame shows the tracking result by applying particle filter (green rectangle) and the result corrected by Kalman filter (white rectangle), and the right-down frame is the final tracking result. Figure 9 makes a comparison to show the

Figure 8. Illustration of the IVS operation process.

utility of adding the estimating speeds of the Kalman filter to the position of each sample of the particle filter, i.e., to show the utility of Eq. (24). It can be seen that a fast-moving bicycle can be tracked by the particle filter with Eq. (24) as shown in Figure 9(a). However, the fast-moving bicycle cannot be tracked by the particle filter without Eq. (24) as shown in Figure 9(b). Figure 10 makes a comparison to show the utility of applying Kalman filter to estimate the position of the occluded object. It can be seen that the particle filter can retrieve tracking after occlusion with applying Kalman filter to estimate the position of the occluded object as shown in Figure 10(a). However, it can be seen that the particle filter cannot retrieve tracking after occlusion without applying Kalman filter to estimate the position of the occluded object as shown in Figure 10(b). Moreover, it can also be seen in Figure 10(a) that all samples of the particle filter are uniformly distributed around the estimating position such that the particle

Particle-Filter-Based Intelligent Video Surveillance System

http://dx.doi.org/10.5772/intechopen.76960

15

Based on the particle filter, an IVS system has been designed in this study. Utilizing the Gaussian mixture background model, the moving objects appearing in the margin of the video frame can be detected and considered as a new person. Then, a new particle filter is established and designated to track the new considered person. Moreover, the Kalman filter is applied to correct the tracking result and estimate the position when the tracked person is occluded. By analyzing the tracking paths, the information of the number of persons in the area, the number of persons having been in the area, and hot spots can be obtained. Finally, the information is

Department of Electrical Engineering, National Taipei University, New Taipei City, Taiwan

[1] Li SR, Tsai HC, Wang YK, Sun TH, Chen YJ. Particle-filter-based intelligent video surveillance system. In: Proceedings of the International Conference on System Science and

[2] Collins RT, Lipton AJ, Kanade T. A system for video surveillance and monitoring. In: Proceedings of American Nuclear Society (ANS) Eighth International Topical Meeting on

filter can retrieve tracking after the object recovering from occluded.

fed back to the user through the user interface.

Address all correspondence to<EMAIL_ADDRESS>
Engineering (ICSSE'16); 7-9 July. 2016. pp. 1-4

Robotic and Remote Systems; April 1999

5. Conclusion

Author details

Ying-Jen Chen

References

Figure 9. A fast-moving bicycle tracked by the particle filter (a) with Eq. (24) and (b) without Eq. (24).

Figure 10. (a) Applying Kalman filter to estimate the position of the occluded object and (b) without applying Kalman filter to estimate the position of the occluded object.

utility of adding the estimating speeds of the Kalman filter to the position of each sample of the particle filter, i.e., to show the utility of Eq. (24). It can be seen that a fast-moving bicycle can be tracked by the particle filter with Eq. (24) as shown in Figure 9(a). However, the fast-moving bicycle cannot be tracked by the particle filter without Eq. (24) as shown in Figure 9(b). Figure 10 makes a comparison to show the utility of applying Kalman filter to estimate the position of the occluded object. It can be seen that the particle filter can retrieve tracking after occlusion with applying Kalman filter to estimate the position of the occluded object as shown in Figure 10(a). However, it can be seen that the particle filter cannot retrieve tracking after occlusion without applying Kalman filter to estimate the position of the occluded object as shown in Figure 10(b). Moreover, it can also be seen in Figure 10(a) that all samples of the particle filter are uniformly distributed around the estimating position such that the particle filter can retrieve tracking after the object recovering from occluded.

## 5. Conclusion

Figure 9. A fast-moving bicycle tracked by the particle filter (a) with Eq. (24) and (b) without Eq. (24).

Figure 10. (a) Applying Kalman filter to estimate the position of the occluded object and (b) without applying Kalman

filter to estimate the position of the occluded object.

14 Intelligent Video Surveillance

Based on the particle filter, an IVS system has been designed in this study. Utilizing the Gaussian mixture background model, the moving objects appearing in the margin of the video frame can be detected and considered as a new person. Then, a new particle filter is established and designated to track the new considered person. Moreover, the Kalman filter is applied to correct the tracking result and estimate the position when the tracked person is occluded. By analyzing the tracking paths, the information of the number of persons in the area, the number of persons having been in the area, and hot spots can be obtained. Finally, the information is fed back to the user through the user interface.

## Author details

#### Ying-Jen Chen

Address all correspondence to<EMAIL_ADDRESS>
Department of Electrical Engineering, National Taipei University, New Taipei City, Taiwan

## References

[3] Nam Y, Rho S, Park JH. Intelligent video surveillance system: 3-tier context-aware surveillance system with metadata. Multimedia Tools and Applications. 2012;57:315-334. DOI: 10.1007/s11042-010-0677-x

[15] Yang JH, Hung JW. A preliminary study of emotion recognition employing adaptive Gaussian mixture models with the maximum a posteriori principle. In: Proceedings of the 2014 International Conference on Information Science, Electronics and Electrical Engi-

Particle-Filter-Based Intelligent Video Surveillance System

http://dx.doi.org/10.5772/intechopen.76960

17

[16] Davis B, Blair WD. Adaptive Gaussian mixture modeling for tracking of long range targets. In: Proceedings of the 2016 IEEE Aerospace Conference; 5-12 March 2016; Big

[17] Huh J, Lee B, Lee DD. Adaptive motion planning with high-dimensional mixture models. In: Proceedings of the 2017 IEEE International Conference on Robotics and Automation

[18] Nummiaro K, Koller-Meier E, Van Gool L. Object tracking with an adaptive color-based particle filter. In: Van Gool L, editor. Pattern Recognition. DAGM 2002. Lecture Notes in Computer Science, vol 2449. Berlin, Heidelberg: Springer; 2002. pp. 353-360. DOI: 10.1007/

[19] Yang S, Hao K, Ding Y, Liu J. Adaptively self-driving tracking algorithm based on particle filter. In: Proceedings of the 4th International Conference on Smart and Sustainable City

[20] Sans-Muntadas A, Brekke E, Pettersen KY. Vehicle guidance with control action computed by a rao-blackwellized particle filter. In: Proceedings of the 11th Asian Control Conference (ASCC'17); 17-20 December 2017; Gold Coast, QLD, Australia. pp. 2855-2860

[21] Xiao Y, Ou Y, Feng W. Localization of indoor robot based on particle filter with EKF proposal distribution. In: Proceedings of the 2017 IEEE International Conference on Cybernetics and Intelligent Systems and IEEE Conference on Robotics, Automation and

[22] Welch G, Bishop G. An Introduction to the Kalman Filter. Tech. Rep. TR95041. Chapel Hill:

[23] Si F, Zhao Y, Zhang X. Memory fading unscented Kalman filter and its application in navigation by stellar refraction. In: Proceedings of the 2017 IEEE Aerospace Conference; 4-

[24] Zhang T, Gao J, Ye F. Anti-spoofing algorithm based on adaptive Kalman filter for high dynamic positioning. In: Proceedings of the 2017 Progress in Electromagnetics Research Symposium-Fall (PIERS-FALL'17); 19–22 November 2017; Singapore, Singapore. pp. 838-

[25] Li K, Zhang K. Building occupancy estimation with robust Kalman filter. In: Proceedings of the 11th Asian Control Conference (ASCC'17); 17-20 December 2017; Gold Coast, QLD,

[26] Ren X, Du S, Zheng Y. Parallel RCNN: A deep learning method for people detection using RGB-D images. In: Proceedings of the 10th International Congress on Image and Signal

Mechatronics; 19–21 November 2017; Ningbo, China. pp. 568-571

Dept. Comput. Sci., Univ. North Carolina; July 2006

11 March 2017; Big Sky, MT, USA. pp. 1-8

neering (ISEEE'14); 26–28 April 2014; Sapporo, Japan. pp. 1576-1579

(ICRA'17); 29 May-3 June 2017; Singapore, Singapore. pp. 3740-3747

Sky, MT, USA. pp. 1-9

3–540-45783-6\_43

845

Australia. pp. 1406-1410

(ICSSC'17); 5-6 June 2017; Shanghai, China

[15] Yang JH, Hung JW. A preliminary study of emotion recognition employing adaptive Gaussian mixture models with the maximum a posteriori principle. In: Proceedings of the 2014 International Conference on Information Science, Electronics and Electrical Engineering (ISEEE'14); 26–28 April 2014; Sapporo, Japan. pp. 1576-1579

[3] Nam Y, Rho S, Park JH. Intelligent video surveillance system: 3-tier context-aware surveillance system with metadata. Multimedia Tools and Applications. 2012;57:315-334. DOI:

[4] Stauffer C, Grimson WEL. Adaptive background mixture models for real-time tracking.

[5] KaewTraKulPong P, Bowden R. An improved adaptive background mixture model for real-time tracking with shadow detection. In: Proceedings of the 2nd European Workshop on Advanced Video Based Surveillance Systems (AVBS'01); September. 2001. pp. 1-5

[6] Zhai Y, Yeary M. An intelligent video surveillance system based on multiple model particle filtering. In: Proceedings of the IEEE Instrumentation and Measurement Technol-

[7] Li SR, Tsai HC, Wang YK, Sun TH, Chen YJ. Particle-filter-based intelligent video surveillance system. In: Proceedings of the International Conference on System Science and

[8] Gaddigoudar PK, Balihalli TR, Ijantkar SS, Iyer NC. Pedestrian detection and tracking using particle filtering. In: Proceedings of the International Conference on Computing, Communication and Automation (ICCCA'17); 5–6 May 2017; Greater Noida, India. pp.

[9] Amor N, Chebbi S. Performance comparison of particle swarm optimization and extended Kalman filter methods for tracking in non-linear dynamic systems. In: Proceedings of the International Conference on Control, Automation and Diagnosis (ICCAD'17);

[10] Cheng HY, Hsu SH. Intelligent highway traffic surveillance with self-diagnosis abilities. IEEE Transactions on Intelligent Transportation Systems. 2011;12:1462-1472. DOI: 10.1109/

[11] Banu VC, Costea IM, Nemtanu FC, Bădescu I. Intelligent video surveillance system. In: Proceedings of the IEEE 23rd International Symposium for Design and Technology in Electronic Packaging (SIITME'17); 26–29 October 2017; Constanta, Romania. pp. 208-212

[12] Wahyono, Filonenko A, Kurnianggoro L, Jo KH. A fuzzy model-based integration framework for vision-based intelligent surveillance systems. In: Proceedings of the IEEE International Conference on Mechatronics (ICM'17); 13-15 February 2017; Churchill, VIC,

[13] Chung YC, Lai YK. A shape-perceived object tracking algorithm for intelligent surveillance systems. In: Proceedings of the IEEE International Conference on Consumer

[14] Filonenko A, Hernández DC, Shahbaz A, Jo KH. Unified smoke and flame detection for intelligent surveillance system. In: Proceedings of the IEEE 25th International Symposium on Industrial Electronics (ISIE'16); 8–10 June 2016; Santa Clara, CA, USA. pp. 953-957

Electronics-Taiwan (ICCE-TW'15); 6-8 June 2015; Taipei, Taiwan. pp. 394-395

ogy Conference Proceedings (IMTC'08); 12-15 May 2008; Victoria, BC, Canada

Computer Vision and Pattern Recognition. 1999;2:252-258

Engineering (ICSSE'16); 7-9 July 2016; Puli, Taiwan. pp. 1-4

19–21 Jan. 2017; Hammamet, Tunisia. pp. 116-119

10.1007/s11042-010-0677-x

16 Intelligent Video Surveillance

110-115

TITS.2011.2160171

Australia. pp. 358-361

Processing, BioMedical Engineering and Informatics (CISP-BMEI'17); 14–16 October 2017; Shanghai, China, China

**Chapter 2**

Provisional chapter

**Video Surveillance-Based Intelligent Traffic**

DOI: 10.5772/intechopen.76386

Visualization of video is considered as important part of visual analytics. Several challenges arise from massive video contents that can be resolved by using data analytics and consequently gaining significance. Though rapid progression in digital technologies resulted in videos data explosion that incites the requirements to create visualization and computer graphics from videos, a state-of-the-art algorithm has been proposed in this chapter for 3D conversion of traffic video contents and displaying on Google Maps. Time stamped visualization based on glyph is employed efficiently in surveillance videos and utilized for event detection. This method of visualization can possibly decrease the complexity of data, having complete view of videos from video collection. The effectiveness of proposed system has shown by obtaining numerous unprocessed videos and algorithm is tested on these videos without concerning field conditions. The proposed visualization technique produces promising results and found effective in conveying meaningful information while alleviating the need of searching exhaustively colossal amount of video data. Keywords: video visualization, traffic surveillance, smart cities, glyph-based visualization,

The quantity of surveillance video cameras increases at the public places results in increase in automated analysis of video contents and traffic video surveillance [43] considered as one of its application. These automated systems identify a number of traffic rule violations. Video features at object, pixel, and semantic level are extracted for analysis [53, 56, 59, 60]. The basic

> © 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and eproduction in any medium, provided the original work is properly cited.

© 2019 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

Video Surveillance-Based Intelligent Traffic

**Management in Smart Cities**

Management in Smart Cities

Shoab A. Khan and Richard Jiang

Shoab A. Khan and Richard Jiang

http://dx.doi.org/10.5772/intechopen.76386

Abstract

Google Maps

1. Video visualization in smart cities

Fozia Mehboob, Muhammad Abbas, Abdul Rauf,

Fozia Mehboob, Muhammad Abbas, Abdul Rauf,

Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

[27] Andriluka M, Roth S, Schiele B. People-tracking-by-detection and people-detection-bytracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08); 23-28 June 2008; Anchorage, AK, USA

#### **Chapter 2** Provisional chapter

#### **Video Surveillance-Based Intelligent Traffic Management in Smart Cities** Video Surveillance-Based Intelligent Traffic Management in Smart Cities

DOI: 10.5772/intechopen.76386

Fozia Mehboob, Muhammad Abbas, Abdul Rauf, Shoab A. Khan and Richard Jiang Fozia Mehboob, Muhammad Abbas, Abdul Rauf, Shoab A. Khan and Richard Jiang

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.76386

#### Abstract

Processing, BioMedical Engineering and Informatics (CISP-BMEI'17); 14–16 October 2017;

[27] Andriluka M, Roth S, Schiele B. People-tracking-by-detection and people-detection-bytracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recog-

nition (CVPR'08); 23-28 June 2008; Anchorage, AK, USA

Shanghai, China, China

18 Intelligent Video Surveillance

Visualization of video is considered as important part of visual analytics. Several challenges arise from massive video contents that can be resolved by using data analytics and consequently gaining significance. Though rapid progression in digital technologies resulted in videos data explosion that incites the requirements to create visualization and computer graphics from videos, a state-of-the-art algorithm has been proposed in this chapter for 3D conversion of traffic video contents and displaying on Google Maps. Time stamped visualization based on glyph is employed efficiently in surveillance videos and utilized for event detection. This method of visualization can possibly decrease the complexity of data, having complete view of videos from video collection. The effectiveness of proposed system has shown by obtaining numerous unprocessed videos and algorithm is tested on these videos without concerning field conditions. The proposed visualization technique produces promising results and found effective in conveying meaningful information while alleviating the need of searching exhaustively colossal amount of video data.

Keywords: video visualization, traffic surveillance, smart cities, glyph-based visualization, Google Maps

#### 1. Video visualization in smart cities

The quantity of surveillance video cameras increases at the public places results in increase in automated analysis of video contents and traffic video surveillance [43] considered as one of its application. These automated systems identify a number of traffic rule violations. Video features at object, pixel, and semantic level are extracted for analysis [53, 56, 59, 60]. The basic

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and eproduction in any medium, provided the original work is properly cited. © 2019 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

purposes of surveillance video-based systems are vehicle tracking, analyzing their patterns and behaviors, abnormal event prediction, and detecting anomalies before their occurrence. This research aims to develop a glyph-based system for the real-time video visualization covering a comprehensive set of traffic videos on complete length of highways.

video processing to human operator. Since statistical results are not easily comprehensible,

Video Surveillance-Based Intelligent Traffic Management in Smart Cities

http://dx.doi.org/10.5772/intechopen.76386

21

Conventional video surveillance systems heavily rely on human operators for activity monitoring and determining actions to be taken upon incident occurrence. There are several actionable incidents that miss-detect in such a manual system due to inherent limitations from deploying solely human operators eyeballing CCTV screens [58]. Hence, automatic VV [56] will prove very beneficial in improved traffic management. Miss-detections might be caused by monitoring excessive number of video screens to monitor as shown in Figure 1 and tiredness due to prolonged monitoring. In fact, numerous studies have shown the limits of human-dependent surveillance. The United States Sandia National Laboratories conducted a study in which most people attention fell below an adequate level after only 20 minutes of video surveillance screen monitoring [67]. The video content analysis paradigm is shifting from a fully human operated model to an intelligent machine-assisted automated model [58].

In the field of visualization, Borgo et al. [51] carried out a comprehensive survey on video visualization. Effectiveness of VV for conveying meaningful information enclosed in video sequences was demonstrated by Daniel et al. [1]. Andrienko et al. [47] also illustrated visual analytical technique to visualize huge amount of video data. Data were clustered and aggregated to display on map by using color arrows. Wang et al. [48] presented situational understanding approach by combining the video frame in 3D environment. Romero et al. [49] used visualization approach to analyze human behavior and explored the activity visualization in

Hoummady proposed survey on sensory device shortcomings that are used for collection of traffic information real time [40], and video camera usage as data collection was also proposed for traffic management. This approach relies on computational device mainly for pedestrian

For traffic visualization, commonly employed approach is coloring the areas demonstrating roads on the map [44]. Ang et al. [46] presented analytical approach for management of traffic from multiple cameras. Vehicle trajectory estimation and extraction of features was done. Subsequently, Jiang [62] demonstrated the analytical technique for visualizing the huge video data. Data were clustered and aggregated to display them on map by using color arrows. Afterward, Botchen [53] proposed technique for flow and volume signature visualization. It discovered that common people can recognize events on the basis of event signatures quite

End users and technology providers identify that manual process is inadequate to search comprehensively massive amount of video contents and screening timely. In order to lessen these issues of visualization, we try to project camera activity on Google Maps and have summarized and holistic view of video contents. Massive video data render ineffectual manual

whereas sequences of difference images again need sequential viewing [1].

2. State of the art

normal settings over time.

recognition and vehicle, 2-wheel vehicles, etc.

than viewing entire video contents.

Intelligent monitoring has rapidly progressed in last 10 years and intended to provide situational awareness and semantic information for understanding the environmental activity [14, 69]. VV illustrates the joint process of video analysis and subsequent derivation of representative presentation of essence of visual contents [2, 4, 19, 34, 45, 54, 57, 68]. The visualization of videos is gaining more attention because of addressing challenges of data analysis arisen from video camera contents [1, 15, 16]. Over the past decade, VV usefulness for traffic surveillance [17, 18] application has been effectively demonstrated by researchers [3, 75, 76].

VV offers spatio-temporal summary and overview of large collection of videos, and its abstract representation of meaningful information assists the users in video content [3, 35]. Conversely, conventional techniques [67] of visual representation such as time series plot have difficulties in conveying impressions from large video collection [3].

In addition, there is need to present visual contents of videos in compact forms such that user can quickly navigate through different segments of video sequence to locate segment of interest and zoom in to different detail levels [1]. Viewing videos is time-consuming process, consequently it is desirable to develop methods for highlighting and extraction interesting features in videos. There are numerous techniques designed for data analysis in images and a variety of statistical indicators for data processing. On the contrary, there is lack of effective techniques for conveying complex statistical information spontaneously to a layperson such as a security officer, apart from using line graphs to portray 1D signal levels [1]. Many researchers studied video processing in the context of video surveillance [16], monitoring vehicles, and monitoring crowds. However, main problem in automatic video processing is communication of results of

Figure 1. Video wall.

video processing to human operator. Since statistical results are not easily comprehensible, whereas sequences of difference images again need sequential viewing [1].

Conventional video surveillance systems heavily rely on human operators for activity monitoring and determining actions to be taken upon incident occurrence. There are several actionable incidents that miss-detect in such a manual system due to inherent limitations from deploying solely human operators eyeballing CCTV screens [58]. Hence, automatic VV [56] will prove very beneficial in improved traffic management. Miss-detections might be caused by monitoring excessive number of video screens to monitor as shown in Figure 1 and tiredness due to prolonged monitoring. In fact, numerous studies have shown the limits of human-dependent surveillance. The United States Sandia National Laboratories conducted a study in which most people attention fell below an adequate level after only 20 minutes of video surveillance screen monitoring [67]. The video content analysis paradigm is shifting from a fully human operated model to an intelligent machine-assisted automated model [58].

## 2. State of the art

purposes of surveillance video-based systems are vehicle tracking, analyzing their patterns and behaviors, abnormal event prediction, and detecting anomalies before their occurrence. This research aims to develop a glyph-based system for the real-time video visualization

Intelligent monitoring has rapidly progressed in last 10 years and intended to provide situational awareness and semantic information for understanding the environmental activity [14, 69]. VV illustrates the joint process of video analysis and subsequent derivation of representative presentation of essence of visual contents [2, 4, 19, 34, 45, 54, 57, 68]. The visualization of videos is gaining more attention because of addressing challenges of data analysis arisen from video camera contents [1, 15, 16]. Over the past decade, VV usefulness for traffic surveil-

VV offers spatio-temporal summary and overview of large collection of videos, and its abstract representation of meaningful information assists the users in video content [3, 35]. Conversely, conventional techniques [67] of visual representation such as time series plot have difficulties

In addition, there is need to present visual contents of videos in compact forms such that user can quickly navigate through different segments of video sequence to locate segment of interest and zoom in to different detail levels [1]. Viewing videos is time-consuming process, consequently it is desirable to develop methods for highlighting and extraction interesting features in videos. There are numerous techniques designed for data analysis in images and a variety of statistical indicators for data processing. On the contrary, there is lack of effective techniques for conveying complex statistical information spontaneously to a layperson such as a security officer, apart from using line graphs to portray 1D signal levels [1]. Many researchers studied video processing in the context of video surveillance [16], monitoring vehicles, and monitoring crowds. However, main problem in automatic video processing is communication of results of

covering a comprehensive set of traffic videos on complete length of highways.

lance [17, 18] application has been effectively demonstrated by researchers [3, 75, 76].

in conveying impressions from large video collection [3].

Figure 1. Video wall.

20 Intelligent Video Surveillance

In the field of visualization, Borgo et al. [51] carried out a comprehensive survey on video visualization. Effectiveness of VV for conveying meaningful information enclosed in video sequences was demonstrated by Daniel et al. [1]. Andrienko et al. [47] also illustrated visual analytical technique to visualize huge amount of video data. Data were clustered and aggregated to display on map by using color arrows. Wang et al. [48] presented situational understanding approach by combining the video frame in 3D environment. Romero et al. [49] used visualization approach to analyze human behavior and explored the activity visualization in normal settings over time.

Hoummady proposed survey on sensory device shortcomings that are used for collection of traffic information real time [40], and video camera usage as data collection was also proposed for traffic management. This approach relies on computational device mainly for pedestrian recognition and vehicle, 2-wheel vehicles, etc.

For traffic visualization, commonly employed approach is coloring the areas demonstrating roads on the map [44]. Ang et al. [46] presented analytical approach for management of traffic from multiple cameras. Vehicle trajectory estimation and extraction of features was done. Subsequently, Jiang [62] demonstrated the analytical technique for visualizing the huge video data. Data were clustered and aggregated to display them on map by using color arrows. Afterward, Botchen [53] proposed technique for flow and volume signature visualization. It discovered that common people can recognize events on the basis of event signatures quite than viewing entire video contents.

End users and technology providers identify that manual process is inadequate to search comprehensively massive amount of video contents and screening timely. In order to lessen these issues of visualization, we try to project camera activity on Google Maps and have summarized and holistic view of video contents. Massive video data render ineffectual manual analysis of videos; however, present automatic analytic techniques of videos undergo better performance.

successive frames. Image is scanned pixel by pixel, and gray value of central pixel is compared with those of the top and left pixels. Surface or region grows, until it finds all the connected pixels. The values of the pixels are compared with the 3 3 neighboring pixels. If there is disconnect in the connected pixels and the gap is greater than threshold value, algorithm classifies the pixel to a new region. It is a user defined threshold whose value is chosen on the basis of distance between the pixels. All the pixels which are part of the object are set to value 1, and those which are not part of the object are set to value zero. In region growing method, 3 3 window finds all the neighboring pixels 1 and keep growing the region until pixel with zero value is found. Algorithm keeps finding the gap, and if the gap is greater than the threshold value, the algorithm classifies it as a new region. If there are only isolated pixels, they are marked as outliers. Proposed system is robust in handling problems such as occlusion and illumination variation encountered in surveillance videos. In case of sunny day, there are moving shadows of vehicles which can produce false alarms. But the proposed system estimates the vehicle size and is able to predict the shadow size. Based on the moving vehicles size, shadow can be removed. Proposed method is able to remove the extracted shadow of the vehicles. Proposed system is tested on several surveillance videos of different scenarios such as different weather conditions and densities. Proposed system is robust in handling problems such as occlusion and illumination variation encountered in surveillance videos. The data set

Video Surveillance-Based Intelligent Traffic Management in Smart Cities

http://dx.doi.org/10.5772/intechopen.76386

23

contains a diverse set of scenarios in terms of traffic density and violations.

created for representing semantic info on Google space and video.

the base of flag value.

Traffic flow is assessed on each video frame, and the number of vehicles is counted in every frame. For every vehicle, mean speed is computed. The flow rate is found by dividing total vehicles by time. Top level flow diagram of the proposed approach is depicted in Figure 3.

Figure 3 represents the flow of proposed approach. Object tracking [7, 8] is part of the proposed system which collects temporal and spatial information about the object under consideration from the video sequence. Semantic information such as trajectories of detected objects is acquired by motion tracking that is given as input for mapping and 3D computation and revealing the outcomes on Google Maps. As Google space and video space coordinates are different, 3D mapping is performed amongst the two different spaces. Time-based glyph is

Layout of table in order to store coordinates of vehicle is revealed in Figure 4. Blobs detected within frame signify the number of vehicles. It is illustrated in Figure 4 that single vehicle exists in current video frame. Array is well-defined for storing vehicle coordinates. First two columns of array illustrate the y and x vehicle coordinates present in first frame, whereas the following y and x coordinates represents next frame coordinates of vehicle. Fifth column value demonstrates the number of frames consumed by vehicle in which vehicle becomes visible in field of view. Vanishing flag in last column defines the status of vehicle, for example, vehicle departure. Flag value remains zero till vehicles are in field of view and value will turn 1 when vehicles disappear. Last column is significant because values reshuffling in arrays change on

Though, trajectories of vehicle are of different spans even vehicle travel on the same route since vehicle travel at different mean speeds [8, 12]. Motion vectors [77] are used for demonstrating information as motion information has strong relationship with semantic occurrence. Different

A state of art visualization technique for surveillance videos is presented and tested by using several traffic videos. It receives suitable visual representations to assist the process of decision making. One can perceive level and pattern of activities that are recorded from visualization of videos as it offers more spatial info than using statistical indicators. Semantic info is obtained from numerous surveillance videos which are connected to Google Maps in order to perform 3D association. In the same time, glyph [5, 20] is familiar and conveys multi-field video visualization [10]. Well-developed visualization approach based on glyph is proposed that enables efficient and effective information encoding and visual communication.

## 3. Glyph-based semantic information visualization

Proposed approach aims to visualize semantic information of traffic videos using time stamped glyph. Input video frames are processed continuously to detect change in visual information. The proposed approach consists of several steps for estimation of traffic flow.

#### 3.1. Preprocessing

First step involves the segmentation of object from the surveillance video by using thresholding and subsequently converting it to binary image from grayscale image. Parts of road are thinned out, and holes are filled in video frames using morphological operations as shown in Figure 2.

Object segmentation considered to be a vital process in understanding of image in the preprocessing step. The purpose of object segmentation is to divide the image into region of interest, and objects are identified from the video frame using region growing method. The process of image segmentation results in binary image contains connected components which represents the multiple objects. Connected component analysis is performed to distinguish between the connected components. Features are extracted to track the moving objects in

Figure 2. Vehicle segmentation from video space.

successive frames. Image is scanned pixel by pixel, and gray value of central pixel is compared with those of the top and left pixels. Surface or region grows, until it finds all the connected pixels. The values of the pixels are compared with the 3 3 neighboring pixels. If there is disconnect in the connected pixels and the gap is greater than threshold value, algorithm classifies the pixel to a new region. It is a user defined threshold whose value is chosen on the basis of distance between the pixels. All the pixels which are part of the object are set to value 1, and those which are not part of the object are set to value zero. In region growing method, 3 3 window finds all the neighboring pixels 1 and keep growing the region until pixel with zero value is found. Algorithm keeps finding the gap, and if the gap is greater than the threshold value, the algorithm classifies it as a new region. If there are only isolated pixels, they are marked as outliers. Proposed system is robust in handling problems such as occlusion and illumination variation encountered in surveillance videos. In case of sunny day, there are moving shadows of vehicles which can produce false alarms. But the proposed system estimates the vehicle size and is able to predict the shadow size. Based on the moving vehicles size, shadow can be removed. Proposed method is able to remove the extracted shadow of the vehicles. Proposed system is tested on several surveillance videos of different scenarios such as different weather conditions and densities. Proposed system is robust in handling problems such as occlusion and illumination variation encountered in surveillance videos. The data set contains a diverse set of scenarios in terms of traffic density and violations.

analysis of videos; however, present automatic analytic techniques of videos undergo better

A state of art visualization technique for surveillance videos is presented and tested by using several traffic videos. It receives suitable visual representations to assist the process of decision making. One can perceive level and pattern of activities that are recorded from visualization of videos as it offers more spatial info than using statistical indicators. Semantic info is obtained from numerous surveillance videos which are connected to Google Maps in order to perform 3D association. In the same time, glyph [5, 20] is familiar and conveys multi-field video visualization [10]. Well-developed visualization approach based on glyph is proposed that

Proposed approach aims to visualize semantic information of traffic videos using time stamped glyph. Input video frames are processed continuously to detect change in visual information. The proposed approach consists of several steps for estimation of traffic flow.

First step involves the segmentation of object from the surveillance video by using thresholding and subsequently converting it to binary image from grayscale image. Parts of road are thinned out, and holes are filled in video frames using morphological operations as

Object segmentation considered to be a vital process in understanding of image in the preprocessing step. The purpose of object segmentation is to divide the image into region of interest, and objects are identified from the video frame using region growing method. The process of image segmentation results in binary image contains connected components which represents the multiple objects. Connected component analysis is performed to distinguish between the connected components. Features are extracted to track the moving objects in

enables efficient and effective information encoding and visual communication.

3. Glyph-based semantic information visualization

performance.

22 Intelligent Video Surveillance

3.1. Preprocessing

shown in Figure 2.

Figure 2. Vehicle segmentation from video space.

Traffic flow is assessed on each video frame, and the number of vehicles is counted in every frame. For every vehicle, mean speed is computed. The flow rate is found by dividing total vehicles by time. Top level flow diagram of the proposed approach is depicted in Figure 3.

Figure 3 represents the flow of proposed approach. Object tracking [7, 8] is part of the proposed system which collects temporal and spatial information about the object under consideration from the video sequence. Semantic information such as trajectories of detected objects is acquired by motion tracking that is given as input for mapping and 3D computation and revealing the outcomes on Google Maps. As Google space and video space coordinates are different, 3D mapping is performed amongst the two different spaces. Time-based glyph is created for representing semantic info on Google space and video.

Layout of table in order to store coordinates of vehicle is revealed in Figure 4. Blobs detected within frame signify the number of vehicles. It is illustrated in Figure 4 that single vehicle exists in current video frame. Array is well-defined for storing vehicle coordinates. First two columns of array illustrate the y and x vehicle coordinates present in first frame, whereas the following y and x coordinates represents next frame coordinates of vehicle. Fifth column value demonstrates the number of frames consumed by vehicle in which vehicle becomes visible in field of view. Vanishing flag in last column defines the status of vehicle, for example, vehicle departure. Flag value remains zero till vehicles are in field of view and value will turn 1 when vehicles disappear. Last column is significant because values reshuffling in arrays change on the base of flag value.

Though, trajectories of vehicle are of different spans even vehicle travel on the same route since vehicle travel at different mean speeds [8, 12]. Motion vectors [77] are used for demonstrating information as motion information has strong relationship with semantic occurrence. Different

where the flow vectors

tracking the object.

length of trajectories varies.

3.2. Bezier fitting for glyph generation

ft ¼ xt; yt

Bezier curve is defined as straight line between two points such as,

within field of view than outer circle of glyph is green.

Figure 5. (a) Chaotic vehicle trajectory, (b) smooth vehicle trajectory, and (c) time stamped glyph.

; vt x; vt y; at x; at y

represent object velocity [vx, vy], position [x, y], and direction [ax, ay] at time t extracted by

Bezier curve mostly employed for modeling and smoothing the chaotic vehicle trajectories. Control points are used to define Bezier curve which have geometric modeling interpretation and can model trajectories inconsistency [61]. Curve is confined in control point's which are showed graphically and can be utilized for curve manipulation. By offering P0 and P0 points,

B tðÞ¼ P0 þ t Pð Þ¼ 1 � P0 ð Þ I � t P0 þ tP1

That is equivalent to linear interpolation. Bezier curve is used to smooth the chaotic trajectories of vehicles obtained using motion tracking. As each car moves with different speeds, so the

Figure 5(c) depicts the video taken from area around Northumbria University having frame rate 30fps and video resolution 1920 � 1080. Video consists of 25 frames. Figure 5(a) illustrates the chaotic trajectories of different vehicles that are smoothed using Bezier curve to visualize the traffic pattern as shown in Figure 5(b). Time stamped semantic information is represented using glyph. Vehicle trajectory is tracked over time, and semantic info is delivered as presented in Figure 5. Outer circle of glyph denotes that vehicle changes lane although vehicle type was small which is signified by circle having red color. If vehicle is small and do not change the lane

0 ≤ t ≤ 1 (3)

Video Surveillance-Based Intelligent Traffic Management in Smart Cities

http://dx.doi.org/10.5772/intechopen.76386

h iT

(2)

25

Figure 3. Top level diagram of proposed approach.

Figure 4. Vehicle tracking information.

event identifications are done by analysis of motion features. Path demonstrates the vehicle movement and dynamical measurements that represent the raw vehicle trajectory. A common trajectory depiction is flow succession, for example,

$$\mathbf{F\_{T}} = \{f\_1, f\_2, \dots, \dots, f\_T\} \tag{1}$$

where the flow vectors

event identifications are done by analysis of motion features. Path demonstrates the vehicle movement and dynamical measurements that represent the raw vehicle trajectory. A common

FT ¼ f 1; f 2; ::………f T

(1)

trajectory depiction is flow succession, for example,

Figure 4. Vehicle tracking information.

Figure 3. Top level diagram of proposed approach.

24 Intelligent Video Surveillance

$$\mathbf{f}\_{\mathbf{t}} = \begin{bmatrix} \mathbf{x}\_{\mathbf{t}}, \mathbf{y}\_{\mathbf{t}}, \mathbf{v}\_{\mathbf{x}}^{t}, \mathbf{v}\_{\mathbf{y}}^{t}, \mathbf{a}\_{\mathbf{x}}^{t}, \mathbf{a}\_{\mathbf{y}}^{t} \end{bmatrix}^{T} \tag{2}$$

represent object velocity [vx, vy], position [x, y], and direction [ax, ay] at time t extracted by tracking the object.

#### 3.2. Bezier fitting for glyph generation

Bezier curve mostly employed for modeling and smoothing the chaotic vehicle trajectories. Control points are used to define Bezier curve which have geometric modeling interpretation and can model trajectories inconsistency [61]. Curve is confined in control point's which are showed graphically and can be utilized for curve manipulation. By offering P0 and P0 points, Bezier curve is defined as straight line between two points such as,

$$\begin{aligned} \mathbf{B}(\mathbf{t}) &= \mathbf{P}\_0 + t(P\_1 - P\_0) = (I - t)\mathbf{P}\_0 + t\mathbf{P}\_1 \\ \mathbf{0} &\le t \le 1 \end{aligned} \tag{3}$$

That is equivalent to linear interpolation. Bezier curve is used to smooth the chaotic trajectories of vehicles obtained using motion tracking. As each car moves with different speeds, so the length of trajectories varies.

Figure 5(c) depicts the video taken from area around Northumbria University having frame rate 30fps and video resolution 1920 � 1080. Video consists of 25 frames. Figure 5(a) illustrates the chaotic trajectories of different vehicles that are smoothed using Bezier curve to visualize the traffic pattern as shown in Figure 5(b). Time stamped semantic information is represented using glyph. Vehicle trajectory is tracked over time, and semantic info is delivered as presented in Figure 5. Outer circle of glyph denotes that vehicle changes lane although vehicle type was small which is signified by circle having red color. If vehicle is small and do not change the lane within field of view than outer circle of glyph is green.

Figure 5. (a) Chaotic vehicle trajectory, (b) smooth vehicle trajectory, and (c) time stamped glyph.

from one plane to second one. Homography amongst image space and video space is estimated requires four-point correspondence [42]. Calibration of image is acquired through transformation H, in which pixels pf image mapping on ground plane matches to latitude

Individual location of vehicle in each video frame sequence is signified by plus symbol in Figure 7 that is computed by homography matrix in order to calculate map and of video space coordinates. In perspective projection, location or points are alike in two dissimilar spaces, however, not equivalent because of universal scale uncertainty. The homography [6, 11, 45, 72, 79] in camera-based view geometry attains a particular interpretation H = KE, where E represents Euclidean transformation matrix which defines camera pose while viewing, and K characterizes the matrix of camera perspective recognized as intrinsic measures. Consider a pair of correspon-

� �T and u ¼ x2; y2; z2

h11 h12 h13 h14 h21 h22 h23 h24 h31 h32 h33 h34

Thus, each correspondence p ⇆ u results in two linear equations in the unknowns h ¼ h11; h12,……………, h34 ð ÞT. With manifold correspondences, numerous pairs of linear constraints need to be stored for obtaining coefficient matrix A. Least square h solution is acquired by

The h solution is acquired as eigenvector which corresponds to AT A smallest value. Corner points of video after the H computation are projected on Google Maps correspondence points. Each position of pixel in dimension space is estimated on map by the use of H matrix, and

3 7 5

2 6 4

x1 y1 z1

3 7

Video Surveillance-Based Intelligent Traffic Management in Smart Cities

http://dx.doi.org/10.5772/intechopen.76386

27

ATA � �h ¼ 0 (5)

x2 y2 z2

Figure 7. Homographic computation and perspective view of video and Google Maps.

2 6 4

2 6 4 � �T is related by homography H:

5 (4)

and longitude coordinates of maps.

dence points, for example, p ¼ x1; y1; z1

solving the

Figure 6. Proposed approach vehicle tracking.

#### 3.3. Motion tracking and semantic event display

Motion tracking significance [9, 39, 64, 66, 71, 73] in surveillance videos is unquestionable; subsequently, it is valuable in countless applications. Semantic analysis [62, 63] of video is utilized for extraction of vital information particularly type of vehicle, speed and lane changing, and trajectory from the video [38, 41]. This semantic info is extracted automatically in order to represent indexing, high level descriptors, retrieving, and searching the video contents. Tracking of vehicle comprises of velocity, maintenance of appearance, and positioning of detected object over time. Vehicle detection is done by object linking to most alike object in consecutive video frames.

Flow vectors are used to symbolize the common trajectory representation which is basis of further analysis. Figure 6(a) characterizes the chaotic vehicle trajectories which are taken from different surveillance videos. Every trajectory of vehicle is attained by individual tracking of detected vehicle. Figure 6(b) displays the smoothed curves that are acquired by applying Bezier curve on the chaotic vehicle trajectories.

## 4. 3D conversion and perspective view from video space to Google Maps

To capture the real time, info is considered as main challenge in dynamic VV [39]. 3D info recovery from surveillance video is essential to acquire some significant information from the videos. As frame of videos is the projection of 3D space, abstraction of vital information is difficult task. In proposed approach, 3D transformation on Google Maps from surveillance video is processed by using homographic transformation. In homographic transformation, plane mapping to image space is performed by projective transformation that maps the point from one plane to second one. Homography amongst image space and video space is estimated requires four-point correspondence [42]. Calibration of image is acquired through transformation H, in which pixels pf image mapping on ground plane matches to latitude and longitude coordinates of maps.

Individual location of vehicle in each video frame sequence is signified by plus symbol in Figure 7 that is computed by homography matrix in order to calculate map and of video space coordinates. In perspective projection, location or points are alike in two dissimilar spaces, however, not equivalent because of universal scale uncertainty. The homography [6, 11, 45, 72, 79] in camera-based view geometry attains a particular interpretation H = KE, where E represents Euclidean transformation matrix which defines camera pose while viewing, and K characterizes the matrix of camera perspective recognized as intrinsic measures.  With manifold correspondences, numerous pairs of linear constraints need to be stored for obtaining coefficient matrix A. Least square h solution is acquired by solving the

3.3. Motion tracking and semantic event display

Figure 6. Proposed approach vehicle tracking.

26 Intelligent Video Surveillance

Bezier curve on the chaotic vehicle trajectories.

consecutive video frames.

Motion tracking significance [9, 39, 64, 66, 71, 73] in surveillance videos is unquestionable; subsequently, it is valuable in countless applications. Semantic analysis [62, 63] of video is utilized for extraction of vital information particularly type of vehicle, speed and lane changing, and trajectory from the video [38, 41]. This semantic info is extracted automatically in order to represent indexing, high level descriptors, retrieving, and searching the video contents. Tracking of vehicle comprises of velocity, maintenance of appearance, and positioning of detected object over time. Vehicle detection is done by object linking to most alike object in

Flow vectors are used to symbolize the common trajectory representation which is basis of further analysis. Figure 6(a) characterizes the chaotic vehicle trajectories which are taken from different surveillance videos. Every trajectory of vehicle is attained by individual tracking of detected vehicle. Figure 6(b) displays the smoothed curves that are acquired by applying

4. 3D conversion and perspective view from video space to Google Maps

To capture the real time, info is considered as main challenge in dynamic VV [39]. 3D info recovery from surveillance video is essential to acquire some significant information from the videos. As frame of videos is the projection of 3D space, abstraction of vital information is difficult task. In proposed approach, 3D transformation on Google Maps from surveillance video is processed by using homographic transformation. In homographic transformation, plane mapping to image space is performed by projective transformation that maps the point

$$(\mathbf{A}^T \mathbf{A})\mathbf{h} = \mathbf{0} \tag{5}$$

The h solution is acquired as eigenvector which corresponds to AT A smallest value. Corner points of video after the H computation are projected on Google Maps correspondence points. Each position of pixel in dimension space is estimated on map by the use of H matrix, and

Figure 7. Homographic computation and perspective view of video and Google Maps.

resultant longitude and latitude coordinates are stored. Inverse of H is also computed to map the space coordinates on video.

## 5. Time stamped semantic glyph representation

Visualization based on glyph is considered as common procedure of visual scheme in which group of graphic objects is employed for representing data set known as glyph [35]. Glyph method is utilized for visualizing motion vectors that are over laid on video stream frame. Our main concern is to collect visual info that seems in all frames of video till the object remains within the view. Time stamped glyph is also generated in order to signify the type of car, speed with distinctive colors, and event information such as lane change information. The proposed system accurately determines the lane change of vehicle at a specific time due to precise localization. In the proposed system, an abnormal event detection is performed by specifically giving vehicle trajectories [52]. Trajectory analysis and interaction with scene feature allows recognizing interesting events. A time stamped glyph is generated to represent speed and lane change information of vehicle. For any image point, the position of corresponding scene point in every video frame is determined until the vehicle leaves the field of view.

There is variation in vehicle speed even in obstacles' absence because of curves and turns. Experimental data authenticate the common insight of speed which is considered most significant factors of safe driving. Variation in vehicle speed considered to be one of likely factors of congestions and accidents [37, 51, 94]. Therefore, proposed algorithm determines the speed variation of vehicles in each frame on the basis of trajectory analysis. Trajectories with different speeds are identified and represented using glyph. At each time frame, if vehicle speed is lower than the threshold, then the same color is assigned; however, if vehicle speed changes abruptly then at each instance of time, it is assigned a different color. With this time stamped identification method, precise instance of speed variation is identified in the video frame that causes a disruption in flow of the traffic movement.

7. Holistic view of video using Google Maps

Figure 8. Holistic view of videos on Google Maps.

to represent unusual events in videos as depicted in Figure 8.

trajectory on Google Maps as shown in Figure 10.

7.1. Small scale

Figure 9.

A surveillance video naturally takes the perspective view of the visual scene which is recognized as quasi-3D. Significant information is gathered from the different videos and is viewed

Video Surveillance-Based Intelligent Traffic Management in Smart Cities

http://dx.doi.org/10.5772/intechopen.76386

29

In video surveillance-based system, identification of unusual events is considered to be most significant task. Anomalous behavior can be drastic and subtle [36, 58, 63]. Changing of lanes on highways is traumatic. The proposed system precisely identifies the vehicle lane change at specific time because of precise localization. Anomalous detection of events [40] can be performed by giving the trajectory [62]. Subsequently, now the vehicle trajectory specifies the frightening behavior by performing trajectory analysis. Different glyph colors during the video visualization portray the type, vehicle position, and event information within video frame.

The proposed technique has been tested on the small scale, for example, area across Northumbria University City Campus, Newcastle Upon Tyne, UK. Detected object trajectories are shown in outcome till the objects remain in the scene using semantic glyph as shown in

There is possibility of future work in the area of visualization. Proposed visualization approach can be utilized for traffic management system at city level and have precise view of bigger cite. Spatio-temporal view of collection of videos can be acquired by mapping the

## 6. Association between Google Maps and video visualization

To properly visualize analysis of results on Google Maps, the output must be properly aligned to the map coordinates [13]. Rectification of camera image is automatically done and mapped on the map. In surveillance video, activity is detected in each frame, and location of vehicle on ground is gained through correspondence points and trajectory learning which are mapped on map. Consequently, video inspection of several road cameras is upgraded by projecting the activity of outdoor surveillance camera on Google Maps. In order to localize the vehicle coordinates on the Google Maps or association of video and Google Maps space, homography is computed, and its perspective view is drawn. Transformation matrix provides the association information and its mapping. And as the events occur, correspondence video is visualized on the Google Maps as shown in Figure 8.

Video Surveillance-Based Intelligent Traffic Management in Smart Cities http://dx.doi.org/10.5772/intechopen.76386 29

Figure 8. Holistic view of videos on Google Maps.

## 7. Holistic view of video using Google Maps

A surveillance video naturally takes the perspective view of the visual scene which is recognized as quasi-3D. Significant information is gathered from the different videos and is viewed to represent unusual events in videos as depicted in Figure 8.

In video surveillance-based system, identification of unusual events is considered to be most significant task. Anomalous behavior can be drastic and subtle [36, 58, 63]. Changing of lanes on highways is traumatic. The proposed system precisely identifies the vehicle lane change at specific time because of precise localization. Anomalous detection of events [40] can be performed by giving the trajectory [62]. Subsequently, now the vehicle trajectory specifies the frightening behavior by performing trajectory analysis. Different glyph colors during the video visualization portray the type, vehicle position, and event information within video frame.

#### 7.1. Small scale

resultant longitude and latitude coordinates are stored. Inverse of H is also computed to map

Visualization based on glyph is considered as common procedure of visual scheme in which group of graphic objects is employed for representing data set known as glyph [35]. Glyph method is utilized for visualizing motion vectors that are over laid on video stream frame. Our main concern is to collect visual info that seems in all frames of video till the object remains within the view. Time stamped glyph is also generated in order to signify the type of car, speed with distinctive colors, and event information such as lane change information. The proposed system accurately determines the lane change of vehicle at a specific time due to precise localization. In the proposed system, an abnormal event detection is performed by specifically giving vehicle trajectories [52]. Trajectory analysis and interaction with scene feature allows recognizing interesting events. A time stamped glyph is generated to represent speed and lane change information of vehicle. For any image point, the position of corresponding scene point

There is variation in vehicle speed even in obstacles' absence because of curves and turns. Experimental data authenticate the common insight of speed which is considered most significant factors of safe driving. Variation in vehicle speed considered to be one of likely factors of congestions and accidents [37, 51, 94]. Therefore, proposed algorithm determines the speed variation of vehicles in each frame on the basis of trajectory analysis. Trajectories with different speeds are identified and represented using glyph. At each time frame, if vehicle speed is lower than the threshold, then the same color is assigned; however, if vehicle speed changes abruptly then at each instance of time, it is assigned a different color. With this time stamped identification method, precise instance of speed variation is identified in the video frame that

To properly visualize analysis of results on Google Maps, the output must be properly aligned to the map coordinates [13]. Rectification of camera image is automatically done and mapped on the map. In surveillance video, activity is detected in each frame, and location of vehicle on ground is gained through correspondence points and trajectory learning which are mapped on map. Consequently, video inspection of several road cameras is upgraded by projecting the activity of outdoor surveillance camera on Google Maps. In order to localize the vehicle coordinates on the Google Maps or association of video and Google Maps space, homography is computed, and its perspective view is drawn. Transformation matrix provides the association information and its mapping. And as the events occur, correspondence video is visualized

in every video frame is determined until the vehicle leaves the field of view.

6. Association between Google Maps and video visualization

causes a disruption in flow of the traffic movement.

on the Google Maps as shown in Figure 8.

the space coordinates on video.

28 Intelligent Video Surveillance

5. Time stamped semantic glyph representation

The proposed technique has been tested on the small scale, for example, area across Northumbria University City Campus, Newcastle Upon Tyne, UK. Detected object trajectories are shown in outcome till the objects remain in the scene using semantic glyph as shown in Figure 9.

There is possibility of future work in the area of visualization. Proposed visualization approach can be utilized for traffic management system at city level and have precise view of bigger cite. Spatio-temporal view of collection of videos can be acquired by mapping the trajectory on Google Maps as shown in Figure 10.

data and videos. While visualizing the traffic information, approach was ineffective in simulating all types of kinematics and dynamics because of driving behavior in various regions.

The concern of VV is with visual illustration of input surveillance video for see-through vital features and events in surveillance video. It is envisioned for providing assistance in intellectual reasoning whereas easing the load of observing videos. A novel visualization approach based on glyph has been proposed that can be efficiently utilized for road surveillance videos. A visual analysis is done on the basis of motion tracking to monitor live road traffic on the highways. The proposed approach has been verified on numerous video frame rates and resolution for visualizing the traffic flows. Experimental outcomes illustrate that approach can be employed in field

, Abdul Rauf2

, Shoab A. Khan1 and Richard Jiang3

Video Surveillance-Based Intelligent Traffic Management in Smart Cities

http://dx.doi.org/10.5772/intechopen.76386

31

conditions and permit better utilization of previous systems of traffic management.

8. Conclusion

Author details

References

2003

1667-1678

puting Systems; ACM; 2013

Fozia Mehboob1,3\*, Muhammad Abbas1

\*Address all correspondence to<EMAIL_ADDRESS>
1 National University of Sciences and Technology, Islamabad, Pakistan

Visualization and Computer Graphics; 2012. pp. 2095-2103

Visualization and Computer Graphics; 2015. pp. 980-993

2 Imam Mohammed ibn Saud Islamic University (IMSIU), Riyadh, Saudi Arabia

3 Northumbria University of Digital Science and Technology, Newcastle Upon Tyne, UK

[1] Daniel G, Chen M. Video visualization. Proceedings of the 14th IEEE Visualization. IEEE;

[2] Höferlin M et al. Evaluation of fast-forward video visualization. IEEE Transactions on

[3] Duffy B et al. Glyph-based video visualization for semen analysis. IEEE Transactions on

[4] Morris BT et al. Real-time video-based traffic measurement and visualization system for energy/emissions. IEEE Transactions on Intelligent Transportation Systems; 2012. pp.

[5] Fuchs J et al. Evaluation of alternative glyph designs for time series data in a small multiple setting. In: Proceedings of the SIGCHI Conference on Human Factors in Com-

Figure 9. Time stamped glyph-based video visualization on Google Maps.

Figure 10. Multiple video visualization on large scale.

To interpret the data in real time system, visualization of video data offers instinctive information that can be expended for acquiring trends and patterns. Conversely, gathering statistics automatically from video data are computationally costly. Subsequently, Walton et al. [39] visualized the traffic video data on Google Maps to display traffic info. Though, displaying numerous traffic videos instantaneously was challenging because of heavy transmission load. Human intellect was used to gather semantic features from surveillance videos in graphic mapping scheme. Lately, Hsieh and Wang [50] proposed a traffic system for visualizing traffic information by inferring vehicle data and constitute a video in the database. Flow of traffic was assessed from surveillance videos and Google mapping was created amongst vehicle detector data and videos. While visualizing the traffic information, approach was ineffective in simulating all types of kinematics and dynamics because of driving behavior in various regions.

## 8. Conclusion

The concern of VV is with visual illustration of input surveillance video for see-through vital features and events in surveillance video. It is envisioned for providing assistance in intellectual reasoning whereas easing the load of observing videos. A novel visualization approach based on glyph has been proposed that can be efficiently utilized for road surveillance videos. A visual analysis is done on the basis of motion tracking to monitor live road traffic on the highways. The proposed approach has been verified on numerous video frame rates and resolution for visualizing the traffic flows. Experimental outcomes illustrate that approach can be employed in field conditions and permit better utilization of previous systems of traffic management.

## Author details

Fozia Mehboob1,3\*, Muhammad Abbas1 , Abdul Rauf2 , Shoab A. Khan1 and Richard Jiang3

\*Address all correspondence to<EMAIL_ADDRESS>

## References

To interpret the data in real time system, visualization of video data offers instinctive information that can be expended for acquiring trends and patterns. Conversely, gathering statistics automatically from video data are computationally costly. Subsequently, Walton et al. [39] visualized the traffic video data on Google Maps to display traffic info. Though, displaying numerous traffic videos instantaneously was challenging because of heavy transmission load. Human intellect was used to gather semantic features from surveillance videos in graphic mapping scheme. Lately, Hsieh and Wang [50] proposed a traffic system for visualizing traffic information by inferring vehicle data and constitute a video in the database. Flow of traffic was assessed from surveillance videos and Google mapping was created amongst vehicle detector

Figure 9. Time stamped glyph-based video visualization on Google Maps.

30 Intelligent Video Surveillance

Figure 10. Multiple video visualization on large scale.

[6] Ranganathan P, Olson E. Locally-weighted homographies for calibration of imaging systems. IEEE/RSJ International Conference on Intelligent Robots and Systems; IEEE; 2014

[21] Bertin J. Semiology of Graphics: Diagrams, Networks, Maps; Madison, Wis.; London:

Video Surveillance-Based Intelligent Traffic Management in Smart Cities

http://dx.doi.org/10.5772/intechopen.76386

33

[22] Maguire E, Rocca-Serra P, Sansone S-A, Davies J, Chen M. Taxonomy-based glyph design —With a case study on visualizing workflows of biological experiments. IEEE Transac-

[23] Post F, Vrolijk B, Hauser H, Laramee R, Doleisch H. The state of the art in flow visualisation: Feature extraction and tracking. Computer Graphics Forum. 2003;22(4):775-792 [24] Wong PC, Foote H, Kao DL, Leung R, Thomas J. Multivariate visualization with data

[25] Hlawitschka M, Scheuermann G, Hamann B. Interactive glyph placement for tensor fields. In: International Symposium on Visual Computing. Berlin Heidelberg: Springer;

[26] Fuchs R, Hauser H. Visualization of multi-variate scientific data. Computer Graphics

[27] Pearlman J, Rheingans P. Visualizing network security events using compound glyphs from a service-oriented perspective. In: VizSEC 2007. Berlin Heidelberg: Springer; 2008. pp. 131-146

[28] Aigner W, Miksch S, Schumann H, Tominski C. Visualization of Time-Oriented Data.

[29] Hlawatsch M, Leube P, Nowak W, Weiskopf D. Flow radar glyphs—Static visualization of unsteady flow with uncertainty. IEEE Transactions on Visualization and Computer

[30] Peng Z, Grundy E, Laramee R, Chen G, Croft N. Mesh-driven vector field clustering and visualization: An image-based approach. IEEE Transactions on Visualization and Com-

[31] Ropinski T, Preim B. Taxonomy and usage guidelines for glyph-based medical visualiza-

[32] Ropinski T, Oeltze S, Preim B. Survey of glyph-based visualization techniques for spatial

[33] Chung DH, Legg PA, Parry ML, Bown R, Griffiths IW, Laramee RS, et al. Glyph sorting: Interactive visualization for multi-dimensional data. Information Visualization. 2015;

[34] Botchen RP, Bachthaler S, Schick F, Chen M, Mori G, Weiskopf D, et al. Action-based multifield video visualization. IEEE Transactions on Visualization and Computer Graphics.

[35] Parry ML, Legg PA, Chung DHS, Grif-Fiths IW, Chen M. Hierarchical event selection for video storyboards with a case study on snooker video visualization. IEEE Transactions on

multivariate medical data. Computers & Graphics. 2011;35(2):392-401

Visualization and Computer Graphics. 2011;17:1747-1756

tions on Visualization and Computer Graphics. 2012;18(12):2603-2612

fusion. Information Visualization. 2002;1(3/4):182-193

University of Wisconsin Press; 1983

2007. pp. 331-340

Forum. 2009;28(6):1670-1690

Berlin, Springer-Verlag; 2011

Graphics. 2011;17(12):1949-1958

puter Graphics. 2012;18(5):283-298

tion. In: SimVis; 2008. pp. 121-138

14(1):76-90

2008;14(4):885-899

[21] Bertin J. Semiology of Graphics: Diagrams, Networks, Maps; Madison, Wis.; London: University of Wisconsin Press; 1983

[6] Ranganathan P, Olson E. Locally-weighted homographies for calibration of imaging systems. IEEE/RSJ International Conference on Intelligent Robots and Systems; IEEE; 2014 [7] Chincholkar AA. Moving object tracking and detection in videos using MATLAB: A review. International Journal of Advent Research in Computer and Electronics (IJARCE).

[8] Morris BT, Mohan MT. Learning, modeling, and classification of vehicle track patterns from live video. IEEE Transactions on Intelligent Transportation Systems; 2008; pp. 425-437 [9] Kappe CP et al. Reconstruction and visualization of coordinated 3D cell migration based on optical flow. IEEE Transactions on Visualization and Computer Graphics; 2016; pp.

[10] Borgo R, Kehrer J, Chung DH, Maguire E, Laramee RS, Hauser H et al. Glyph-based visualization: Foundations, design guidelines, techniques and applications. Eurographics

[11] Dubrofsky E. Homography estimation [Doctoral dissertation]. University of British Columbia

[12] Morris BT, Mohan MT. A survey of vision-based trajectory learning and analysis for surveillance. San Diego: IEEE Transactions on Circuits and Systems for Video Technology;

[13] Morris B, Trivedi MM. Contextual Activity Visualization from Long-Term Video Obser-

[14] Dee HM, Velastin SA. How close are we to solving the problem of automated visual

[15] Cavallaro A, Ebrahimi T. Change detection based on color edges. In: IEEE International Symposium on Circuits and Systems (No. 2, pp. 141-144) (2001, May); IEEE; 1999

[16] Collins RT, Lipton AJ, Kanade T, Fujiyoshi H, Duggins D, Tsin Y et al. A system for video surveillance and monitoring (pp. 1-6). Technical Report CMU-RI-TR-00-12; Robotics Insti-

[17] Dollár P, Rabaud V, Cottrell G, Belongie S. Behavior recognition via sparse spatiotemporal features. In: IEEE International Workshop on Visual Surveillance and Perfor-

[18] Girgensohn A, Kimber D, Vaughan J, Yang T, Shipman F, Turner T et al. DOTS: Support for effective video surveillance. In: Proceedings of the 15th ACM international conference on

[19] Ward MO. Multivariate data glyphs: Principles and practice. In: Handbook of data visu-

[20] Ward MO. A taxonomy of glyph placement strategies for multidimensional data visuali-

vations. San Diego: University of California Transportation Center; 2010

surveillance? Machine Vision and Applications. 2008;19(5):329-343

mance Evaluation of Tracking and Surveillance; IEEE. pp. 65-72

alization. Berlin Heidelberg: Springer; 2008. pp. 179-198

zation. Information Visualization. 2002;1(3/4):194-210

2014;1(5):2348-5523

State of the Art Reports; 2013. pp. 39-63