Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
10.1109/ACCESS.2017.DOI

This work was supported by the Institute for Information and Communications
Technology Promotion Grant through the Korea Government (MSIT) under Grant
R7120-17-1007 (SIAT CCTV Cloud Platform).

Corresponding author: Young-Koo Lee (e-mail<EMAIL_ADDRESS>
# Video Big Data Analytics in the Cloud: A Reference Architecture, Survey,
Opportunities, and Open Research Issues

AFTAB ALAM  IRFAN ULLAH, AND YOUNG-KOO LEE  Department of Computer Science and
Engineering, Kyung Hee University (Global Campus), Yongin 1732, South Korea

###### Abstract

The proliferation of multimedia devices over the Internet of Things (IoT)
generates an unprecedented amount of data. Consequently, the world has stepped
into the era of big data. Recently, on the rise of distributed computing
technologies, video big data analytics in the cloud has attracted the
attention of researchers and practitioners. The current technology and market
trends demand an efficient framework for video big data analytics. However,
the current work is too limited to provide a complete survey of recent
research work on video big data analytics in the cloud, including the
management and analysis of a large amount of video data, the challenges,
opportunities, and promising research directions. To serve this purpose, we
present this study, which conducts a broad overview of the state-of-the-art
literature on video big data analytics in the cloud. It also aims to bridge
the gap among large-scale video analytics challenges, big data solutions, and
cloud computing. In this study, we clarify the basic nomenclatures that govern
the video analytics domain and the characteristics of video big data while
establishing its relationship with cloud computing. We propose a service-
oriented layered reference architecture for intelligent video big data
analytics in the cloud. Then, a comprehensive and keen review has been
conducted to examine cutting-edge research trends in video big data analytics.
Finally, we identify and articulate several open research issues and
challenges, which have been raised by the deployment of big data technologies
in the cloud for video big data analytics. To the best of our knowledge, this
is the first study that presents the generalized view of the video big data
analytics in the cloud. This paper provides the research studies and
technologies advancing the video analyses in the era of big data and cloud
computing.

###### Index Terms:

big data, intelligent video analytics, cloud-based video analytics system,
video analytics survey, deep learning, distributed computing, intermediate
results orchestration, cloud computing.

=-15pt

##  I Introduction

Videos are generated and uploaded regularly to the cloud. Many sources include
CCTV, smartphones, drones, etc., are actively contributing to video generation
leads to the evolution of Intelligent Video Analytics (IVA) and management
systems. IVA is a domain that uses advanced computer vision technologies to
process and extract insights from streaming or stored videos automatically.
Over the past two decades, IVA is extensively rising from real-world needs
driving by a broader range of application domains ranging from security and
surveillance to transportation and healthcare. The IVA market is anticipated
to rise from 4.9 billion USD in 2020 to 11.7 billion USD by 2025 at a compound
annual growth rate of 19.0% [1].

Video management and services providers such as Facebook [2], YouTube [3], and
Netflix [4] are considered as valuable sources of large-scale video data.
Along with these, various leading industrial organizations have successfully
deployed video management and analytics platforms that provide more bandwidth
and high-resolution cameras collecting videos at scale and has become one of
the latest trends in the video surveillance industry. For example, more than
400 hours of videos are uploaded in a minute on Youtube [5], and more than one
hundred and seventy million video surveillance cameras have been installed in
china only [6]. It has been reported that the data generated by various IoT
devices will see a growth rate of 28.7% over the period 2018-2025, where
surveillance videos are the majority shareholder [7].

Such an enormous video data is considered as “big data” because a variety of
sources generates a large volume of video data at high velocity that holds
high Value. Even though 65% of the big data shares hold by surveillance videos
are monitored, but still, a significant portion of video data has been failed
to notice[8]. That neglected data contain valuable information directly
related to real-world situations. Video data provide information about
interactions, behaviors, and patterns, whether its traffic or human patterns.
However, handling such a large amount of complex video data is not worthwhile
utilizing traditional data analytical approaches. Therefore, more
comprehensive and sophisticated solutions are required to manage and analyses
such large-scale unstructured video data.

Due to the data-intensive and resources hungry nature of large scale video
data processing, extracting the insights from the video is a challenging task.
A considerable size of video data poses significant challenges for video
management and mining systems that require powerful machines to deal with
large-scale video data. Moreover, a flexible solution is necessary to store
and mine this large volume of video data for decision making. However, large-
scale video analytics becomes a reality due to the popularity of big data and
cloud computing technologies.

Cloud computing is an infrastructure for providing convenient and ubiquitous
remote access to a shared pool of configurable computing resources. These
resources can be managed with minimal management effort or service [9]. Big
data technologies, such as Hadoop or Spark echo system, are software platforms
designed for distributed computing to process, analyze, and extract the
valuable insights from large datasets in a scalable and reliable way. The
cloud is preferably appropriate to offer the big data computation power
required for the processing of these large datasets. [10], Amazon web service
[11], Microsoft Azure [12], and Oracle Big Data Analytics [13] are some
examples of video big data analytics platforms. Large-scale video analytics in
the cloud is a multi-disciplinary area, and the next big thing in big data,
which opens new research avenues for researchers and practitioners.

This work aims to conduct a comprehensive study on the status of large scale
video analytics in the cloud-computing environment while deploying video
analytics techniques. First, this study builds the relationship between video
big data and cloud computing and defines the terminologies that govern the
study. Then service-oriented and a layered reference architecture have been
proposed for large-scale video analytics in the cloud while focusing on
architectural properties like reliability, scalability, fault-tolerance,
extensibility, and intermediate results orchestration. Further, an intensive
survey has been conducted to project the current research trends in video
analytics that encompass the taxonomy of video analytics approaches, and
cloud-based scholarly and industrial study. Finally, open research issues and
challenges are discussed, with a focus on proposed architecture, i.e., the
deployment of an array of computer vision algorithms for large-scale videos in
the cloud.

###  I-A Video Big Data, Cloud Computing, and their Relationship

The term big data appeared and popularized by John R. Masey in the late 1990s
[14], which refers to a large volume of data that are impractical to be
stored, processed and analyzed using traditional data management and
processing technologies [15]. The data can be unstructured, semi-structured,
and structured data, but mostly unstructured data is considered. The
definition of big data evolved and has been described in terms of three, four,
or five characteristics. In literature, among these characteristics, three are
shared, i.e., Volume, Velocity, and Variety, while the others are Veracity and
Value [16, 17, 18, 19]. Various video stream sources generate a considerable
amount of unstructured video data on a regular bases and becoming a new
application field of big data. The data generated by such sources are further
subject to contextual analysis and interpretation to uncover the hidden
patterns for decision-making and business purposes.

In the context of a large volume of video data, we specialize the generic big
data characteristics. The size of data is referred to as Volume [20], but the
majority of the shares, i.e., 65%, are held only by surveillance videos. The
type of data generated by various sources such as text, picture, video, voice,
and logs are known as Variety [20]. The video data are acquired from
multimodal video stream sources, e.g., IP-Camera, depth camera, body-worn
camera, etc., and from different geolocations, which augments the Variety
property. The pace of data generation and transmission is known as Velocity
[21]. The video data also possess the Velocity attribute, i.e., the Video
Stream Data Source (VSDS) primarily produce video stream 24/7 and acquired by
the data center storage servers. Veracity can be defined as the diversity of
quality, accuracy, and trustworthiness of the data [22]. Video data are
acquired directly from real-world domains and meet the Veracity
characteristic. The Value refers to contextual analysis to extract the
significant values for decision-making and business purpose [23, 24]. Video
data has high Value because of its direct relation with real-word. Automatic
criminal investigation, illegal vehicle detection, and abnormal activity
recognition are some of the examples of Value extraction. Almost all the big
data properties are dominated by the video data, which encourage us to give
birth to Video Big Data.

These five characteristics impose many challenges on the organizations when
embracing video big data analytics. Storing, scaling, and analyzing are some
apparent challenges associated with video big data. To cope with these
challenges, converged and hyper-converged infrastructure and software-defined
storage are the most convenient solutions. Distributed databases, data
processing engines, and machine learning libraries have been introduced to
overcome video big data management, processing, and analysis issues,
respectively.

These big data technologies are deployed over a computer cluster to process
and manage a massive amount of video data in parallel. A computer cluster may
consist of few to hundreds, and even thousands of nodes work together as a
single integrated computing resource, on different parts of the same program
[25, 26]. Deploying an indoor computer cluster is an option for big data
technologies, but hardware cost and maintenance issues are associated with it.
An alternative solution can be cloud computing that elegantly reduces the
costs associated with the management of hardware and software resources [27].

Typically, cloud services are provided on-demand in a “pay-as-you-go” manner
for the conveniences of end-users and organizations [28]. Cloud computing
follows the philosophy of the “as-a-Service” and offers its “services”
according to different models, for example, Infrastructure-as-a-Service
(IaaS), Platform-as-a-Service (PaaS), Software-as-a-Service (SaaS) [9].

Under IaaS (e.g., Amazon’s EC2 ), the cloud service provider facilitates and
allows the consumers to provision fundamental computing resources and deploy
arbitrary software. In PaaS, the service provider provides a convenient
platform enabling customers to develop, run, and manage applications without
considering the complexities of building and maintaining the infrastructure.
The examples of PaaS are Google’s Apps Engine and Microsoft Azure. In SaaS,
applications (e.g., email, docs, etc.) are deployed on cloud infrastructure by
service providers and allow the consumer to subscribe. These applications can
be easily accessed from various client devices using a thin client or program
interfaces [9].

###  I-B Research Objectives and Contributions

This paper presents a detailed survey and review of cloud-based large-scale
IVA. We also propose big data technological solutions for the challenges faced
by IVA researchers and practitioners. The contributions of this paper are
listed below:

  * •

We standardize the basic nomenclatures that govern the IVA domain. The term
video big data has been coined and clarified how it inherits the big data
characteristics while establishing its relationship with cloud computing.

  * •

We propose a distributed, layered, service-oriented, and lambda style [29]
inspired reference architecture for large-scale IVA in the cloud. Each layer
of the proposed Cloud-based Video Analytics System (CVAS) has been elaborated
technologically, i.e., layerwise available big data technological
alternatives. The base layer of the proposed architecture, i.e., video big
data curation layer, is based on the notion of Intermediate Results (IR)
orchestration, which can play a significant role in the optimization of the
IVA pipeline. Under the proposed architecture, we perform a thorough
investigation of scalable traditional video analytics and deep learning
techniques and tools on distributed infrastructures along with popular
computer vision benchmark datasets.

  * •

To the best of our knowledge, this is a first surveying video big data
analytics, our research targets the most recent approaches that encompass
broad IVA research domains like content-based video retrieval, video
summarization, semantic-based approaches, and surveillance and security.

  * •

We also investigate how the researchers exploit big data technologies for
large-scale video analytics and show how the industrial IVA solutions are
provided.

  * •

In this study, we further write real word IVA application areas, which
projects the significance of big data and cloud computing in IVA.

  * •

Finally, we identify the research gap and list several open research issues
and challenges. These research issues encompass orchestration and optimization
of IVA pipeline, big dimensionality, online learning on video big data, model
management, parameter servers, and distributed learning, evaluation issues and
opportunities, IVA services statistics maintenance and ranking in the cloud,
IVA-as-a-Service (IVAaaS) and cost model, video big data management, and
privacy, security and trust.

The remaining paper is organized as follows. Section II and III is about
recent studies, and scope and nomenclature, respectively. Section IV discusses
the proposed CVAS. Literature review has been presented in Section V and VI.
The IVA applications can be seen in Section VII. In Section VIII several video
big data challenges and opportunities are discussed. Finally, Section IX
concludes this study.

##  II Recent Studies

Various studies have been conducted that discus video analytics, which can be
classified as IVA and Big data, as shown in Table I. In the former class, the
surveys overlook the big data characteristics and challenges of video analysis
in the cloud. The majority of the work focuses on a specific domain of IVA and
the application of video analyses. For instance, a comprehensive survey was
presented by Liu et al. [30], and Olatunji et al. [31], where they have
discussed IVA and its applications in the context of surveillance
conventionally. Context-Based Video Retrieval (CBVR) was reviewed by Hu et al.
[32], Patel et al. [33], and Haseyama et al. [34]. Similarly, vehicle
surveillance systems in Intelligent Transportation System (ITS), abnormal
behavior analytics, in surveillance videos were studied by Tian et al. [35],
and [36], respectively.

TABLE I: Recent Surveys Related to Video Big Data Analytics,

Class | References | Target Domain | Pros | Cons
---|---|---|---|---
Int. Video Analytics |  Liu et al. [30] Olatunji et al. [31] |  | Video analytics
---
based surveillance
| Comprehensive theory & applications of
---
video analytics-based surveillance
| -Lacking big data & cloud computing
---
and its research issues
\- Not service oriented
Hu et al. [32] Patel et al. [33] Haseyama et al. [34] |  | CBVR
---
| \- CBVR components, research trends
---
and issues
| -Lack of big data & cloud computing,
---
& its research issues.
-Limited to CBVR
Khan et al. [37] Tsai et al. [38] |  | Vehicle surveillance
---
systems in ITS
| Generic layered architecture for video
---
analytics-based vehicle surveillance
| \- Lacking big data and cloud computing
---
techniques and challenges
\- Abstract architecture \- Consider IVA for ITS
Mabrouk et al. [36] |  | Abnormal behavior in
---
surveillance videos
| A generic architecture for abnormal behaviors
---
in video surveillance systems
| -Lacking big data and cloud computing
---
discussion.
-Limited to abnormal behavior
Big data |  Hashem et al. [16] Agrawal et al. [39] |  | Cloud computing
---
| A thorough overview of big data techniques,
---
challenges and open issues in cloud computing
| -Big data processing over cloud computing only.
---
-Lack of video big data analytics,
service-oriented architecture,
research issues, and challenges.
Khan et al. [37] Tsai et al. [38] |  | Big data technologies
---
| Overview of big data techniques,opportunities,
---
and challenges
| Limited discussion on video big data
---
analytics
Zhou et al. [40] |  | Machine Learning (ML)
---
| A framework of ML on big data, and open
---
research issues
| Lack of IVA discussion
---
Che et al. [41] |  | Big data mining
---
| A comprehensive overview on big data
---
mining and challenges
| Lack of video analytics and its
---
research issues
Pouyanfar et al. [42] |  | Multimedia
---
| A comprehensive overview of multimedia data
---
| Limited discussion on video big data analytics
---
Zhu et al. [43] | ITS |  | -A study of big data analytics in ITS
---
-Consider multimedia for ITS
| Limited discussion of video big data
---
in the context of ITS analytics
H. Zahid et al. [44] | Telecommunications | A study of big data analytics in telecommunication. |  | Limited discussion of video big data
---
in the context of telecommunication analytics

Furthermore, big data studies were conducted from different perspectives,
i.e., cloud computing, ML, mining, multimedia, and ITS. Hashem et al. [16] and
Agrawal et al. [39] presented research issues and challenges on big data
analytics in the cloud computing. Tsai et al. [38] and Khan et al. [37] focus
on big techniques and general applications along with challenges. Zhou et al.
[40], Che et al. [41], and Pouyanfar et al. [42] studied and reported the
research issues of big data practices concerning ML techniques, data mining
algorithms, and multimedia data respectively. Zhu et al. [43] presented a
comprehensive study on big data analytics in ITS, analytical methods and
platforms, and categories of big data video analytics. In the study of big
data, IVA has been overlooked, and minimal discussion can be found in the
study of Khan et al. [37], Tsai et al. [38], Pouyanfar et al. [42], Zhu et al.
[43], H. and Zahid et al. [44].

From the literature, it is clear that the recent studies ignored large-scale
unstructured video analytics in the cloud in the discussion of big data. Some
surveys focus on big data management and its related tools, while others have
limited investigation of IVA in a particular context. They also do not
consider the growing unstructured videos as video big data. Unlike existing
work, this paper provides a comprehensive assessment of state-of-the-art
literature and proposes an in-depth distributed cloud computing-based IVA
reference architecture.

##  III Scope and Nomenclature

We clarify some nomenclature being used and scoping this study. The first one
is IVA, which is “any surveillance solution that utilizes video technologies
to automatically manipulate and/or perform actions on live or stored video
images [45]”. The IVA services are implemented through hardware called Video
Analytics System (VAS) [30]. VAS assist acquires videos continuously and
monitors unblinkingly. The VAS falls into four categories, i.e., Embedded
Video Analytics System (EVAS), On-site Video Analytics System (OVAS), Fog-
based Video Analytics System (FVAS), and CVAS.

EVAS embeds IVA solutions and performs video analysis directly on the edge
device, e.g., camera or encoder, and can produce alerts in case of
abnormality. EVAS provides very plain video analytics solutions and can
simultaneously perform two or three rules on its stream and cannot accomplish
complex algorithms such as fire detection, facial recognition, or cross video
stream analytics. Under OVAS, small and middle-sized companies consist of
networked or wireless cameras, a network router, a system running the video
analytics and management software (e.g., IBM smart surveillance system [46],
and Zoneminder [47]), and a storage device. All the cameras send the video
data for analytics against contextual video analytics algorithms and warn the
operator if anomaly detected. OVAS has many limitations, e.g., maintenance,
software up-gradation, expensive hardware, scalability, and unable to deal
with large-scale video data.

When IVA solutions are provided in fog and cloud computing environment, then
it is called FVAS, and CVAS, respectively. In such environments the IVA
solutions are made available under the as-a-Service (aaS) paradigm, i.e.,
IVAaaS. In FVAS, the IVA solutions are geographically distributed and
configured near the edge devices, i.e., video stream sources, to meet the
strict real-time IVA requirements of large-scale video analytics, which must
address latency, bandwidth, and provisioning challenges. Whereas the CVAS is
more suitable for offline IVA because of the relatively high response time and
latency. The hierarchy and relation among CVAS, FVAS, and EVAS is shown in
Fig. III. The batch video data analytics are performed in the cloud, while the
real-time IVA is performed in the fog. The EVAS can play a passive role in the
proposed architecture, e.g., feed the video streams to the CVAS if motion is
detected.

The scope of this paper is FVAS, and CVAS, i.e., the real-time and batch IVA
solutions are deployed in the fog and cloud computing environment,
respectively, while utilizing big data computing technology. However, for the
ease of understandability, throughout this paper, we use the notion of CVAS.

[!t]()[width=0.47]figures/CloudFogEdge Geo-distributed video analytics
infrastructure.

To analyze a video in CVAS, a video undergoes through different phases, as
shown in Fig. 1a. In Fig. 1a, Video Source are the sources which either
generate videos streams from sources connected directly to real-world domains
such as IP-camera or can be already acquired videos in the form of datasets
residing in a file system. If IVA in the cloud are performed on real-time
video streams, then we called it Real-time IVA (RIVA) and Batch IVA (BIVA) if
performed on batch videos.

The Ingest phase implement interfaces to acquire videos from the Video Source.
In the context of IVA, the acquired videos can be represented as a hierarchy,
as shown in Fig. 1b. An acquired video from a Video Source may be decomposed
into its constituent units either in the temporal or spatial domain. In a
video, a frame represents a single image, whereas a shot denotes a consecutive
sequence of frames recorded by a single camera. A scene is semantically
related shots in a sequence that depicts a high-level story. A collection of
scenes composes a sequence/story. Frames and shots are low-level temporal
features suitable for machines, while scenes and sequence/story are considered
to be the high-level features that are suitable for human perceptions. Such
constituent units are further subject to low, mid, or/and high-level
processing. In low-level processing, primitive operations (in Transformations
phase) are performed e.g., noise reduction, histogram equalizer. The Infer
phase encompasses mid and high-level processing. The mid-level processing
extracts features from the sequence of frames, e.g., segmentation,
description, classification, etc. The high-level processing, make sense of an
ensemble of recognized objects; perform the cognitive functions normally
associated with vision. Finally, the extracted information can be persisted to
the data store and/or published to the end-user.

Figure 1: (a) A generic IVA service pipeline. (b) Hierarchal representation of
video units.

Furthermore, the basic unit of an IVA service pipeline is an algorithm, e.g.,
encoder, feature extractor, classifier, etc. The input of an algorithm can be
a Video Source, keyframes, or features. Similarly, the output of an IVA
algorithm can be high, mid, or low-features. Throughout this study, all
possible outputs of IVA algorithms are termed as IR. Multiple algorithms can
be pipelined to build a domain-specific IVA service. The input and output of
an IVA service are restricted to the Video Source and IR, respectively. The
User represents the stakeholder of the CVAS, such as administrator, consumer,
IVA researchers, and practitioners. A Domain is a specific real-word
environment, e.g., street, shop, road traffic, etc., for which an IVA service
needs to be built for automatic monitoring. Domain knowledge facilities IVA in
discovering interesting patterns from domain video streams. The combination of
software and hardware constitutes s distributed System (cloud environment)
where IVA service and algorithms can run fast. Nevertheless, upgrading
existing IVA algorithms to distributed architecture requires customization of
how IVA algorithms should be implemented and deployed. Moreover, the specific
needs of IVA may require the design and development of new system
architecture. All the symbols being used in this paper are listed below.

## Glossary

aaS

    as-a-Service
ACID

    Atomicity, Consistency, Isolation, Durability
AFS

    Adaptive Feature Scaling
AI

    Artificial Intelligence
B2B

    Business-to-Business
B2C

    Business-to-Customer
BIVA

    Batch IVA
C2C

    Customer-to-Customer
CAP

    Consistency, Availability, Partition Tolerance
CBVR

    Context-Based Video Retrieval
CCDG

    Controlled Cyclic Dependency Graph
CNN

    Convolutional Neural Network
CVAS

    Cloud-based Video Analytics System
DAG

    Directed Acyclic Graph
DBDS

    Distributed Big Datastore
DFS

    Distributed File System
DMBM

    Distributed Message Broker Manager
DPDS

    Distributed Persistent Data Store
EVAS

    Embedded Video Analytics System
FVAS

    Fog-based Video Analytics System
FVSA

    Fused Video Surveillance Architecture
HDFS

    Hadoop File System
HOG

    Histogram of Oriented Gradients
IaaS

    Infrastructure-as-a-Service
IoT

    Internet of Things
IR

    Intermediate Results
IRM

    Intermediate Results Manager
ISDDS

    Immediate Structured Distributed Data Store
ITS

    Intelligent Transportation System
IVA

    Intelligent Video Analytics
IVAAaaS

    IVA-Algorithm-as-a-Service
IVAaaS

    IVA-as-a-Service
IVAS

    IVA services
KCL

    Knowledge Curation Layer
LBP

    Local Binary Pattern
LVSM

    Lifelong Video Stream Monitor
ML

    Machine Learning
OVAS

    On-site Video Analytics System
PaaS

    Platform-as-a-Service
PCA

    Principal Component Analysis
QoS

    Quality of Service
RDBMS

    Relational Database Management System
RIVA

    Real-time IVA
RNN

    Recurrent neural network
RVSAS

    Real-time Video Stream Acquisition and Synchronization
SaaS

    Software-as-a-Service
L-CVAS

    Lambda CVAS
SIFT

    Scale Invariant Feature Transform
SVM

    Support Vector Machine
UPDDS

    Unstructured Persistent Distributed Data Store
VAS

    Video Analytics System
VBDCL

    Video Big Data Curation Layer
VBDML

    Video Big Data Mining Layer
VBDPL

    Video Big Data Processing Layer
VSAC

    Video Stream Analytics Consumer
VSAS

    Video Stream Acquisition Service
VSCS

    Video Stream Consumer Service
VSDS

    Video Stream Data Source
WSL

    Web Service Layer

##  IV Lambda CVAS: A Reference Architecture

In this section, we briefly presents the technical details of the proposed
CVAS (called Lambda CVAS (L-CVAS)) and the technical detail of each layer in
the consecutive sub-sections. Fig. IV presents the proposed reference cloud-
based layered architecture for distributed RIVA and BIVA. The proposed L-CVAS
architecture consists of five layers i.e., Video Big Data Curation Layer
(VBDCL), Video Big Data Processing Layer (VBDPL), Video Big Data Mining Layer
(VBDML), Knowledge Curation Layer (KCL), and Web Service Layer (WSL).

VBDCL is the foundation layer and is responsible for large-scale big data
management throughout the life-cycle of IVA, i.e., from data acquisition to
early persistence to archival and deletion [48]. VBDPL is responsible for
distributed video pre-processing, feature extraction, etc. The VBDML deploys
IVA algorithms on the top of distributed processing engines intending to
produce high-level semantics from the processed sequence of frames. On top of
the VBDML layer, the KCL layer has been designed to link the low-level
features in spatial and temporal relation across videos in a multi-stream
environment. KCL deploys a generic video ontology. The KCL layer maps the
extracted IR to the video ontology to bridge the semantic gap between the low-
level features in Euclidean space and temporal relation across videos while
utilizing semantic rich queries. The proposed architecture incorporates top-
notch functionalities of the above four layers into a simple unified role base
WSL, which enables the L-CVAS users to manage, built, and deploy a wide array
of domain-specific near RIVA and BIVA services.

All the layers are made available as aaS. These IVAaaS are provided to the
domain experts and allow them to pipelined in a specific context to built an
IVA service. These IVA services are made available as IVAaaS to which users
can subscribe Video Sources.

Functionalities like security, scalability, load-balancing, fault-tolerance,
and performance are mandatory and common to all the layers, which are shown as
a cross-cutting in Fig. IV. The cloud infrastructure provides the underlying
hardware and software under IaaS, on which the L-CVAS can be deployed. The
cloud infrastructure is out of the scope of this paper. It has already been
studied in detail in the context of big data by [16, 39, 36]. However, in the
discussion of IVA in the cloud, some resources like CPU, GPU, FPGA, HDD, and
SSD can be considered.

[!t]()[width=0.98]figures/IVAaaSArchitecture.pdf Lambda CVAS: A reference
architecture for real-time and batch intelligent video analytics in the cloud.

###  IV-A Video Big Data Curation Layer

Effective data management is key to extract insights from the data. It is a
petascale storage architecture that can be accessed in a highly efficient and
convenient manner. We design the VBDCL for L-CVAS to efficiently manage video
big data. L-CVAS ’s data storage stack consists of three main components:
Real-time Video Stream Acquisition and Synchronization (RVSAS), Distributed
Persistent Data Store (DPDS), and VBDCL Business Logic.

TABLE II: Popular message-oriented middlewares.

|  Kafka [49] |  RabbitMQ [50] |  RocketMQ [1] |  ActiveMQ [51]
---|---|---|---|---
Format | String, Byte | Map, String | String, Byte | String, Map, Byte, Object
Protocol | Kafka |  | AMQP, MQTT,
---
STOMP, XMPP
Rocket, MQ |  | STOMP, REST,
---
XMPP, AMQP
Pub/Sub | ✓ | ✓ | ✓ | ✓
| Point-to-Point
---
- | ✓ | ✓ | ✓
Cluster |  | Stateless cluster
---
| Simple clusters
---
| Stateless cluster
---
| Simple clusters
---
| Load balancing
---
✓ | ✓ | ✓ | ✓
Persistence | ✓ | ✓ | ✓ | ✓
Encryption | SSL | TLS | TLS | TLS
| Authentication
---
SASL | SASL | - | JAAS
APIs |  | Java, .NET, PHP, Go, JS
---
Python, Ruby, Perl, Erlang
| Java, .NET, PHP, Ruby,
---
Python, Go, JavaScript
| Java, C++, Go
---
| Java, PHP, C++, Ruby,
---
Perl, C#, Python, C

####  IV-A1 Real-time Video Stream Acquisition and Synchronization

The real-time video stream needs to be collected from the source device and
forwarded to the executors for on-the-fly processing against the subscribed
IVA service. Handling a tremendous amount of video streams, both processing
and storage are subject to lose [52]. To handle, large-scale video stream
acquisition in real-time, to manage the IR, anomalies, and the communication
among RIVA services, we design the RVSAS component while assuming a
distributed messaging system.

Distributed Message Broker, also known as message-oriented-middleware [53], is
an independent application that is responsible for buffering, queuing,
routing, and delivering the messages to the consumers being received from the
message producer [54]. Message broker should be able to handle permission
control and failure recovery. A message broker generally supports routing
methods like direct worker queue, and/or publish-subscribe [55]. Similarly,
the message consumer component receives the messages from the message broker
either periodically (cron-like consumer) or continuously (daemon-like
consumer). Generally and for the sake of scalability, message consumers are
deployed on separate servers independently of message producers [55]. Some
popular distributed messaging systems are shown in table II.

RVSAS provides client APIs on the top of a distributed messaging system for
the proposed framework. The RVSAS component is responsible for handling and
collecting real-time video streams from device-independent video data sources.
Once the video stream is acquired, then it is sent temporarily to the
distributed broker server. The worker system, on which an IVA service is
configured, e.g., activity recognition, reads the data from the distributed
broker and process. The RVSAS component is composed of five sub-modules, i.e.,
Distributed Message Broker Manager (DMBM), Video Stream Acquisition Service
(VSAS), Video Stream Consumer Service (VSCS), Intermediate Results Manager
(IRM), and Lifelong Video Stream Monitor (LVSM).

##### Distributed Message Broker Manager

DMBM are used to manage the queues in the distributed message broker cluster
considering RIVA services. Three types of queues, RIVA_ID, RIVA_IR_ID, and
RIVA_A_ID as shown in Fig. 2, are automatically generated by the DMBM module
on the distributed message broker when a new RIVA service is created. Here
RIVA, ID, IR, and A stands for RIVA service, unique identifier of the service,
IR, and Anomalies, respectively. These queues are used to hold the actual
video stream being acquired by VSAS, IR produced by an algorithm, and
anomalies detected by the video analytics services.

##### Video Stream Acquisition Service

VSAS module is used to provide interfaces to VSDS and acquires large-scale
streams from device-independent video data sources for on-the-fly processing.
If a particular video stream source is subscribed against an RIVA service,
then the VSAS gets the configuration metadata from the Data Source DS in
Immediate Structured Distributed Data Store (ISDDS) and configure the source
device for video streaming. After successful configuring the source device,
VSAS decodes the video stream, detects the frames, and then performs some
necessary operations on each frame such as meta-data extraction and frame
resizing, which is then converted to a formal message. These messages are then
serialized in the form of mini-batches, compressed, and sent to the
Distributed Broker. If a video-stream source “$C_{1}$” is subscribed to the
RIVA service “$S_{1}$” then the VSAS will rout the mini-batch of the video
stream to queue RIVA_1 in the Broker Cluster as shown in Fig. 2.

Figure 2: Real-time Video Stream Acquisition and Sync.

##### Video Stream Consumer Service

As the acquired video streams are now residing in the distributed broker in
different queues in the form of mini-batches. To process these mini-batches of
the video stream, we have different groups of computer cluster know as Video
Stream Analytics Consumer (VSAC) Cluster. On each cluster, three types of
client APIs are configured, i.e., RIVA services, VSCS, and LVSM. Each VSAC
cluster has different domain-specific RIVA services where the VSCS are common
for all. The VSCS assists the RIVA service to read the mini-batches of the
video stream from the respective queue in the distributed broker for
analytics, as shown in Fig. 2. The VSCS module has two main functions. First,
this module allows RIVA service to read the mini-batches of the video stream
from the respective queue in the distributed broker. The second task is to
save the consumed unstructured video streams and its meta-data to the row
video space in the DPDS and to the Data Source DS meta-store, respectively.

##### Intermediate Results Manager

During the IVA service life-cycle, a sequence of algorithms are executed. Thus
the output of an algorithm can be the input of another algorithm. The IR
demands proper management in the distributed environment because one algorithm
may be on one computer while the other may be on another computer. Thus we
design IRM that sends and gets the IR to and from the topic RIVA_IR_ID in the
distributed broker cluster. Similarly, this module is also responsible for
reading the IR from the respective queue and persists to the IR data store for
future use so that to avoid recomputation.

##### Lifelong Video Stream Monitor

The domain-specific RIVA service process the video stream for anomalies or
abnormal activities. If an anomalies are detected, then it is sent to the
distributed broker queue (i.e., RIVA_A_ID) by using the LVSM instance. To
generate notification base response, LVSM follow standard observer-based
concept [56]. Based on this approach, the LVSM module reads the anomalies from
the respective distributed broker queue, i.e., RIVA_A_ID and notify the
clients in near real-time and simultaneously persisted to the ISDDS.

####  IV-A2 Distributed Persistence Data Store

The second component of the VBDCL is DPDS. The DPDS component provides the
facilities of permanent and distributed big-data persistence storage of both
structured and unstructured data. The DPDS provides abstraction in two levels
on the acquired video data ie. ISDDS and Unstructured Persistent Distributed
Data Store (UPDDS). The philosophy behind DPDS and two levels of abstraction
in the context of L-CVAS is many folds. From the users’ perspective, the CVAS
demands geo-based, real-time, low latency, and random read-write access to the
data in the cloud. Similarly, the DPDS should also provide high-performance
locality-based access to the data when the other layers deploy data-intensive
IVA services. To meet such a diverse amount of requirements of the DPDS
component, technologically, a Distributed File System (DFS) and Distributed
Big Datastore (DBDS) can be leveraged.

##### Immediate Structured Distributed Datastore

TABLE III: Popular Distributed File Systems.

Features |  CEPH [57] |  GlusterFS [58] |  HDFS [59] |  Lustre [60] |  MooseFS [61] |  Quantcast File [62]
---|---|---|---|---|---|---
Initial release | 2007 | 2007 | 2006 | 2003 | 2008 | 2011
License | LGPL 2.1 | GPLv3+ | Apache 2.0 | GPLv2 | GPLv2 | Apache 2.0
Availability | ✓ | ✓ |  | Transparent
---
master failover
✓ | Master | Master
Tolerate disk failures | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Scale | PBs | PBs | PBs | PBs | PBs | PBs
Shards | ✓ | ✓ | ✗ | ✓ | ✗ | ✗
Commodity Hardware | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Language | C++ | C | Java | C | C | C
POSIX Complaint | ✓ | ✓ | ✗ | ✓ | ✓ | ✓
APIs |  | Librados (C,
---
Python, C++,
Ruby), S3,
Swift, FUSE
| NFS, FUSE, SMB,
---
Swift, libgfapi,
libglusterfs
| Java and
---
C client, HTTP
| NFS-Ganesha,
---
NFS, SMB
FUSE |  | C++ client,
---
FUSE

The ISDDS is provided to manage large-scale structured data in the distributed
environment over DBDS. Because of the data-intensive operation and according
to the requirements of the other layer, technologically, a distributed big
data store, can be deployed. The ISDDS hosts five types of data. The detailed
description of each type of data has been described in this section.

L-CVAS provides role-based access to its user. L-CVAS user logs and the
respective role information are maintained through the User Profile and Logs
meta-store. The proposed framework manages two types of video data sources
through the Data Source meta-store. These are video data sources, for example,
IP-cameras, Kinect, body-worn cameras, etc., and batch video datasets. The
former one can be subscribed to RIVA service while the later one is eligible
for BIVA services. The meta-information of these sources, along with access
rights, are managed through the Data Source meta-store. Administrator and
developer roles can develop, create, and deploy video analytics algorithms
through the L-CVAS. Similarly, different IVA algorithms can be pipelined into
an IVA service. The management of video analytics algorithms and services is
managed through Video Analytics Algorithm and Service meta-store,
respectively. As stated that in IVA pipelining environment, the output of one
IVA algorithm can be the output of another algorithm. In this context, we
design a general container called IR datastore to persist and index the output
of an RIVA algorithm, and services. This datastore is significant and can play
a vital role in IVA pipeline optimization, and fast content-based searching
and retrieval. Finally, the L-CVAS users are allowed to subscribe to the data
sources to the IVA services. The subscription information is maintained
through the Subscription meta-store, and the anomalies are maintained through
the Anomalies meta-store.

The ISDDS Data Model of the L-CVAS demands an efficient distributed data
store. The distributed data store should have the ability of horizontal
scalability, high availability, partition tolerance, consistency, and
durability. Furthermore, the data store should fulfill the read/write access
demands of the BIVA operations, and RIVA, interaction, and visualization. It
is a fact that traditional relational databases have little or no ability to
scale-out to accommodate the growing demands of the big data, and resultantly
new distributed data stores have emerged. The distributed data stores can be
grouped into two major categories, i.e., NoSQL (Not Only SQL) and NewSQL
(excluding graph data stores).

NoSQL is a schema-free data store designed to support massive data storage
across distributed servers [63, 64]. The features of NoSQL data stores include
horizontal scalability, data replication, distributed indexing, simple API,
flexibility, and consistency [65]. NoSQL lacks true Atomicity, Consistency,
Isolation, Durability (ACID) transactions, unlike Relational Database
Management System (RDBMS). In the context of Consistency, Availability,
Partition Tolerance (CAP) theorem [66], it has to compromise on either
consistency or availability while choosing partition tolerance. NoSQL can
further be categorized as Document, Key-value, and Extensible stores. A key-
value data store is responsible for storing values and indexes for searching.
Document datastore is used for document storage, indexing, and retrieval.
Extensible data store stores extensible records that can be partitioned
vertically and horizontally across the nodes.

NewSQL data stores provide the characteristics of both NoSQL and RDBMS: ACID
transactional consistency of relational databases with facilities of SQL; and
the scalability and performance of NoSQL. MySQL Cluster, VoldDB, ClustrixDB
are examples of NewSQL. Some of the popular NewSQL and NoSQL datastores are
shown in Table IV along with the respective properties. The ‘A’ and ‘C’ in CAP
is not equal to the ‘A’ and ‘C’ in ACID [67].

TABLE IV: Popular Distributed Data Stores.

Type | DS Name | Release | Developer | OS | Language | D | HS | SL | Properties
---|---|---|---|---|---|---|---|---|---
Key-value | Voldemort | 2009 | LinkedIn | ✓ | Java | ✓ | ✓ | ✓ | CAP
Riak | 2009 |  | Basho
---
Tech.
✓ | Erlang | ✓ | ✓ | ✓ | CAP
Redis [68] | 2009 |  | Salvatore
---
Sanfilippo
✓ | C | ✓ | ✓ | ✓ |  CAP
Document | SimpleDB | 2007 | Amazon | ✗ | Erlang | ✓ | ✓ | ✓ | CAP
CouchDB | 2005 | ASF | ✓ | Erlang | ✓ | ✓ | ✓ | CAP
MongoDB [69] | 2009 | 10gen | ✓ | C, C++ | ✓ | ✓ | ✓ |  CAP
DynamoDB | 2012 | Amazon | ✗ | Java | ✓ | ✓ | ✓ | CAP
Extensible | HBase [70] | 2010 | ASF | ✓ | Java | ✓ | ✓ | ✓ |  CAP
HyperTable | 2008 | Zvents | ✓ | C++ | ✓ | ✓ | ✓ |  CAP
Cassandra | 2008 | ASF | ✓ | Java | ✓ | ✓ | ✓ | CAP
BigTable [71] | 2005 | Google | ✗ | C, C++ | ✓ | ✓ | ✓ |  CAP
NewSQL | MySQL Cluster | 2004 | Oracle | ✓ | C++ | ✓ | ✓ | ✗ | ACID
VoldDB [72] | 2009 | VoltDB Inc | ✓ | Java, C++ | ✓ | ✓ | ✗ | ACID
ClustrixDB | 2007 |  | Clustrix,
---
Inc.
✗ | C | ✓ | ✓ | ✗ | ACID
NuoDB | 2008 | NuoDB | ✗ | C++ | ✓ | ✓ | ✗ | ACID
| P=Persistence, D=Durability, HS=High Scalability, SL=Schema-less
---
CAP=Consistency, Availability, Partition Tolerance
ACID= Atomicity, Consistency, Isolation, Durability
In the properties column, the underline character represents the supported
properties in the CAP theorem.

##### Unstructured Persistent Distributed Datastore

The UPDDS component built on the top of the DFS such as Hadoop File System
(HDFS) that facilitates permanent and distributed big-data storage. The data
are stored and mapped systematically according to the business logic of the
proposed system. The UPDDS component is designed to effectively and
dynamically manage the data workload in the life-cycle of an IVA service. Upon
new registration with the L-CVAS, a formal User Space is created on the top of
DFS. The User Space is managed through the proper hierarchical directory
structure and special read and writes access are granted to the owner. All the
directories are synchronized and mapped in the corresponding user profile
logs. Under the User Space, three types of distributed directories are
created, i.e., Raw Video Space, Model Space, and Project Space. The
hierarchical structure of the User Space in DFS is shown in Fig. 3.

Figure 3: Hierarchical structure of the User Space in DFS.

Raw Video Space is used for the management of the video data. Raw Video Space
is further divided into two types of video spaces. The first type is a batch
video which has been uploaded to the L-CVAS for batch analytics, where the
second type is acquired and persisted from the VSDS. The entire acquired
stream is time-stamped on persistence. The granularity level of raw streaming
videos is maintained through video data sources. IVA life-cycle may need
different models for training and testing purposes. The Model Space is
provided to facilitate the developers to manage the training and testing model
according to the deployed IVA algorithm. Similarly, a developer can develop a
new algorithm or a video analytics service. The Project Space is provided to
manage the source code of the respective developer and practitioners.

The UPDDS is supposed to be designed on the top of DFS. In this context,
diverse types of open source DFS have been developed. The implementation
quality of big data applications is relative to the storage tier’s file
system. From an architectural perspective, working and handling large volumes
and throughput of data is challenging. Commonly, big data solutions exploit a
cluster of computers, ranging from few to hundreds of computers, connected
through high-speed networks while deploying specialized distributed data and
management software. In distributed data-intensive applications, large-scale
data is always moving across the cluster and thus demands a distributed,
scalable, reliable, and fault-tolerant file system [73] known as DFS. DFS is a
file system that provides access to replicated files across multiple hosts on
a computer network, ensuring performance, data locality, high availability,
scalability, reliability, security, uniform access, and fault-tolerance.
Currently, various DFS are available and may differ in terms of performance,
fault-tolerance, content mutability, and read/write policy. Some state-of-the-
art popular DFS, which can be used for UPDDS, along with a short description,
are shown in Fig III.

##### Active and Passive Data Readers and Writer

This module gives read-write access to the underlying data securely according
to the business logic of the VBDCL Business Logic and according to the
registered user access rights. This sub-module is composed of four types of
readers and writers, i.e., ISDDS Active Data Reader, ISDDS Passive Data
Reader, UPDDS Active Data Reader, and UPDDS Passive Writer. For real-time,
read-write operation over the data residing in the ISDDS and UPDDS, such as
CRUD operation, file creation, video stream writing to DFS, etc. the active
data reader and writer are used. In the context of offline analytics over the
bulk of videos while using distributed processing engines. The Passive Data
Reader and Writer (PDRW) is provided to allow processing engines to load the
bulk of data and persist the same to the ISDDS and UPDDS.

####  IV-A3 VBDCL Business Logic

VBDCL Business Logic provides the actual business logic. It implements six
different modules, i.e., User Manager, Data Source Manager, Model Manager,
(R/B)IVA Algorithm and Service Manager, (R/B)IVA Service Discovery and
Subscription Manager, Video Ontology Manager, Cluster Management and
Monitoring.

The User Manager module encapsulates all the user-related operations such as
new user account creation, access role assignment, and session management.
Through the Data Source Manager Model Manager modules, the user can manage the
VSDS, video data uploading, and model management. The (R/B)IVA Algorithm and
Service Manager are built to manage, develop, and deploy new IVA algorithms
and services, respectively. The former one is provided aaS to the L-CVAS
developers, while the latter one is provided aaS to the consumers. The
developer role can create and publish a new video analytics algorithm. The
algorithm is then made available aaS to other developers and can use it. Once
IVA services are created, then the L-CVAS users are allowed to subscribe to
the streaming video data sources and batch data against the provided RIVA and
BIVA services, respectively.

Similarly, the Ontology Manager allows the developer to get the IR for
decision making. The ontology manager provides a secure way of getting the IR
and maps it according to ontology. This module also allows the user to manage
the functionalities of the KCL. Finally, Cluster Management and Monitoring
allows the administrator to monitor the health of the cluster.

###  IV-B Distributed Intelligent Video Analytics

IVA performs the complex tasks of extracting significant knowledge and
information of interest from the video data, i.e., structural patterns,
behavior patterns, content characteristics, event patterns [74, 75] and their
relationship in the form of classification and clustering. Extracting
knowledge and information from video big data is a processing-intensive task.
Value extraction from video big data is behind the capabilities of traditional
tools and demands for technological solutions to meet the processing
requirements.

Therefore, video big data analytics is preferably performed in the distributed
environment in a parallel manner while utilizing distributed scale-out
computing technologies [76], such as MapReduce and Apache Spark. Distributed
IVA not only significantly improves the performance, but also reduces the
analytics cost. Based on the generic IVA life-cycle and motivated by scikit-
learn [77], the distributed IVA are divided into two layers, i.e., VBDPL, and
VBDML. These two layers are further elaborated in the following sub-sections.

####  IV-B1 Video Big Data Processing Layer

IVA requires video data pruning and strong feature extraction. With such
intentions, the VBDPL layer consists of three components, i.e., Video
Preprocessing, Feature Extractor, and Dimensionality Reduction is designed.

##### Video Preprocessing

The quality of data plays an active and significant role in solving a problem
with ML. Raw videos have an unstructured format and contain
noise/uncertainties, making it unsuitable for knowledge and information
mining. Video Preprocessing component is designed with the same objectives and
is supposed to deploy several distributed video preprocessing operations
including frame extraction [78], frame-resizing, frame-conversion from RGB to
grayscale [79], shot boundary detection [80], segmentation [81], transcoding
[82], and many more. In the first step, frames are extracted from a video for
processing. Several frame selection algorithms are available, for example,
keyframe extraction. The number of frames to be extracted is dependent on the
user objective and task-dependent. Candidate frames can be all frames, step
frames (every second frame, fifth frame, etc.) or keyframes [83]. The spatial
operations highly depend on the scenario and objective. Spatial operations
include frame resizing (for reducing computational complexity), corrections
(brightness, contrast, histogram equalization, cropping, keyframes), mode
(RGB, Grayscale, etc.), and many other operations. Segmentation is used for
various purposes, such as partitioning video into semantically related chunks.

##### Feature Extractor

The Feature Extractor component implements distributed feature extraction
algorithms. The performance of ML is highly dependent upon the type of data
representation or features [84]. Features represent the characteristics of
classes in the dataset and have a heavy impact on the ML algorithm’s
generalizability and performance. The data features that used to train ML
models have a huge influence on the performance of the algorithm.
Inappropriate or irrelevant features affect the performance of the algorithm.
Thus, feature extraction extracts the features from the raw videos that can be
interpreted by the ML algorithm [85, 86]. In this context, several feature
extraction algorithms have been introduced for video data. These feature
extraction approaches can be categorized into static features of keyframes
[87, 88, 89], object features [90, 91], dynamic/motion feature extraction [85,
92, 93], trajectory-based features extraction [85, 86, 94, 95], and deep
learning-based feature extraction [96, 97, 98, 99, 100, 101, 102, 103, 104].

##### Feature Selection and Dimensionality Reduction

Feature selection and dimensionality reduction reduce the size of the
features. Large sizes of feature sets are expensive in terms of time for
training and/or performing classification acquired by trained classifiers. For
example, Principal Component Analysis (PCA) and its variants are used to
reduce the size of features. During feature selection, most relevant features
are selected by discarding irrelevant and weak features. The performance of ML
classifiers is also directly related to the quality of features; GIGO (garbage
in garbage out). Inappropriate or partially relevant features can negatively
affect model performance. Therefore, only a limited set of features should be
selected and used for training classifiers. This is what precisely the purpose
of this component is and deploy different algorithms in this context.
Similarly, some feature reduction techniques available that selects the
specific set of limited features in real-time. For example, Online Feature
Selection selects and inputs a specific number of small features to the
classifiers in real-time [105]. In order to accelerate the training process,
[106] used non-linear and group-based feature selection techniques, based on
Adaptive Feature Scaling (AFS), to process the data with substantial dimension
sizes. [107] proposed an unsupervised feature reduction technique that selects
extremely relevant features and indicates suitable weights to the distinctive
feature dimensions.

####  IV-B2 Video Big Data Mining Layer

The VBDML utilizes diverse types of machine-learning algorithms, i.e.,
supervised, semi-supervised, and unsupervised algorithms to find different
type of information from the videos [74, 75]. In this context, VBDML layer
hosts three types of components, i.e., Classification, Regression, Clustering.

Classification component provides various ML algorithms, e.g., Support Vector
Machine (SVM), Nearest Neighbors, Random Forest, Decision Tree, Naïve Bayes,
etc., that identifies that a particular object in a video frame belongs to
which category while using predefined classes. The Regression component
includes different algorithms, e.g., Linear Regression, Decision Tree
Regression, Logistic Regression, and many more, predicting a continuous-valued
attribute associated with objects rather than discrete values. The Clustering
component encapsulates algorithms, e.g., K-Mean, spectral clustering, etc.,
that produces groups of data depending upon the similarity of data items.

L-CVAS has the ability of BIVA and RIVA. In this context, the VBDML should
support batch learning and online learning. The former case considers the
complete training data to learn and generates models. The batch-learning
algorithm is expected to generalize, that usually does not perform well in the
real environment. Unlike batch learning, online learning continuously learns
from new input without making any statistical assumptions about the data
[108]. In the context of model generalization, online learning is expected to
work well by accurately predicting the predefined set of inputs [108]. Online
learning is used in the environment when continuous learning from the data is
required to learn new patterns instead of batch learning.

TABLE V: Comparision of popular deep learning architectures.

Architecture | Year | Error | Parameters | Depth | Description and Pros | Cons
---|---|---|---|---|---|---
LeNet | 1998 | - | 60 K | 5 |  | – First popular CNN architecture
---
-Utilize spatial correlation to decrease the
parameters and computation
-Automatic learning of feature hierarchies
| -Poor scaling to diverse classes
---
of images
-Large size filters
-Extraction of low-level features
AlexNet | 2012 | 15.3% | 60 M | 8 |  | -Low, mid and high-level feature extraction
---
using large and small size filters on initial
(5x5 and 11x11) and last layers (3x3)
-The notion of deep and wide CNN architecture
-Introduced regularization in CNN
-Initiated GPUs parallelization
\- Deeper and wider than the LeNet
\- Uses Relu ,dropout and overlap Pooling
\- GPUs NVIDIA GTX 580
| -Inactive neurons in the first and
---
second layers
-Aliasing artifacts in the learned
feature-maps due to large filter size
ZFNet | 2013 | 14.8% | - | 8 |  | -The notion of parameter tuning by visualizing
---
the output of intermediate layers
-Reduced both the filter size and stride in the
first two layers of AlexNet
| -Extra information processing is
---
required for visualization
GoogleNet | 2014 | 6.67% | 4 M | 22 |  | \- Introduces block concept
---
-The notion of utilizing Multiscale Filters within
the layers
-The notion of a split, transform and merge
-Parameters reduction by utilizing the bottleneck
layer, global average pooling at the last layer and
Sparse Connections
-Utilization of auxiliary classifiers to enhance the
convergence rate
| -Heterogeneous topology led to
---
tedious parameter customization
-Chances of losing the
valuable information because of
representational bottleneck
VGGNet | 2014 | 7.3% | 138 M | 19 |  | \- Small kernel size
---
-The notion of effective receptive field
-The notion of simple and homogenous topology
| -Use of computationally expensive
---
fully connected layers
ResNet | 2015 | 3.6% | - | 152 |  | -The error rate was decreased for deeper networks
---
-Introduced the notion of residual learning
-Improves the effect of vanishing gradient issue
\- Identity mapping based skip connection
| -A bit complex architecture
---
-Lowers information of feature-map
in feed forwarding
-Over adaption of hyperparameters
for a specific task, due to the
stacking of the same modules

####  IV-B3 Distributed Deep Learning for IVA

Handcrafted features, e.g., Scale Invariant Feature Transform (SIFT) [109],
Local Binary Pattern (LBP) [110], Histogram of Oriented Gradients (HOG) [111],
etc., generates high dimensional features vectors and resultantly facing the
issue of scalability. Recently, Convolutional Neural Network (CNN) based
approaches have shown performance superiority in tasks like optical character
recognition [112], and object detection [113]. The motive of the deep learning
is to scale the training in three dimensions, i.e., size and complexity of the
models [114], proportionality of the accuracy to the amount of training data
[115], and the hardware infrastructure scalability [116]. Results of deep
learning are so promising that soon the deep learning will give equivalent or
higher performance compared to humans when trained over large data sets [113].
A CNN or ConvNet is a type of neural network that can recognize visual
patterns directly from the pixels of images with less preprocessing. CNN based
video classification methods have been proposed in the literature to learn
features from raw pixels from both short video and still images [98, 97, 117,
118]. In the proposed L-CVAS framework, both the VBDPL, and VBDML are capable
to deploy deep-learning approaches for distributed IVA.

Since on the dawn of deep learning, various open-source architecture have been
developed. Some of the well-known and state-of-the-art CNN architectures are
LeNet-5 [119], AlexNet [118], ZFNet [120], GoogleNet [121], VGGNet [122], and
ResNet [123]. The comparison of these architectures can be found in Table V.
Similarly, several frameworks have been developed to eliminate the need for a
manual definition of gradient propagation. Table VI summarizes these libraries
along with the comparisons.

TensorFlow is a popular deep learning library designed for ML and deep
learning. It supports the deployment of computation on both CPUs and GPUs.
TensorFlow allows the fast implementation of deep neural networks on the
cloud. TensorFlow is also suitable for other data-driven research purposes and
is equipped with TensorBoard (a visualization tool). Higher-level programming
interfaces such as Luminoth, Kera, and TensorLayer were built on the top of
TensorFlow. Caffe2, developed by Berkeley AI Research, is another library to
build their deep learning models efficiently along with GPUs’ support in a
distributed environment. PyTorch, maintained by Facebook, is a scientific
computing framework with wide support for machine learning models and
algorithms. PyTorch offers rich pre-trained models that can be easily reused.
MXNET is a deep learning library suitable for fast numerical computation for
both single and distributed ecosystems. Likewise, some more deep learning
libraries have been developed, such as CNTK, Deeplearning4j, Blocks, Gluon,
and Lasagne, which can also be employed cloud environment.

TABLE VI: Summary of popular deep learning libraries.

Features | TensorFlow [124] | Caffe2 [125] | PyTorch [126] | MXNet [127] | CNTK [128] | Matlab DL Toolbox [129]
---|---|---|---|---|---|---
Release | 2015 | 2013 | 2016 | 2015 | 2016 | 2017
Licence | Apache 2.0 | BSD | BSD | Apache 2.0 | MIT | Proprietary
Written in | C++, Python, CUDA | C++ | Python, C, C++, CUDA | C++ | C++ | C, C++, Java, MATLAB
Interface |  | Python (Keras), C/C++,
---
Java, Go, JavaScript,
R, Julia, Swift
| Python,
---
MATLAB, C++
Python, C++ |  | C++, Python, Julia,
---
Matlab, JavaScript,
R Go, Scala, Perl, Clojure
| Python, C++,
---
C#, and Java.
MATLAB
GPU | ✓ | ✓ | ✓ | ✗ | ✓ | ✓
DNN Support | RNN, CNN, LSTM, etc. | CNN, RNN, LSTM | RNN, CNN, LSTM, etc. | RNN, CNN, LSTM, etc. | CNN, RNN | ConvNets, CNN, LSTM, etc.
| Top-Level
---
Libraries
TensorLayer, Keras, | - | - | Gluon | Keras | -
Pros |  | Tensorboard for
---
visualization, well
documented, large
user community
| Fast, scalable,
---
lightweight,
cross-platform
support, server
optimized interface
| Easy to use and debug,
---
flexible, well documented,
declarative data parallelism
| Lightweight and fast,
---
memory-efficient, highly
scalable
| Good performance
---
and scalability,
highly optimized,
support for Spark
| easy for scientists,
---
model visualization,
high-performance
CUDA code generation,
Cons |  | Debugging issues, heavy,
---
difficult for novice
| Small community,
---
modest documentation
| Lacking model serving,
---
visualization tools
| Small community,
---
difficult to learn
Limited community |  | Closed-source,
---
expensive in terms of
execution and price

Big DL models training with large-scale training data is a challenging task.
For example, S Gao at al. [130] utilized six learning algorithms, i.e.,
biogeography-based optimization, particle swarm optimization, genetic
algorithm, ant colony optimization, evolutionary strategy, and population-
based incremental learning, for the best combination of neural network user-
defined parameters during training. It is a hectic job for a single system
when the training datasets are large. Distributed infrastructure with multiple
computing nodes (equipped with powerful GPUs) is the best option, but it leads
to numerous challenges. First is the effective utilization of resources
(costly GPU stalling preventions). Second, the resources are shared among
different users in the cloud for cost reduction and elasticity. Such
challenges are attracting the attention of researchers. There are three
approaches, i.e., model, data, and pipeline distribution for leveraging
distributed computing in distributed computing [131, 132]. In the former, the
DL model is partitioned in logical fragments, and loaded to different worker
agents for training, as shown in Fig. 4.

Figure 4: Scalable deep learning utilizing model distribution.

In the second approach, the deep learning model is replicated to all the
cluster’s worker-agents, as shown in Fig. 5. The training dataset is
partitioned into non-overlapping sub-dataset, and each sub-dataset are loaded
to the different worker-agents of the cluster. Each worker-agent executes the
training on its sub-dataset of training data. The model parameters are
synchronized among the cluster worker-agents to updates the model parameters.
The data distribution approached naturally fits in the distribute computing
MapReduce paradigm [132]. The MapReduce splits the input based on some
predefined parameters. The map tasks, then, process these chunks in parallel a
manner. After processing, the output is shuffled for relevance and is directed
to map tasks for generating intermediate results. The output from the map
tasks is shuffled for relevance and is given as input to the reduce tasks for
generating intermediate results. The intermediate results are combined to
produce the complete result. Hadoop and Spark require and process data
naturally distributed in the manner and popular research trend nowadays.

Figure 5: Scalable deep learning utilizing data distribution.

In the third case, the DL model is partitioned, and each worker-agent loads a
different segment of the DL model for training, as shown in Fig. 6. The
training data are given to the worker-agents that carry the input layer of the
DL model. In the forward pass, the output signal is computed, which is
transmitted to the worker-agents that hold the next layer of the DL model. In
the backpropagation pass, gradients are calculated starting at the workers
that carry the DL model’s output layer, propagating to the workers that hold
the input layers of the DL model [133].

Figure 6: Deep learning with pipelining (Figure motivated by [133]). TABLE
VII: Comparison of big data engines.

Features |  Hadoop [134] |  Spark [135] |  Flink [136] |  Storm [137] |  Samza [138]
---|---|---|---|---|---
Batch | ✓ | ✓ | ✓ | ✗ | ✗
Stream | ✗ | near real-time | Real-time | Real-time event based | Event based
Data Flow |  | No loops, Chain of stages
---
| Direct Acyclic Graph
---
| Controlled Cyclic
---
Dependency Graph
| Direct Acyclic Graph
---
| Kafka Dependent
---
| Fault-
---
tolerance
| High and application
---
restart is not required
in case of failure.
| Recovers lost work
---
in case of failure.
| Based on distributed
---
snapshots
| Automatically restart dead
---
workers or migrate in case
of node failure.
| Migrates the task into
---
another node in case of
node failure.
Scalability | ✓ | ✓ | ✓ | ✓ | ✓
Latency | Higher | Low | Low | Low | Low
Cost | Less expensive | Expensive | Expensive | Expensive | Expensive
Security | Kerberos authentication |  | Shared secret, HDFS
---
ACLS and Kerberos
| Kerberos authentication,
---
SSL
Kerberos authentication | Kerberos authentication
Scheduler |  | By default Fair Scheduler,
---
Capacity Scheduler,
and is pluggable.
| Acts as its own flow
---
scheduler because of
in-memory computation.
| Own scheduler and can
---
use YARN scheduler.
| Default, isolation,
---
multi-tenant and resource
aware scheduler.
Use YARN scheduler.
| Language
---
Support
| Java, C/C++, Ruby,
---
Groovy, Perl and Python
| Java, Scala, Python,
---
and R.
| Java, Scala,
---
Python, and R.
| Java, Clojure, and
---
can support non-JVM
languages.
| Java, scala and can
---
support JVM languages.
Windowing | ✗ | Time-based | Time-based, Count based | Time-based, Count based | Time-based
|  ML Library
---
| Mahout [139], DL4J [140],
---
Keras [141], Caffe [125],
TensorFlow [124],
BigDL [142]
| MLLib (Native) [143],
---
DL4J, Mahout, DL4J,
TensorFlow, Keras, Caffe,
H20, PyTorch [126],
BigDL
| FlinkML (Native),
---
Mahout, TensorFlow,
Keras, Caffe, H20
| Don’t have native
---
ML Lib, SAMOA 3rd
Party [144]
| Don’t have native.
---
SAMOA 3rd Party

####  IV-B4 Big Data Engines, ML Libraries, and IVA

The VBDPL, and VBDML are assumed to be built on the top of distributed
computing engines. This section overview some latest big data engines that can
be utilized for scale-out IVA.

Hadoop MapReduce [145] is a distributed programming model, developed based on
GFS [73], for data-intensive tasks. Apache Spark follows a similar programming
model like MapReduce but extends it with Resilient Distributed Datasets
(RDDs), data sharing abstraction [135]. Hadoop’s MapReduce operations are
heavily dependent on the hard disk while Spark is based on in-memory
computation, which makes Spark a hundred times faster than Hadoop [135, 146].
Spark support interactive operations, Directed Acyclic Graph (DAG) processing,
and process streaming data in the form of mini-batches in near real-time
[147]. Apache Spark is batch centric and treats stream processing as a special
case, lacking support for cyclic operations, memory management, and windows
operators. Such issues of Spark has been elegantly addressed by Apache Flink
[136]. Apache Flink treats batch processing as a special and does not use
micro-batching. Similarly, Apache Storm and Samza is another prominent
solution focused on working with large data flow in real-time. A brief
description and comparison of the open-source big data frameworks are listed
in Table VII.

To achieve scalability, big data techniques can be exploited by existing video
analytics modules. The VBDPL is not provided by default and needs its
implementation on the top of these big data engines. However, The ML
approaches can be categorized into two classes in the context of VBDML. One
class re-implements the existing ML task by providing a middleware layer to
run them on a big data platform. This general type of middleware layer
provides general primitives/operations that assists in various learning tasks.
Users who want to try different ML algorithms in the same framework can take
benefits from it. In the second class, the individual video analytics and ML
algorithm are executed on a big data platform that is directly built on top of
a big data engine for better scalability.

Spark MLlib [143], Mahout [139], FlinKML [136] are list of some open-source ML
packages built on the top of Hadoop, Spark and Flink, respectively, that
support many scalable learning algorithms. For deep learning, various open-
source libraries have been develop including TensorFlow [124], DeepLearning4J
[140], Keras [141], Caffe [125], H20 [148], BigDL [142], and PyTorch [126].
All these libraries provide support for various ML algorithms and feature
engineering. These libraries introduce an independent layer between front-end
algorithms and a back-end engine to facilitate the migration from one big data
engine to another, as shown in Table VII. These algorithms can be used to
process large datasets, just like processing it on a single machine by
providing the distributed environment abstraction and optimization.

TABLE VIII: Popular computer vision benchmark datasets.

Dataset | Instances | Classes | Ground Truth | Description
---|---|---|---|---
YFCC100M [149] | 100M | 8M | Partially | Video and image, understanding
YouTube-8M [150] | 8M | 4,716 | Automatic | Video classification
Trecvid 2016 [151] | Different | Different | Partially | Human action detection
UCF-101 [152] | 13K | 101 | Yes | Human action detection
Kinetics [153] | 306K | 400 | Yes | Human action detection
MediaEval [154] | - | - | - | Videos from BBC
YACVID [155] | 75207 Frames | - | Yes | A multi-camera high-resolution dataset
ActivityNet [156] | 849 video hours | 203 | Yes | Human activity
HMDB [157] | 7K Videos | 51 Classes | Yes | Human Motion Recognition
Sports-1M [98] | IM | 487 | Yes | The YouTube Sports-1M Dataset
AVA [158] | 57K | Different | Yes |  | Spatiotemporally Localized Atomic Visual Actions
---
HowTo100M [159] | 100M | 23k | - | Narrated instructional web videos

####  IV-B5 Computer vision benchmark datasets

In the advancement of IVA, public datasets always play a vital and active
role. Over the year, server benchmark datasets have emerged. Some of the
recent popular datasets are listed in Table VIII. ImageNet [149] is one of the
significant datasets in deep learning and is utilized for training neural
networks such as ResNet, AlexNet, and GoogleNet. Some more datasets have been
developed aiming human action and motion recognition, including [151, 151,
153, 156, 157]. Google released YouTube-8M [160] and consisting of eight
million diverse types of automatically labeled videos. Deng A et al. [159]
proposed the HowTo100M dataset comprising of web videos with narrated
instructions. Dataset like MediaEval2015 [154], and Trecvid2016 [151] are
designed to support CBVR related research. For sports IVA, the Sports-1M
dataset karpathy2014large is proposed and composed of 487 classes along with
ground truth. YACVID [155] is a labeled image sequence dataset for
benchmarking video surveillance algorithms. All these benchmark datasets are
used for different IVA, such as action recognition, event detection, and
classification.

###  IV-C Knowledge Curation Layer

Videos low-level processing produces feature descriptors that summarize
characteristics of data quantitatively. The high-level analytics is more
associated with the visual data understanding and reasoning. The features
descriptors work as input to the high-level analytics and generate abstract
descriptions about contents. The difficult problem is to bridge the semantic
gap between the low-level features and high-level concepts suitable for human
perception [161].

With the same indentations, the KCL layer has been proposed under L-CVAS
architecture, on the top of VBDML, which map the IR (both online and offline)
into the video ontology in order to allow domain-specific semantic video and
complex event analysis. The KCL is composed of five components, i.e., Video
Ontology Vocabulary, Video Ontology, Semantic Web Rules, FeatureOnto Mapper,
and SPARQL queries.

Video Ontology Vocabulary standardizes the basic terminology that governs the
video ontology, such as concept, attributes, objects, relations, video
temporal relation, video spatial relation, and events. Video Ontology is a
generic semantic-based model for the representation and organization of video
resources that allow the L-CVAS users for contextual complex, event analysis,
reasoning, search, and retrieval. Semantic Web Rules express domain-specific
rules and logic for reasoning. When videos are classified and tagged by the
VBDML then the respective IR are persistent to VBDCL and also mapped to the
Video Ontology while using the FeatureOnto Mapper. Finally, SPARQL based
semantic rich queries are allowed for knowledge graph, complex event
reasoning, analysis, and retrieval.

###  IV-D Web Services Layer

Finally, to provide the functionality of the proposed L-CVAS over the web, it
incorporates top-notch functionality into simple unified role-based web
services. The Web Service Layer is built on the top of VBDCL Business Logic.
Sequence diagrams for IVA algorithm and service creation is shown in Fig. 7.
Whereas, role-based use case diagram of the proposed platform is shown in Fig.
7.

Figure 7: Lambda CVAS user roles and use case diagram.

[!t]()[width=0.46]figures/AlgorithmAndServiceSequenceDiagram The sequence
diagram for the IVA algorithm and service creation. In the Actor lifeline, A
and D represent Admin and Developer, respectively. These two roles are allowed
to create a new video analytics algorithm and service.

###  IV-E IVA service example scenarios

We show two example scenarios, i.e., how to develop BIVA and RIVA services
under L-CVAS. Hadoop and Spark MapReduce type operations naturally fit in a
BIVA. Fig. 8, shows the block diagram along with sample script for distributed
BIVA, i.e., object classification, where a DFS is configured to read and store
video files, e.g., in a standard video file format such as MPEG, AVI, H.264,
etc. First, the videos are loaded, the distributed video transcoding (a
preprocessing algorithm under VBDPL) are performed. For example, a user
uploads a MPEG or other video files to the DFS. The transcoder algorithm first
split the file into image frames, which may be throttled to key-value frames,
and converts them into a sequence file format. These frames are then mapped,
and features are extracted (utilizing some feature extractor from VBDPL). The
features are then classified (using a classification algorithm of VBDPL). For
each detected object, a key-value and coordinates of where the object is
located within the image are computed. For each detected object in each frame,
the map element process the frame. The map operation generates and provides
output as composite visual value pair that includes the visual key, a time-
stamp that identifies the frame, and the coordinates. The map stages then send
the output in the form of a composite key-value pair to the respective reduce
stages. The reduce stages provide the output to an output stage, which is
persisted in the IR-DS of ISDDS.

Figure 8: BIVA service example pipeline under Lambda CVAS.

The flow for a single RIVA along with an example pseudo-code is shown in Fig.
9. The VSAS component sends the video stream to the queue in the broker
cluster. Then the consumer service (VSCS) is used to extract the mini-batch of
video streams from the queue and process it. Once the mini-batch is consumed,
then it is transcoded, features are extracted for classification, and finally,
the classification results are persisted to the required destination (IR-DS
and/or dashboard).

Figure 9: RIVA service example pipeline under Lambda CVAS.

###  IV-F Execution Scenarios

L-CVAS follows the lambda architecture style [29], and the execution scenarios
undergo through two types of execution scenarios, i.e., Streaming Execution
Scenario, and Batch Execution Scenario. These two scenarios aim to execute a
massive amount of real-time video stream and batch videos against the
subscribed IVA services. In literature, these execution scenarios are referred
to as Speed Layer, and Batch Layer respectively [29]. The data of both the
scenarios are managed through a common layer called Serving Layer. L-CVAS
components are deployed on various types of clusters in the cloud, and each
cluster is subject to scale-out on-demand. Fig. 10 illustrates these execution
scenarios, and the explanation is given in the following subsections.

Figure 10: Lambda CVAS: Streaming and Batch execution scenarios.

####  IV-F1 Streaming Execution Scenario

L-CVAS is supposed to deploy a pool of contextual RIVA services that are made
available to the user for the subscription. Once a video stream data source is
subscribed to service in the pool of contextual RIVA services, then the life-
cycle of Streaming Execution Scenario encompasses through different stages
while using distinct L-CVAS components. For the ease of understandability,
these components are deployed on six types of computing clusters in the cloud,
which are labeled explicitly as ‘P’, ‘V’, ‘K’, ‘S’, ‘N’, ‘I’, as shown in Fig.
10.

The cluster ’P’ hosts VSAS and provides interfaces to external VSDS. On
configuration, the video streams are captured and transformed into a proper
message, which is then grouped into micro-batch, compressed, and loaded to the
respective queue in the cluster ’K’.

The cluster ’K’ deploys the distributed messaging system, where the acquired
video streams, IR, and anomalies produced by LVSM are buffered. In this
context, the cluster ‘K’ is composed of a set of three types of queues for
each service, i.e., RIVA_ID, RIVA_IR_ID, and RIVA_A_ID, as described in IV-A1.

The mini-batches of video streams residing in the distributed broker’s RIVA_ID
queue need to be persisted to the UPDDS and ISDDS data stores. For this
purpose, the cluster ’V’ deploys three types of L-CVAS modules, i.e., VSCS,
Video Processor and Persistence. The first module allows the cluster ’V’ to
read the video stream mini-batches from RIVA_ID topics in the Cluster ’K’. The
cluster ’V’ then processes the consumed video data, encodes it, and extracts
the metadata from the video. Finally, the video stream persistence module
saves the video data and the respective metadata to the UPDDS and ISDDS,
respectively.

The cluster ’S’ is responsible for processing the video stream in near real-
time while using the IVA services. Different stream processing engines, e.g.,
Apache Spark Stream, can be used for RIVA. The cluster ’S’ deploys four types
of L-CVAS modules. The first module is VSCS and is used to consume the video
streams from the RIVA_ID queue in the cluster ’K’. The second type of module
is RIVA services. The RIVA service is the actual video stream analytics
service that analyzes the videos. The video RIVA service is loaded according
to the RIVA services subscription contract made by the L-CVAS user. The RIVA
services can be pipelined, and the IR might need other applications in the
multi-subscription scenario. Thus the IR producer/subscriber is used to send
and receive the IR according to the application logic to and from the IR queue
in Cluster ’K’.

The fourth type of module is LVSM producer. The contextual IVA services (IVAS)
instance deployed in the cluster ’S’ should have some domain-specific goal and
can produce anomalies if analyzed any. The L-CVAS support real-time anomalies
delivery system. The IVAS sent the anomalies continuously to the LVSM producer
and the LVSM producer to the respective anomalies queue RIVA_A_ID in the
cluster ’K’.

The cluster group ’I’ read the IR from the RIVA_A_ID queue in cluster ’K’
continuously and sent it to the ISDDS ’s IR data store for persistence. If the
subscribed service is using the ontology then the IR are also mapped to the
VidOnto triple residing in the Knowledge Curation Server ‘T’.

The final type of cluster in the Streaming Execution Scenario is cluster ’N’
and is known as Anomalies Notification Cluster. This cluster aims to read
anomalies from the RIVA_A_ID queue in cluster ’K’ and send the same to the
ISDDS for persistence and also delivered in real-time to the video stream
source owner in the form of alerts.

####  IV-F2 Batch Execution Scenario

The L-CVAS architecture is also equipped with BIVA. Unlike the RIVA, the BIVA
is analyzed as an offline manner. The execution time of offline analytics is
proportional to the video dataset size and the subscribed BIVA service
computation complexity. The Batch execution life-cycle undergoes through three
types of clusters, i.e., ’R’, ’M’, ’B’.

The cluster ’R’ allows the L-CVAS user to upload the batch video dataset to
the L-CVAS cloud and configure three types of L-CVAS modules. The first type
of service is Batch Video Acquisition Service, which is used to acquire batch
video datasets. Once uploaded to the node buffer, the batch dataset is
processed by the activated Video Processor to extract the metadata from the
batch videos. After processing the batch video dataset and the respective
metadata are persisted to the UPDDS and ISDDS, respectively. Similarly, the
cluster ’M’ works the same way as that of cluster ’R’, but this one is
responsible for model management.

In the batch video analytics, the supporting layers deploy various contextual
multi-domain offline BIVA services. This cluster loads the instance of the
RIVA services as per user contract and processes the videos in an offline
manner. Once subscribed, this cluster loads the batch video data set and model
from the UPDDS. Similarly, the IR and anomalies are maintained in the ISDDS.
The acquired video streams residing in the UPDDS is also illegible for offline
analytics.

Finally, the Web Server ’W’ deploys the Web User Interface (as described in
IV-D), i.e., allow the users to interact with L-CVAS.

##  V IVA; Constituents, and Predominant Trends

This section review the existing IVA literature and can be classified into
four classes under the umbrella of IVA, i.e., CBVR, IVA Surveillance and
Security, Video Summarization, and Semantics Approaches, as shown in Fig. 11.
We also show that how L-CVAS can be used under these application areas.

Figure 11: IVA taxonomy based on literature.

###  V-A Content-Based Video Retrieval

CBVR has applications from video browsing to intelligent management of video
surveillance and analysis. To uphold advancement in CBVR, since 2001, the
National Institute of Standards and Technology has been sponsoring the annual
Text Retrieval Conference Video Retrieval Evaluation [162, 163]. The CBVR is
an active research area, and several surveys are available, i.e., [32, 164,
163, 165, 166]. However, here we discuss some of the scalable CBVR system
being proposed in the literature.

In literature, some researchers tried to exploit distributed computing
technology for the development of large-scale CBVR systems. Shang et al. [167]
utilize the time-oriented temporal structure of videos and the relative gray-
level intensity distribution of the frames as a feature base. Their method is
expensive in terms of parallel processing due to the video semantics, and then
all the frames of a video must be processed within the same execution
environment. Hence this approach is challenging to parallelize accurately and
efficiently even for the state-of-the-art big-data frameworks. Wang et al.
[168] proposed a novel MapReduce framework called Multimedia and Intelligent
Computing Cluster for near-duplicate video retrieval for large-scare
multimedia data processing by joining the computing power of CPU’s and GPU’s
to speed up the video data processing. They extract the keyframes using
uniform sampling, store the keyframes to HDFS, perform local feature
extraction using the Hessian-Affine detector [169] to detect interest points.
K-means clustering over the feature vectors is utilized to generate visual
words following the BoF [170] model, thus generating BoF-based feature
vectors. Ding et al. [171] used big data processing technologies to design a
human retrieval system on extensive surveillance video data called SurvSurf.
Motion-based segments called M-clop were detected, which were utilized to
remove redundant videos. Hadoop MapReduce framework was used to process
M-clips for human detection and motion feature extraction. Vision algorithms
were accelerated by processing only sub-areas with significant motion vectors
rather than entire frames. Further, a distributed data store called V-BigTable
on top of HBase was designed to structuralize M-clips’ semantic information
and enables large-scale M-clips retrieval. They stated that SurvSurf
outperforms the baseline solutions in terms of computational time and with
acceptable human retrieval accuracy.

Authors in [172] proposed Marlin for video big data similarity search. They
used parallel computing to extract features from the acquired video micro-
batches, which are then persisted in a distributed feature indexer. The
proposed indexer was able to index incremental updates and real-time queries.
They designed a fine-grained resource allocation with a resource-aware data
abstraction layer over streaming videos to upsurge the system throughput. They
reported Marlin achieved 25X and 23X speedup against the sequential feature
extraction algorithm and similarity search, respectively. The challenge of
extracting distinctive features is addressed by Lv et al. [173] for the
efficiency of extraction closely related videos from the large scale data
based on local and global features utilizing Spark. To balance precision and
efficiency, they introduced a multi-feature based distributed system,
including local and global features. They combined local feature SIFT, Local
Maximal Occurrence, and global feature Color Name. Lastly, they developed the
system in a distributed environment based on Apache Spark. Further, M. N. Khan
et al. [174] proposed FALKON for large-scale CBVR that utilized distributed
deep learning on top of Apache Spark for accuracy, efficiency, fault
tolerance, and scalability. Motivated by the fact that Apache Spark, by
default, does not provide native video data structure, they developed a
wrapper on the top of Spark’s RDD called VidRDD. Utilizing VidRDD, first, they
performed structural analysis on the videos, and then index the extracted deep
spatial and temporal features in their designed distributed indexer. Finally,
they evaluate their proposed system and show performance improvement in terms
of scalability and accuracy. Likewise, Lin FC. et al. [175] put forward a
cloud-based face video retrieval system while utilizing deep learning. First,
pre-processing operations like termination of blurry images, and face
alignment are performed. Then the refined dataset is constructed and used to
pre-train the CNN models, i.e., ArcFace, FaceNet, and VGGFace for face
recognition. The results of these three models are compared, and the efficient
one was chosen for the retrieval system development. The input query in their
proposed system is a person’s name. If the system detects a new person, it
performs enrolling that person. Finally, timestamped results are returned
against a query. A prototype of the proposed face retrieval system was
implemented and reported its recognition accuracy and computational time.

####  V-A1 CBVR under L-CVAS Architecture

L-CVAS provides an elegant and flexible six steps solution for the
implementation and customization of scalable video indexing and retrieval, as
shown in Fig. 12. First, the VSAS component acquires the video streams from
VSDS in the form of mini-batches, and in the case of batch analytics, video
data is loaded from the RAW DS and feeds to the VBDPL.

In the second step, VBDPL perform pre-processing operations and feature
extraction. The former one encompasses structural exploration of a video,
i.e., video scenes, shots detection, frames, and keyframes extraction, while
the latter one extracts the low-level features. These low-level features can
be keyframes’ static features (texture-based, color-based, and shape-based),
object features, and/or motion features (trajectory-based, statistics-based,
and objects’ spatial relationships-based). The extracted features are then
handed over to the VBDML for classification and annotation.

In the third step, the semantic and high dimensional video feature vectors’
indexes make the representative index for persisted video sequences in IR-DS.
The IR-DS is synchronized with RAW-DS.

Figure 12: Components of CBVR under the Lambda CVAS architecture.

In the fourth step, through the WSL, the users are allowed to query the
desired videos. In literature for video retrieval, various types of queries
have been utilized for video retrieval. The queries can be categorized as
Query by Example, Sketch, Objects, Keyword, and Natural Language. In theQuery
by Example, similar videos or images to the given sample video or image is
extracted using feature similarity. In Query by Sketch, the features are
extracted from users drawn sketches (sketch represent the required videos) and
compared against the features generated from the stored videos [176]. In Query
by Object, the user give an object image then the CBVR system search for all
occurrences of the object in the video database [177]. Likewise, the last two
approaches use Keywords and Natural Language [178] as query. Combination-based
Query combines different types of queries for CBVR, was also adopted by some
researchers such as [179, 180].

In the fifth step, the CBVR system search for similar videos against the user
query. For similarity measure approaches can be classified as feature, text,
Ontology and combination based matching. The similarity measure depends on the
type of query. Feature matching based similarity measure is the average
distance between the features of the corresponding frames [181]. Query uses
uses low-level features for extracting relevant videos. The benefit of this
approach is that the video similarity can be found easily from the features
but is not appropriate for semantic similarity, which is conversant to users.
In text matching based similarity measure, the name of each concept is matched
with given query terms so that to find the videos that satisfy the query. The
best example of this approach is that of Snoek et al. [182]. This approach is
simple, but the query text must include the relevant concepts to get pleasing
search results. In Ontology-Based Matching, the similarity between between
semantic concepts is measured utilizing ontology. Query descriptions are
enriched from knowledge sources, such as the ontology of concepts. Adding
extra concepts can improve the retrieval results [183] but can also decline
search results unexpectedly. This approached is further explained in section
V-D. Combination-based matching “leverages semantic concepts from a training
collection by learning the combination strategies [87], and query-class-
dependent combination models [180]” [89]. Up to some extent, using this
approach, the concept weights can be automated, and hidden semantic concepts
can be handled but are difficult to learn query combination models.

Finally, the ranked result-set is presented to the user for browsing. To
increase efficiency and to optimize the results of the CBVR system, many
researchers use the obtained relevance feedback from the user. This feedback
can be categorized as explicit, implicit, and pseudo feedback. In the first
case, users are asked to select relevant videos from the previous results
actively [184]. The explicit feedback approach obtains better results than the
other two approaches, but direct user involvement is required. In the second
case, the retrieval results are refined by exploiting the user’s interaction
with the system, i.e., clicking pattern [185]. In the third case, there is no
involvement of the users. The user’s feedback is produced through positive
(closely related to the query sample) and negative sample (different from the
query sample) from the previous retrieval. These samples are directed to the
system for the next search. Yan et al. [186], and Hauptmann et al. [187]
approaches are based on pseudo-relevance feedback approach. This approach
substantially reduces the user interaction, but the semantic gap between low
and high-level features obtained from different videos does not always agree
with the similarities between the user-defined videos.

###  V-B Real-time Intelligent Video Analysis and Surveillance

In the context of growing security concerns, the surveillance meant to
criminality and intrusion detection. Video surveillance is not limited to
these, but it encapsulates all aspects of monitoring to capture the dynamics
of diverse application areas, e.g., transportation, healthcare, retail, and
service industries. A generic use case for RIVA has been shown in section
IV-E, Fig. 9. Likewise, domain-specific RIVA services for security and
surveillance can be created under L-CVAS architecture. In this section, we
discuss recent literature on how the researchers advancing the activity and
behavior analytics in the video streams with the aim of intrusion and crime
detection, scene monitoring, and resource tracking.

####  V-B1 Video Segmentation for Action Recognition

A combination of numerous actions, objects, and scenes forms complex events
[188]. Video analytics against complex events is a nontrivial task. Complex
event detection demands the association of multiple semantic concepts because
it is almost impossible to capture the complex event through a single event
class label [189]. Video segmentation is required to mine informative segments
regarding the event happening in the video. For effective event detection in
video segmentation, it is vital to take into account the temporal relations
between key segments in a particular event. The event videos hold intra-class
variation, and several training videos are required to consider all possible
instances of event classes. Song and Wu [190] suggested a methodology to
extract key segments automatically for event detection in videos by learning
from a loosely tagged pool of web videos and images. For the positions of key
segments and content depiction in the video, they used an adaptive latent
structural SVM model and semantic concepts, correspondingly. They also
developed two types of models, i.e., Temporal Relation Model and Segment-Event
Interaction Model, for the temporal relations between video segments, and for
evaluating the correlation between key segments and events respectively. They
adapted labeled videos and images from the Web into their model and employed
’N’ adjacent point sample consensus [191] for noise elimination. A knowledge-
base was produced by Zhang et al. [192] to decrease the semantic gap between
complex events. To effectively model event-centric semantic concepts, they
used a large-scale of web images for learning noise-resistant classifiers.

Action recognition in videos encompasses both segmentation and classification.
Action recognition can be tackled by sliding windows and aggregation in
sequential as well as isolated manner, or by performing both tasks parallel.
[193, 189] are some of the good examples of video segmentation based event
detection. The related literature on video segmentation based action
recognition can be classified as action segmentation, a depth-based approach,
and deep learning-based motion recognition.

The most popular action segmentation model is dynamic time warping scheme
[194]. Action segmentation from videos using an appearance-based method
considers a comparison between the start and end frames of contiguous actions.
Quantity of movement [195] and KNN along with HOG [196] are commonly used to
identify the start and stop frames of actions. Similarly, for action
recognition, depth-based approaches have been developed which consist of
binary range-sample feature [197], capturing local motion and geometry
information [198], histogram oriented 4D normal [199], and combination of
depth motion map and HOG [200]. [201, 202, 203] have applied deep learning
approaches to depth-based action recognition methods. Besides, for motion
recognition, deep learning has been utilized in many ways. One is a suboptimal
way, in which video represented as still images is fed to the channel of a CNN
[97, 117]. Sometimes video as compact images are given as input to the already
trained CNN to achieve good performance [204]. Video in the form of a sequence
of images is input to the Recurrent neural network (RNN) for sequential
parsing of video frames for long as well as short-term patterns [205, 206]. In
order to introduce a temporal dimension, video is regarded as a volume and
substitutes the 2D filters of CNN with 3D filters [207, 208].

In many application areas, e.g., resource tracking, action recognition, human
behavior recognition, and traffic control, object tracking and motion
detection in videos are vital. The process of detecting moving objects in
videos is known as object tracking. For object tracking, initially, the
foreground information is extracted in videos, and then the background
modeling of the scene is captured using a background subtraction algorithm,
i.e., IAGMM [209, 210]. To increase the accuracy, subsequently, shadow
elimination algorithms are applied to the foreground frame [211]. A connected
component algorithm is used to determine the bounding box of an object. To
ensure frame-to-frame matching of the detected object, a method such as
adaptive mean shift [212] can be used for comparison. Factors like size and
distance are used to object matching between frames. Finally, the occlusion
scheme is used to detect and resolve occlusion. On the other hand, motion
detection can be detected via foreground images extracted by the Gaussian
Mixture Model background and connected component algorithm for noise removal.
The area of detection is refined using a connected component algorithm and
produces the bounding box information of the moving objects [209]. The output
of the detection is a binary mask representing the moving object for each
frame in a particular sequence. Object detection for moving objects is
challenging, especially in the case of shadows and cloud movement [213]. The
related literature for moving object detection and classification can be
categorized as a stationary camera with a moving object and moving objects
with moving camera.

##### Moving object detection with stationary camera

In fixed camera video, the background image pixels in each frame remain the
same, and thus simple background subtraction techniques are required. The
object detection approaches using a fixed camera can be grouped as feature-
based, motion-based, classifier-based, and template-based models [214].
Categorization of object tracking in videos into point tracking, kernel
tracking, and silhouette tracking as well as feature-based, region-based, and
contour-based was performed by [214, 215] respectively. Unlike a fixed camera,
moving object detection with a moving camera is relatively challenging because
of camera motion and background modeling for generating foreground and
background pixel fails [216].

  * •

Trajectory classification involves computing long trajectories for feature
point and discriminating trajectories that belong to different objects from
those backgrounds using the clustering method. Some recommended algorithms
include compensating long term motion based on flow optic technique [217],
bag-of word classifier, and pre-trained CNN method for detecting moving object
trajectories [218].

  * •

In background modeling based methods, for each sequence, the frame-by-frame
background is created utilizing the motion compensation method. Some popular
algorithms are Mixture of Gaussian [219], complex homography [220], gaussian-
based method [221], adaptiveMoG [222], multi-layer homography transform [223],
thresholding [224], and CNN-based method [225] for background modeling.

  * •

In extension of background subtraction method, low rank and sparse matrix
decomposition method for static camera [226] are extended to moving camera. If
there exists coherency between a set of image frames, then a low-rank
representation of the matrix created by these frames contains the coherency,
and the sparse matrix representation contains the outliers, which represents
the moving object in these frames. Low rank and sparse decomposition involve
segmenting moving objects from the fixed background by applying principal
component pursuit. It is a valuable technique in background modeling.
Mathematical formula and optimization of this method can be found in [227].

####  V-B2 Behavior Analysis

Usually, a camera is mounted nearby the digital displays to analyze and
understand human behavior by investigating user interfacing with digital
display [228]. Commercial tools have been developed to analyze audience
behavior using video analytics while considering parameters like age, gender,
distance from the display, and sight and spent time. The obtained data can
then be used to improve advertising campaigns in combination with sales data
[229].

Recently, crowd analytic, i.e., human detection and tracking, have attracted
the attention of the researches. The exploration of both group and individual
behavior to govern abnormality scope the crowd analysis. Congestion analysis,
motion detection, tracking, and behavior analysis are the main attributes of
crowd analytics. While performing crowd analysis, factors like terrain
features, geometrical information, and crowd flow can be considered.

For analysis of the crowded scene, motion features are vital, and can be
categorized as flow-based features, local spatiotemporal features, and
trajectory features [230]. These features have applications in crowd behavior
recognition, abnormality detection in-crowd, and motion pattern segmentation.

##### Flow-based Features

The flow-based features are pixel-level features, and in literature, different
schemes have been proposed [231, 232], which be further classified as optical
flow, partial flow, and streak flow. Optical flow technique encompasses
computing pixel-wise motion between successive frames and can handle multi-
camera objection motion. It has been applied to detection crowd motion and
crowd segmentation [233]. This approach, optical flow, is unable to capture
spatiotemporal attributes of the flow and long-range dependencies. Particle
flow contains moving a grid of particles with the optical flow and providing
trajectories that maps a particle’s initial position to its future or current
position. It has an application in crowd segmentation and detection of
abnormal behavior [230]. Optical flow is unable to handle spatial changes.
Mehran et al. [232] proposed streak flow to overcome the shortcomings of
particle flow and to analyzing crowd video while computing motion field. This
approach, streak flow, captures motion information similar to particle flow;
changes in the flow is faster and performs well in dynamic motion flow.

##### Local Spatiotemporal Features

Flow-based features fail on the very crowded scene, and resultantly local
spatiotemporal features techniques are developed, which are 2D patches or 3D
cubes representation of the scene. Spatiotemporal features can be categorized
as spatiotemporal gradients, and motion histogram. To capture steady-state
motion behavior, Kratz and Nishino [234] used a spatiotemporal motion pattern
model and confirmed the detection of abnormal activities. On the other hand,
motion histogram considers motion information within the local region. It is
not appropriate for crowd analysis because it takes a substantial amount of
time, and is subject to error. However, some improvements have been shown in
the literature to motion histogram, e.g., [235, 231].

##### Trajectory Features

Trajectory features signify tracks in videos. The distance between object-
based motion features can be extracted from the trajectories of objects and
can be utilized to analyze crowd activities. The failure to obtain a full
trajectory in dense crowd leads to the concepts of tracklet. The tracklets are
extracted from the dense region and enforce the spatiotemporal correlation
between them to detect patterns of behavior. Tracklet is a fragment of a
trajectory obtained within a short period, and the occurrence of occlusion
leads to closure. Tracklets have been used for human action recognition [236,
31] and for the representation of motion in crowded scenes [237, 238].

####  V-B3 Anomaly Detection

Anomaly detection an application area of crowd behavior analysis and is
domain-dependent. Anomaly in a video occurs when the analyzed pattern drifts
from the normal in a training video. The related literature of anomaly
detection can be categorized into three, i.e., trajectory-based, global
pattern-based, and grid pattern-based method of anomaly detection.

##### Trajectory-based method of anomaly detection

In trajectory-based anomaly detection, objects are formed from the segment
scenes, and then the object is followed in the video. A trajectory is caused
by the tracked object, which describes the behavior of the object [239]. For
the evaluation of abnormality in trajectory-based methods have been used i.e.,
single-class SVM [240], zone-based analysis [241], semantic tracking [242],
String kernels clustering [243], Spatiotemporal path search [244], and deep
learning-based approach [245] have been used.

##### Global pattern-based method of anomaly detection

In a global pattern-based technique, the video sequence is analyzed in whole,
i.e. low, or medium-level features are extracted from video using
Spatiotemporal gradients or optical flow methods [246]. The technique is
suitable for crowd analysis because it does not individually track each object
in the video but is challenging while locating the position where the anomaly
occurred. Approached used in the global pattern-based method are Gaussian
Mixture Model [247], energy model [248], SFM [249], stationary-map [250],
Gaussian regression [251], PCA model [252], global motion-map [253], motion
influence map [254], and salient motion map [255].

##### Grid pattern-based method of anomaly detection

In a grid pattern-based method, splits frames into blocks and individually
analyze pattern on a block-level basis [256]. If ignoring inter-object
connections that lead to processing efficiency. Spatiotemporal anomaly maps,
local features probabilistic framework, joint sparsity model, mixtures of
dynamic textures with Gaussian Mixture Model, low-rank and sparse
decomposition, cell-based texture analysis, sparse coding and deep networks
are used in evaluating grid pattern-based methods [257].

###  V-C Video Summarization

Video big data are facing the challenge of sparsity and redundancy, i.e.,
hours of videos with less meaningful information, which creates many issues
for viewing, mining, browsing, and storing videos. It has motivated
researchers to find ways to shorten hours of videos and led to the field of
video summarization. Video summarization is a process of generating a shorter
video of the original one without spoiling the capability to comprehend the
meaning of the whole video [258]. The video summarization can be classified as
Static Video Abstracts, Dynamic Video Skimming, and Video Synopsis.

##### Static Video Abstract

These approaches include a video table of contents, a storyboard, and a
pictorial video summary. For example, Xie and Wu [259] propose an algorithm to
generate a video summary for broadcasting news videos automatically. An
affinity propagation-based clustering algorithm is used to group the extracted
keyframes into clusters, aiming to keep the relevant keyframes that
distinguish one scene from the others and remove redundant keyframes. J. Wu et
al. [260] were motivated by the notion from high-density peaks search
clustering algorithm. They proposed a clustering algorithm by incorporating
significant properties of video to gather similar frames into clusters.
Finally, all clusters’ centers were presented as a static video summary.
Bhaumik et al. [261] proposed a summarization technique where they detect
keyframes from each shot that eliminates redundancy at the intra-shot and
inter-shot levels. For frames redundancy elimination, SURF and GIST feature
descriptors were extracted for computing the similarity between the frames.
The quality of the summaries obtained by using SURF and GIST descriptors are
also compared in terms of precision and recall.

Similarly, Zhang et al. [262] propose a subset selection technique that
leverages supervision in the form of human-created summaries to perform
automatic keyframe-based video summarization. They were motivated by the
intuition that similar videos share similar summary structures. The
fundamental notion is to nonparametrically transfer summary structures from
annotated videos to unseen test videos. Concretely, for each fresh video, they
first compute the frame-level similarity between annotated and test videos.
Then the summary structures are encoded in the annotated videos with kernel
matrices made of binarized pairwise similarity among their frames. Those
structures are then combined into a kernel matrix that encodes the summary
structure for the test video. Finally, the summary is decoded by feeding the
kernel matrix to a probabilistic model called the determinantal point process
to extract a globally optimal subset of frames. M. Gygli et al. [263] used a
supervised approach to learn the importance of the global characteristics in
summary by extracting deep features of video frames. J. Mohan et al. [264]
proposed a technique that utilizes sparse autoencoders. Motion vectors have
been used for the elimination of redundant frames, and then high-level
features are extracted from frames using sparse autoencoders. These high-level
feature vectors are then clustered using the K-means algorithm. The frames
closest to the centroid of each cluster are selected as keyframes of the input
video. Ji, Zhong, et al. [265] employed tag information, i.e., titles and
descriptions, as the side information for the generation of summarization. A
sparse auto-encoder was used as the primary model to generate the final
summary, where the input and output were multiple videos and keyframes set,
respectively. They fused the visual and tag information to guide visual
features, which constrained the sparse auto-encoder to select the candidate
keyframes.

##### Dynamic Video Skimming

A summary video is formed from the video segments of the original video to
remove redundancy or to summarize based on object action or events. As an
example of object-based skimming, Peker et al. [266] proposed a video skimming
algorithm while utilizing face detection on broadcast video programs. In the
algorithm, the attention was given to faces, as they establish the focus of
most consumer video programs. Ngo et al. [267] represent a video as a complete
undirected graph and exploit the normalized cut algorithm to form optimal
graph clusters. One-shot was taken from each cluster of visually alike shots
to remove duplicate shots. Xiao et al. [268] mine frequent patterns from a
video. A video shot importance evaluation model is utilized to choose useful
video shots to create a video summary. For personal videos, Gao et al. [269]
developed a video summarization technique, which encompasses a two-level
redundancy detection procedure. First, they terminated redundant video content
with the hierarchical agglomerative cluster method at the shot level. Then
parts of shots were selected, based on the ranking of the scenes and
keyframes, to generate an initial video summary. Finally, to terminate the
redundant information, a repetitive frame segment detection step was utilized.
They verified the proposed technique through a prototype while using TV
datasets (movies and cartoons videos) and reported the performance in terms of
compression ratio (81%) and recall (87.4%). In [270], for user-generated video
summarization, both the representativeness and the quality of the selected
segments from an original video were considered. They stated that user-
generated videos contain semantic and emotional content, and its preservation
is vital. They have designed a scheme to pick representative segments that
include consistent semantics and emotions for the whole video. To ensure the
quality of the summary, they computed quality measures, i.e., motion and
lighting conditions, and integrate them with the semantic and emotional clues
for segment selection.

Ji, Zhong, et al. [271] addresses the issue of supervised video summarization
by formulating it as a sequence-to-sequence learning problem, where the input
and output is a sequence of original video frames and a keyshot sequence,
respectively. The notion is to learn a deep summarization network with
attention mechanism to mimic the way of selecting the human keyshots. The
proposed framework was called attentive encoder-decoder networks for video
summarization. They utilized the BiLSTM encoder for encoding the contextual
information among the input video frames. For the decoder, two attention-based
LSTM networks, are explored by using additive and multiplicative objective
functions, respectively. The results demonstrate the superiority of the
proposed framework against the state-of-the-art approaches, with remarkable
improvements. J. Wu et al. [272] were motivated by the fact that multi-video
summarization is significant for video browsing and proposed a technique where
multi-video summarization was formulated as a graph problem. They also
introduced a dynamic graph convolutional network to measure the importance and
relevance of each video shot locally as well as globally. They adopted two
approaches to address the inherent class imbalance issue of video
summarization. Additionally, a diversity regularization to encourage the model
to generate a diverse summary was introduced. The results demonstrate the
effectiveness of our proposed model in generating a representative summary for
multiple videos with encouraging diversity. Z. Sheng-hua et al. [273] proposed
a deep learning-based dynamic video summarization model. First, they addressed
the issue of the imbalanced class distribution in video summarization. The
over-sampling algorithm is used to balance the class distribution on training
data. They proposed two-stream deep architecture with cost-sensitive learning
to handle the class imbalance problem in feature learning. RGB images are
utilized to represent the appearance of video frames in the spatial stream.
Likely, multi-frame motion vectors with deep learning framework are introduced
to represent and extract temporal information of the input video. Moreover,
they stated that the proposed method highlights the video content with the
active level of arousal in effective computing tasks and can automatically
preserve the connection between consecutive frames.

##### Video Synopsis

In this approach, activities from the stated time interval are collected and
moved in time to form a smaller video synopsis showing maximum activity, as
shown in Figure 13. The notion of video synopsis was pioneered by [274] in
2006 and proposed a two-phase approach, i.e., online and offline. The former
phase includes the queuing of the generated activities. The later phase
started after selecting a time interval of video synopsis with tube
readjustment, background formation, and object stitching. A global energy
function was defined and encompassing activity, temporal consistency, and
collision cost. Then the simulated annealing method was applied for energy
minimization. The video synopsis domain was further researched in single and
multi-view scenarios. Some recent examples of a single-view are [275, 276,
277]. He et al. [275, 276] brought advancement in activity collision analysis
by describing collision statuses between activities such as collision-free,
colliding in the same direction and opposite directions. They also offered a
graph-based optimization technique by considering these collision states to
improve the activity density and put activity collisions at the center of
their optimization strategy. Baskurt and Samet [277] concentrated on rising
robustness of object detection by suggesting an adaptive background generation
method. In another study, Baskurt and Samet [278] planned the object tracking
method specified for video synopsis requirements. Their approach focused on
long term tracking to represent each target with just one activity in video
synopsis. Single view scalable approaches for video synopsis were projected by
Lin et al. [279] while utilizing distributed computing technology. Their
proposed video synopsis approach encompasses steps like object detection,
tracking, classification, and optimization, which were performed in a
distributed environment. Ahmed, S. A. [280] proposed a query-based method to
generate a synopsis of long videos. Objects were tracked and utilized deep
learning for objects classification (e.g., car, bike, etc.). Through
unsupervised clustering, they identified regions in the surveillance scene.
The source and the destination represented spatiotemporal object trajectories.
Finally, user queries were allowed to generate video synopsis by smoothly
blending the appropriate tubes over the background frame through energy
minimization.

Figure 13: Simultaneous activities display in a video [274].

Further, the examples of multi-view video analytics are that of [281, 282].
Zhu et al. [281] proposed a framework to generate a unified synopsis of multi-
view videos. The synopsis is visualized by mapping multiple views to a common
ground plane. Multiple camera activities were allied via trajectory matching
in overlapping camera views. The process of producing a synopsis requires a
balance among minimizing the synopsis length, maximizing the information
coverage, and reducing the collision among objects’ track that are presented
concurrently. Likewise, Mahapatra et al. [282] proposed a multi-camera
approach for an overlapping camera network and modeled the synopsis generation
as a scheduling problem. They utilized three distinct methods, i.e., table-
driven approach, contradictory binary graph coloring approach, and simulated
annealing. Action recognition modules were integrated to recognize significant
actions, i.e., walking, running, bending, jumping, hand-shaking, and one or
both hands waving. The inclusion of such essential actions can help in the
synopsis length reduction while preserving the value. The synopsis length was
further reduced while utilizing a fuzzy inference system that computes the
visibility score for each object tracking. They stated that through the
contradictory binary graph coloring approach, they achieved a maximum
reduction in synopsis length. Zhang, Z. et al. [283], tried to address the
issue of video synopsis by joint object-shifting and camera view-switching to
show multiple synopsis results more compactly and understandably. The input
videos were synchronized and grouped the same object in different videos
together. Then they shifted the grouped objects with respect to the time axis
to gain multiple synopsis videos. They constructed a simultaneous object-
shifting and view-switching optimization framework to achieve encouraging
synopsis results. To address unified optimization, they further presented an
optimization strategy composed of graph cuts and dynamic programming.

####  V-C1 Video summarization aaS under L-CVAS

Fig. 14 shows the flow of the video summarization flow under the proposed
L-CVAS architecture. Through the WSL, the user first subscribes to the video
data-source to the video summarization service. Then the user preferences
allow the users to set the parameters required for video summary service and
personalization. The summary parameters encompass granularity level, type of
summary to be performed (e.g., overview, highlights, synopsis, etc.) and any
other as per the video summary service scenario. The personalization can be in
terms of specific features of the video, like people, objects, events, etc.
Through the VBDCL, videos are acquired from the video data-source and sent to
the VBDPL for pre-processing. In pre-processing, video units are extracted
(segmentation, shots, frame extraction, etc.) as per the requirements of the
video summarization service. Then multiple low and high-level features, such
as motion, color, aesthetics, semantics, etc., are extracted from the video
units. The extracted features from the basic video units are input to the
VBDPL for object/activity identification and clustering. Once done then the
next step is video summarization. The video summarization phase deploys the
actual logic, i.e., video unit selection, and redundancy removal. The video
unit selection and redundancy removal decide which video units should be
included in the video summary based on unit significance, summary length, and
other user’s parameters. This block also removes similar video units within
the video summary to achieve the best possible video summary, covering the
required details in the original video. Finally, the summary results are
delivered to the respective user through the WSL.

Figure 14: Generic flow of video summarization aaS under Lambda CVAS
architecture.

###  V-D Semantic-based Video Analysis

To bridge the semantic gap and to allow the machine to understand the visual
data, Semantic Web technologies can be incorporated [161]. Semantic Web
unlocked a new avenue for knowledge-based computer vision while enabling data
exchange between video analysis systems in an open and extensible manner. The
scientific community has exploited the Semantic Web concept for intelligent
video analytics to bridge the so-called semantic gap between the low-level
features and high-level human-understandable concepts. State-of-the-art
scholarly work can roughly be classified as semantic-based low-level, mid-
level, high-level analysis, and semantic-based video search and retrieval.

####  V-D1 Semantic-based Low-level analysis

Semantic-based low-level analysis refers to the formalization of the extracted
objects of interest from the videos. It then performs reasoning on
formalization such as detection and tracking across domain-specific videos
[284, 285]. Dasiopoulou et al. [286] proposed a multimedia ontology for
domain-specific video analysis. In semantic concepts, they consider object
attributed, low-level features, spatial object relations, and the processing
approaches while defining F-logic rules for reasoning that govern the
application of analysis methods. García et al. [287] proposed a knowledge-
based framework for video object segmentation, where relationships among
analysis phases are utilized. The main contribution is to provide a detailed
description of the scene at low, mid, and high semantic levels through an