A Novel Approach for Robust Multi Human Action Recognition and Summarization based on 3D Convolutional Neural Networks
Noor Almaadeed Omar Elharrouss Somaya Al-Maadeed Ahmed Bouridane Azeddine Beghdadi
cs.CV

















Shell et al.: Bare Demo of IEEEtran.cls for Computer Society Journals










Human actions in videos are 3D signals. However, there are a few methods available for multiple human action recognition. For long videos, it's difficult to search within a video for a specific action and/or person. For that, this paper proposes a new technic for multiple human action recognition and summarization for surveillance videos. The proposed approach proposes a new representation of the data by extracting the sequence of each person from the scene. This is followed by an analysis of each sequence to detect and recognize the corresponding actions using 3D convolutional neural networks (3DCNNs). Action-based video summarization is performed by saving each person's action at each time of the video. Results of this work revealed that the proposed method provides accurate multi human action recognition that easily used for summarization of any action. Further, for other videos that can be collected from the internet, which are complex and not built for surveillance applications, the proposed model was evaluated on some datasets like UCF101 and YouTube without any preprocessing. For this category of videos, the summarization is performed on the video sequences by summarizing the actions in each subsequence. The results obtained demonstrate its efficiency compared to state-of-the-art methods. 





Action recognition, multi human action recognition, video summarization, 3-dimensional convolutional neural networks.




A Novel Approach for Robust Multi Human Action Recognition and Summarization based on 3D Convolutional Neural Networks 
    Noor Almaadeed,
        Omar Elharrouss,
        Somaya Al-Maadeed,
        Ahmed Bouridane,
	Azeddine Beghdadi
        
Noor Almaadeed, Omar Elharrouss, Somaya Al-Maadeed are affiliated with the Department of Computer Science and Engineering, Qatar University,Doha, Qatar e-mail: (n.alali@qu.edu.qa, elharrouss.omar@gmail.com, and S_alali@qu.edu.qa).

Ahmed Bouridane is affiliated with Department of Computer and Information Sciences, Northumbria University, Newcastle upon Tyne NE1 8ST, United Kingdom.

Azeddine Beghdadi is affiliated with Department of Computer and Information Sciences, Galilee Institute,Paris 13 Nord Univeristy, Paris, France.

    July 24, 2019
================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

























§ INTRODUCTION





A large number of videos that capture different events throughout the world are produced on a daily basis using different types of cameras (surveillance, phones, and filming crew). However, detecting the temporal and spatial events requires retrieving the required information contained from the captured videos. This process of creating useful information about the content of videos can be achieved through video content analysis <cit.> which has a wide range of applications including the retrieval of particular information, providing automatic alerts, and data summarization. Human action recognition is another specific application of video content analysis which aims to recognize activities from a series of observations on actions of subjects and the surrounded environment and it is important for many other applications <cit.>. However, human action recognition is a complex technology due to the difficulty to extract information about a person's identity and psychological states. Hence; human action recognition has gained the interest of researchers in the computer vision area and many approaches have been proposed for the aim of developing this technology.



Human action recognition is an important task for many applications including video surveillance systems, video indexing and retrieval, sport application, and multimedia. Action detection, recognition, and summarization can be exploited to support many other tasks. For example, in sports applications, it can help to recognize and understand players' poses allowing to make a good decision on players' fouls, especially for football games that require correct decisions in many situations. Each human action has many characteristic patterns and appearances. In video processing, the effectiveness of such an action recognition method is related to the chosen features. Therefore, in order to recognize actions from video sequences, the existing algorithms available in the literature combine image action and spatiotemporal features, such that, the spatial features represent the visual appearance while the temporal one illustrates the dynamic motion. 


  Human action recognition approaches can be classified into two main categories: detection recognition and video classification. The detection recognition approaches start by first detecting the motion of the person's action, followed by recognizing the action(s). These approaches are validated using popular surveillance datasets such as KTH <cit.>, Weisman <cit.>, IXMAS <cit.>, UCF-ARG <cit.>, MHAD <cit.>, PETS series <cit.><cit.>. These datasets were recorded in controlled conditions, such that an individual actor performs an action in a video clip in nearly an identical manner in terms of camera's position, illumination variation with a simple static background; therefore, the recognition rate is very high (up to 90%) for almost all methods. 

In the case of classification approaches, videos are classified according to the action a video contains. These approaches are evaluated using newly developed datasets with videos that are either gathered from the web, compiled from YouTube or video clips that were recorded in realistic conditions where illumination conditions are not controlled, using moving cameras and unstated complex background. Datasets belonging to this category include Hollywood <cit.>, Hollywood2 <cit.>, UCF sport <cit.>, UCF50 <cit.>, UCF101 <cit.>, HMDB51 <cit.>, HMDB <cit.>, and ActivityNet <cit.>. The diversity of clip content (scale of the actors, objects change positions, etc.), the variation in a camera's motion, and the background obsolete analysis of this category of the dataset are explored. Table 1 summarizes the categories of existing methods and their description related to the utilized datasets.

This paper proposes a video summarization approach based on action recognition where human body actions are first detected and identified. By detecting and recognizing many human body actions  like illustrated in figure 1, the proposed method provides the ability also to summarize these detected action per person, unlike the exixting approach that recognize one action per image. The proposed consists of detecting, recognizing and summarizing  multiple actions made by many persons in the same video. The major contributions of this paper can be summarized as follows:



  
  * Preparation of data using an extraction of sequences of each human silhouette in each time .
  
  * Each sequence is exploited to recognize the action using a 3-dimensional CNN model.

  * An action-based video summarization for all persons in the scene
  
  * The proposed architecture is employed for video classification(for movies, YouTube videos ) without preprocessing to recognize the action in the video.

  The remainder of the paper is organized as follows. The literature overview including sequential-based and CNN-based methods related to our work are presented in section 2. The proposed method is presented in section 3. Experiments performed to validate the proposed method, is discussed in section 4. The conclusion and future works are provided in section 5.   








§ LITERATURE REVIEW 




 §.§ Sequential-based methods 


Several methods of human action classification have been proposed. These methods can be classified into three main types: appearance-based, motion-based, and space-time-based methods. The methods that based on montion in the videos  consist of an optical flow compution followed by a motion templates comparison with optical flow results . For the second catrgory, appearance-based methods, the motion history of the images in the sequence is extracted defore comparing it with the active shape models. For space-time methods, the recogntion is made by traning the space-time features to identify the actio. Some researchers have exploited visual search of compact descriptors to recognize the actions; for example, (CDVS) <cit.> and CDVS feature trajectories of detected human bodies <cit.>. The global and local features of the image given by CDVS features is a good technique for feature extraction in real-time because of the use of computing optimization. Other authors exploited the saliency of images to select the region that describe an action, followed by a clustering operation classify actions <cit.>. In another method, the authors propsoed a  non-linear SVM decision tree method that exploited Smith-Waterman and fast HOG3D features of each frame  <cit.>. Using human motion for action recognition purposes, the authors in <cit.>, extracted view-invariance between 3D and 2D videos. Then,  3D dense trajectories are extracted from 3D videos and used for recogntion. Based on histogram of oriented gradient (HOG) of motion history image (MHI) and Speeded up Robust Features (SURF) a human action recognition method is proposed in <cit.>. The authors in <cit.> proposed an action recognition method using the RGB-D videos as input and the HMM features of the recognition results for summarization step. 


  

In order to detect and recognize multiple human posture in videos, the authors in<cit.>  have used local feature descriptors including HOG and Block Orientation. In the same context, authors in <cit.> extract a new set of the local binary pattern (LBP), histogram of oriented gradient (HOG) and Harlick features. Extracted features are combined to generate another feature using joint entropy-PCA-based method and Euclidean distance. the recognition is made by classifying the selected features using a Multi-class SMV technic. Using Scale Invariant Feature Transform (SIFT) and Histogram for Oriented Gradient (HOG) for extraction of features in addition to Support Vector Machine (SVM) classifier, authors in<cit.> proposed a method to detect and recognize human actions. In <cit.>, authors exploit the interest points and the 3D scale-invariant feature transform (3D SIFT) extracted for each point, to classify human action. For recognition of human actions, authors in <cit.> use a novel representation of videos called high-dimensional vector before applying a dimension reduction operation using compressive sensing with Gaussian mixture random matrix (CS-GMRM). Some authors like in <cit.> have combined the covariance matrices as local spatiotemporal descriptors and log-Euclidean covariance matrices (LECM) extracted densely from action videos.
In the same context, authors proposed in <cit.> a behavior recognition approach where spatiotemporal features are extracted using 2D and 3D interest points. In <cit.>, based on the camera motion estimation, the authors improved dense trajectories in order to recognize actions. Features used in image processing fields <cit.>, are exploited for video action recognition including 3D histogram of oriented gradient (3DHOG), histogram of optical flow, motion boundary histograms (MBH), 3D-SIFT <cit.>, and improved dense trajectories (iDT) <cit.>. The robustness of such features is related to many characteristics. For example, MBH features are useful for camera motion and gives better results that HOF features. Also, iDT has been shown to outperform a combination of HOF and MBH <cit.>. The effectiveness is also related to the type of videos. For example, previous methods give good results in (UCF-101, HMDB-51) datasets but they can fail in the case of crowded videos and videos with large variations of action categories.



  





 §.§ Convolutional-neural-network-based methods
 

To improve the action recognition performance, recent works have employed various deep learning models <cit.>, <cit.>. Since human actions are extracted from multiple movements of human body or some parts of it, it is necessary that the recognition process should involve video browsing over time to learn the patterns of the visual appearance changes <cit.>. To achieve this, existing deep learning models based on 2D convolutional networks can be extended into 3D domain to capture the temporal information <cit.>. For example, the authors in <cit.> use the motion in consecutives frames to extract information for action recognition using a two-stream ConvNet architecture that incorporates both spatial and temporal networks.

In the same context, authors in <cit.> proposed a 3D CNN–based method using stacked frame cubes of actions to capture 3D spatial-temporal action signals. In addition, Wang et al. <cit.> built up a temporal pyramid pooling based CNNs for action representation comprising of an encoding, a pyramid pooling and a concatenation layer, which can transform both motion and appearance features. Moreover, in <cit.>, trajectory-pooled CNNs was designed by fusing Improved Trajectories into CNNs architecture, while an adaptive recurrent convolutional hybrid networks was proposed consisting of a data module and a learning module.




For the same reason, many methods use several input videos for deep learning modelling in order to recognize the human action. In <cit.>, Wang el al. use a combination of saliency-aware, 3D CNN model and long short-term memory (LSTM) to recognize human action. Another example of many features used with CNN model was proposed in <cit.> where the authors exploit the original frames, optical flow, and motion stacked difference mage (MSDI) as inputs of CNN model for human action recognition. Ma et al. <cit.> propose CNN-based methods for action recognition that use six-stream features in order to use a general action recognition where many inputs are used including the full images, human image representing just the human body, operation region represent the part of body in action, and the optical flow result of each one of the previous features. In the same category of methods that use multi-stream CNN-based technic, Tu et al. <cit.>propose a multi-stream convolutional neural network architecture to recognize human action. The proposed approach detects human body in action and the region of interest corresponding to the moving parts. The fusion of all CNN results is used to recognize the human action. Other methods use infrared images to recognize the human action <cit.>. In order to summarize the sport player actions, authors in<cit.> use a 3D CNN model with two streams as input the first represented the video of human body skeleton and the second represent the video of segmented data. For summarizing all the categories of human action recognition methods based on convolutional neural networks, Table 2 illustrates each category and the features used as well as the stream used.











 



§ PROPOSED APPROACH

 This section introduces the proposed approach which consists of four phases as illustrated in figure 2. In the first stage (i.e., the preprocessing phase), we extract detected and tracked persons to generate target-region sequences from the videos. Then, the proposed convolutional neural network architecture which includes both 3-dimensional and 2-dimensional networks where the 3DCNN is used to recognize action from the generated sequence videos of each person while the 2D counterpart is used to recognize the actions based on the Motion History Images (MHIs) of the detected action(s). The third phase, referred to as the post-processing stage,  is a presentation of a multi-action detection and recognition method for many targets in a video.In the last section (action classification), the proposed approach uses our proposed 3DCNN without any pre-processing on videos collected from the web and YouTube especially to deal with the more complex video contents.

After the recognition stage, the summarization of actions from the scene is performed for each person. The video summarization is based on the recognized action of each person at each time in the video. For that, this summarization can be represented by frames(each frame illustrates an action made) or just the timeline of all actions made by each person during his presence in the scene. 



 §.§ Pre-processing stage 
 In this work, we introduce a new representation of data to be suitable for multiple recognition of human action(s). The proposed representation is tailored towards surveillance videos captured by a fixed camera installed in private or public scenes with moving objects targeting surveillance video to recognize human actions. In addition, we are interested in the human motion regions because the human body can occupy a small and specific region in the video while the existing approaches use the entire video sequences to recognize human actions. 

In order to prepare data for convolutional neural networks for training, we isolated human bodies in the video to extract a sequence or clip of body motions during their action and their presence in the video. The detection is made using the method described in <cit.> which consists of background modeling and subtraction before segmenting the moving objects. As a result, we obtained the region of moving objects as presented in figure 2 and figure 3. The extracted sequence of the detected person is the input of the proposed neural network architecture.





 §.§ 3DCNN Model

Depending on the applications, the selection of the optimal CNN architecture is challenging. The proposed deep learning-based approach involves the preprocessing of action videos before feeding them to the convolution neural network. The preprocessing consists of extracting the target region that contains human bodies in action, followed by resizing the data before creating NumPy video. A 3-Dimensional Convolution Neural Network (3DCNN), which is supervised learning with a multistage deep learning network, has been implemented. 3DCNN is capable to learn multiple invariant features from the input's videos.
 
The proposed 3DCNN-based architecture illustrated in Figure 4 consists of two parts, a 3D encoder-decoder network followed by the classification block of convolutional-max-pooling layers. 

The 3D encoder-decoder network, inspired by <cit.>, is based on VGG16 architecture for generating features and increasing a greater spatial extent. The encoder-decoder allows complex multiscale learning from a 3D volume that represents the video. Also, the skip connections allow learning with high-resolution features and generate a new higher resolution response with multi-scale influences. Also, the mixing layers combine the output of two blocks and minimize the multiscale objective function.

The second part of the proposed model of classification block, as illustrated in figure 4, is composed of three 3D convolution-pooling units, six convolutional layers and three MaxPooling layers, one flattened layer, and two fully connected layers. The output layers consist of ten neurons that represent the number of actions. This block allows a transaction from 3D representation to flatten data which makes the model learning from multiscale layers also using a complex architecture that performant learning.

For the last six convolutional layers, we have used a parametric Rectified Linear Unit (PReLU) as the activation function, which is a generalized parametric formulation of ReLU. This activation function adaptively learns the parameters of rectifiers and improves the accuracy at negligible extra computational cost. Only positive values are fed to the ReLU activation function while all negative values are set to zero. PReLU assumes that a penalty for negative values, and it should be parametric. The PReLU function can be defined as:  

    f(y_i)={[    y_i if y_i>0; a_iy_i if y_i≤0 ].

where a_i controlling the slope of the negative part. When  a_i= 0, it operates as an ReLU; when a_i is a learnable parameter, it is referred to as Parametric ReLU (PReLU). Figure 5 shows the shape of PReLU activation. If ai is a small fixed value, PReLU becomes LReLU (a_i= 0.01). PReLU can be trained using the backpropagation concept.

For the training and testing, we have used the preprocessed data from KTH, Weizmann, and UCF-ARG datasets. The model is trained using CrossEntrpy with a batch size of 64 examples, a learning rate of 0.001 like illustrated in Table 3.



 §.§ Multi-Action detection and recognition and post-processing.
 In the case of many moving objects or persons in the scene, we detect and track each person in the scene to generate a sequence of each one of them. The tracking helps generate a sequence of the same detected person during his presence in the scene. To achieve this, we have used the extended version of kernelized correlation filter (KCF) based method for tracking <cit.>. From the tracking results, we are able to extract and generate the sequence of each person which represents the human body actions. Recognizing the activity of different persons in one scene has the added advantage of extracting a sequence for each person.

The proposed approach using the proposed pre-processing can be applied for the recognition of each of these actions. The detection and multiple tracking allow recognizing many actions made by many persons at the same time.



After the extraction of each person sequence, the recognition is performed using the proposed 3DCNN architecture using the sequence as input. Each RGB video (or sequence) may contain some redundant (non-useful) information like the static background. For that, we apply the 3DCNN also on the RGB sequence without background called BS-video. Also, in order to minimize the time of recognition we use Motion History Images (MHI) extracted from BS-video and apply the 2DCNN version of the proposed architecture. The use of MHI improves the recognition rate. Figure 6 illustrates the use of different inputs of the deep learning model.





 §.§ Action classification.
The main difference between surveillance videos and other types of videos from movies, phone cameras, etc. is related to the stability of the camera as well as the coverage scope. Surveillance video focuses on places that can be widely covered. The other category of videos collected from the Web and YouTube or movies usually consist of crowded videos and videos with large variations of action categories. For these kinds of videos, the content is very difficult to be analyzed, making the extraction of patterns and the recognition of the action extremely very difficult. Various researchers have attempted to classify these videos based on the action contained in the videos. Several datasets including YouTube, UCF101, HMDB51 contain millions of videos that have been constructed.

This paper proposes to recognize the action(s) from these videos using our proposed 3DCNN model. The videos are introduced to the neural network without any preprocessing and the recognition is made by learning the patterns using all the datasets. From the experimental results, the proposed model provides an improved recognition rate compared to the state-of-the-art-methods.





   





 §.§ Video summarization based on recognized actions

Video summarization aims to generate a short version of a video as a representation, using key frames of important subsequences <cit.>. This summarization provides a rapid view of the information contained in a large video. It also provides a good evaluation for users of the video and provides knowledge regarding the topic and the most important content in the video<cit.>. Considering the information contained in each video, many methods have been developed using several techniques<cit.>.

Summarizing videos using human actions is propsoed in this paper. After recognizing each action we summarize this action using a timeline. These operation is performed for each person action during his presence in the scene.
For the surveillance video that contains persons acting in the scene tha sumamrization is a representation of the actions made by each one of them.
For the video classification (e.g. recognition of action within the video without analyzing the content), the summarization is performed by representing, using a timeline, the actions in each shot(subsequence) of the video. 




§ EXPERIMENTAL RESULTS 

This section presents the evaluation of the proposed action recognition approach. The experimentation was carried out using well known KTH, Weizmann, UCF-ARG, Interaction, IXMAS, and MHAD datasets. In addition, the 3DCNN model has been trained and tested on UCF101, Hollywood2, Youtube, and HMDB51 datasets. The 3DCNN model, with the new representation, was then evaluated in terms of the detection and recognition performances of multiple person actions at the same time. Further, the accuracy of the proposed method was evaluated using three different inputs such as RGB video, background subtraction video, and MHI features for surveillance videos. The results of the proposed method were then compared with the state-of-the-art methods that are based on surveillance videos, as well as methods based on action video classification. 

For the surveillance videos, the obtained accuracies for each dataset has been compared with a set of recent works. On KTH dataset the obtained results are compared with the methods proposed in <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, and <cit.>. For Weizmann dataset we compare the results with the methods in <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, and <cit.>. Also, for UCF-ARG, the obtained results are compared with the methods in <cit.>, <cit.>, and <cit.>. We compared our method with the method in <cit.>, <cit.>, and <cit.> for Interaction dataset. For IXMAS dataset, the proposed method has been compared with <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, and <cit.>.

The non-surveillance videos dataset which are for video action classification, we compared our approach with different methods on four datasets including UCF101, Hollywood2, YouTube and HMDB51. Some methods are trained and tested on all the four datasets while some others are trained and tested on one or two of these datasets. The list of methods that we compare with for video action classification are : <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, <cit.>, and <cit.>.










 §.§ System setup

To recognize human actions, model only regions of interest from each video, in the three datasets KTH, Weizmann, and UCF-ARG were extracted. This provides a new representation of data for the deep learning model. Each video was divided into shorts clips of 1 to 2 seconds that can represent the time of action, then converted to an NPZ file. The number videos for each action, with the new representation is from 200 to 300 for each action.

The proposed system has been implemented with python programming language using a laptop with GPU NVIDIA 1070 GTX. The training has been made using the new representation on surveillance videos while  MHAD dataset has been used for testing multiple human action detection and recognition. Figure 7 represents some frames from each video of MHAD dataset with also the background. For the training phase, we use two splits 80% and 90% of data where 10% and 5%of data is for validation and 10% and 5% for testing. Using 100 epochs, the accuracy of the proposed 3DCNN-based approach reaches a good accuracy using different inputs respectively.

To ensure good detection and recognition performance, the new presentation of the datasets was used by dividing data between training, validation, and testing. Also, to represent each action in many different situations, a set of videos with individual actions made by different actors, was used.

The recognition using surveillance videos has been performed using the detection of persons in the scene then recognize the action of each one. While the recognition of action in the other types of videos like UCF101 has been made by classifying the action in it using the 3DCNN proposed model.

The summarization of each video depending on the type of video, for surveillance videos we summarize the people's actions during their presence in the scene, but for videos like movies, the summarization is the description of the action in each shot of the video.



 §.§ Evalutation on surveillance data 




  §.§.§ Single human action recognition


In order to evaluate the proposed 3DCNN model on the surveillance videos, the training exploits different existing datasets for good learning of human actions. The video surveillance datasets used include KTH, Weizmann, and UCF-ARG, IXMAS, and Interaction.

The comparison of the proposed method has been performed also on each dataset, and the obtained accuracies are presented in Figure 8. From the presented figure, it can be observed that many methods succeed to recognize the actions with a good performance for all datasets. For the KTH dataset, which is a simple dataset that contains simple actions captured from the same angle of view, the methods [28], [64], [65], as well as the proposed method, reach better recognition rates. For instance, the Weizman dataset shows that all methods achieve good performance where the accuracy rate exceed 90% for all methods, also reach 99% for [85], [65], and the proposed method. The high accuracies for the two datasets come from the fact that the two datasets contain videos with simple backgrounds.

For the remaining dataset including Interaction and IXMAX that contains the action of interaction between more than one person, the accuracies depend on the methods used for interpreting the action in these types of videos. For example, in the IXMAS dataset, the proposed method as well as the methods in [69], [65], and [70] succeed to recognize actions with high accuracies. Where the Interaction dataset can be considered more complex than IXMAS which explains the accuracies in [67] and [68]. While the proposed method and the method in [65] achieve more improved results. 


  






  §.§.§ Multiple human action recognition 


For testing the accuracy of the multiple human action recognition method, the MHAD dataset has been used while the KTH, Weizmann, and UCF-ARG datasets, with the new representation, are used for training. The dataset contains 3 to 5 persons acting in the scene. In addition, three of the videos are taken in an outdoor environment with one in an indoor setup. 




Based on different features used as inputs for our 3DCNN and 2DCNN models, the accuracy increases when using the simpler features (sequence of motion of each person without a background as the input of 3DCNN model and motion history images of each person sequence as the input of 2DCNN model) with our representation of data. The RGB video and background subtraction video were used to evaluate the robustness of the proposed approach. For that, the results shown in Table 4 show the accuracy of human action recognition using the proposed model. From the table, it can be clearly observed that the accuracy increases from 87% to 96% when the background is subtracted from the RGB video and use the results(video without background)  as input. However, the background might contain non-useful information, therefore, remove the background increases the accuracy. With the 2D version of our CNN model, the Motion History Image was used, represented a summarization of the motion in the video, as input for the CNN model. The performance accuracy is better than the background subtraction counterpart; however, depending on the information (the content of the image) in each MHI that can be different from an action to another. For example, the motion history of the action of walking where the all-body move is different from the action of waving two hands was just the two hands are moving. The accuracy increases also when we augment the number of training data, as we can see the accuracy improved by 6% for (3DCNN- RGB video) and by 4% for 2DCNN+MHI. 



Figure 9 shows the training and validation accuracies graphs using the proposed approach for different inputs including RGB video, Background subtraction video, and MHI when the average recognition rates are 88%, 96%, and 99%, respectively. By removing the background from the RGB video, the accuracy is increased.

The 3DCNN model allows the system to recognize the action of each one during his presence in the scene. Figure 10 shows some examples of detected persons and their actions. Tests are made on three videos from the MHAD dataset and one video from the PETS2009 dataset. The visualized results and accuracy results, shown in Figure 10 and Table 4, reveal that the proposed approach detects and recognizes multiple human actions with an attractive accuracy. In addition, the proposed approach can be improved to be used to recognize multiple human actions in real-time. 

As represented in Figure 11, in the testing part a sequence of each detected person is generated and the action is recognized using the proposed model. The succession of analysis (motion detection, motion tracking, and the proposed architecture) provides a multi human action recognition. After that, each action can be represented in the original video. Also, in order to summarize the detected and recognized actions, each action is saved during the entire time of the video. Figure 11 represents a summarization using graphs of the
recognized actions of each person during his presence in the scene.




 §.§ Evaluation of action recogntion using video classfication  


Because of the complexity of the Movies or YouTube videos captured by moving cameras, the variation of points of view, the illumination changes, and video produced by a jitter camera. Many approaches use the entire video to classify the action without analyzing the content of the videos. This work proposes to deploy our 3DCNN model to recognize the actions in these videos. To validate and evaluate our proposed method for video action classification we exploit the other datasets collected from Movie and YouTube and use the video as there are without any preprocessing (no analysis of the content of the videos). We do training and testing on UCF101, UCF sport, YouTube, and HMDB51 datasets. Figure 12 illustrates some example from the used datasets. The Obtained results were compared with the state-of-art methods (video action classification approaches) shown in Table 5. From the accuracy results in table 5, we can observe that the proposed methods can recognize action even without preprocessing. Also, the recognition rates reach 98%, 91%, and 97% for UCF101, Hollywood, and YouTube Datasets respectively.

For summarization of this type of video, we collect a set of scenes from the datasets used and we recognize the action in each subsequence, after that, we represent the action recognized in each time in the video like illustrated in figure 13.





  



§ CONCLUSIONS

In this study, we propose the 3DCNN-based multi human action recognition method. Compared with existing methods, our method can simultaneously detect and recognize actions made by many persons in the same video. For video surveillance systems, our approach enables human action recognition and summarization in public areas where people simultaneously act in the scene. With the proposed representation of data, which was performed prior to the recognition phase, our model accurately recognizes most actions. The advantage of the proposed approach is that recognition and summarization are for many people acting in the scenes. For YouTube and movie videos, in most methods, the videos are classified to extract the actions in them before summarizing these actions. Compared with the results of existing methods, our results are convincing.

  


§ ACKNOWLEDGMENT


This publication was made by NPRP grant # NPRP8-140-2-065 from the Qatar National Research Fund (a member of the Qatar Foundation). The statements made herein are solely the responsibility of the authors.



00
  1  VRIGKAS, Michalis, NIKOU, Christophoros, et KAKADIARIS, Ioannis A. A review of human activity recognition methods. Frontiers in Robotics and AI, 2015, vol. 2, p. 28.
  2 ABDALLAH, Zahraa S., GABER, Mohamed Medhat, SRINIVASAN, Bala, et al. Activity recognition with evolving data streams: A review. ACM Computing Surveys (CSUR), 2018, vol. 51, no 4, p. 71.
  3 Schuldt, C., Laptev, I., and Caputo, B. (2004). “Recognizing human actions: a local SVM approach,” in Proc. International Conference on Pattern Recognition (Cambridge), 32–36
  4 Blank, M., Gorelick, L., Shechtman, E., Irani, M., and Basri, R. (2005). “Actions as space-time shapes,” in Proc. IEEE International Conference on Computer Vision (Beijing), 1395–1402.
  5 D. Weinland, E. Boyer, and R. Ronfard. Action recognition from arbitrary views using 3D exemplars. ICCV, 2007.
  6 Arjun Nagendran et al. New system performs persistent wide-area aerial surveillance 
  7 Elharrouss, Omar, Almaadeed, Noor, et AL-Maadeed, Somaya. Mhad: multi-human action dataset. In : Fourth International Congress on Information and Communication Technology. Springer, Singapore, 2020. p. 333-341.
  8 Fisher, Robert. "BEHAVE: Computer-assisted prescreening of video streams for unusual activities." The EPSRC project GR S 98146 (2007).
  9 Fisher, R. B. (2007b). PETS07 Benchmark Dataset. Available at: http://www.cvg.reading.ac.uk/PETS2007/data.html
  10 I. Laptev, M. Marszałek, C. Schmid, and B. Rozenfeld. Learning realistic human actions from movies. CVPR, 2008
  11 M. Marszałek, I. Laptev, and C. Schmid. Actions in context. CVPR, 2009.
  12 M. Rodriguez, J. Ahmed, and M. Shah. Action mach: A spatiotemporal maximum average correlation height filter for action recognition. CVPR, 2008
  13  http://server.cs.ucf.edu/vision/data.html.
  14 J. Liu, J. Luo, and M. Shah. Recognizing realistic actions from videos ”in the wild”. CVPR, 2009
  15  http://serre-lab.clps.brown.edu/resources/HMDB/
  16 KUEHNE, Hildegard, JHUANG, Hueihan, GARROTE, Estíbaliz, et al. HMDB: a large video database for human motion recognition. In : Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, 2011. p. 2556-2563.
  17 Heilbron, F. C., Escorcia, V., Ghanem, B., and Niebles, J. C. (2015). “ActivityNet: a large-scale video benchmark for human activity understanding,” in Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Boston, MA), 961–970.
  18 Elharrouss, O., Abbad, A., Moujahid, D., and Tairi,H., 2017. Moving object detection zone using a block-based background model. IET Computer Vision, 2017, vol. 12, no 1, p. 86-94.
  181 Elharrouss, O., Moujahid, D., Elkah, S. and Tairi, H., 2016. Moving object detection using a background modeling based on entropy theory and quad-tree decomposition. Journal of Electronic Imaging, 25(6), p.061615.
  19 HENRIQUES, João F., CASEIRO, Rui, MARTINS, Pedro, et al.High-speed tracking with kernelized correlation filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, vol. 37, no 3, p. 583-596.
  20 Kamiński, Ł., S. Maćkowiak, and M. Domański. Human activity recognition using standard descriptors of MPEG CDVS. in Multimedia & Expo Workshops (ICMEW), 2017 IEEE International Conference on. 2017. IEEE
  21 Dasari, R. and C. W. Chen (2018). MPEG CDVS Feature Trajectories for Action Recognition in Videos. 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), IEEE.
  22 El-Masry, M., et al. (2017). "Action recognition by discriminative EdgeBoxes." IET Computer Vision 12(4): 443-452.
  23 El-Henawy, I., et al. (2018). "Action recognition using fast HOG3D of integral videos and Smith–Waterman partial matching." IET Image Processing 12(6): 896-908.
  24 Zhang, J., et al. (2018). "Action Recognition From Arbitrary Views Using Transferable Dictionary Learning." IEEE Transactions on Image Processing 27(10): 4709.
  25 Zhang, S., et al. (2016). Human Action Recognition Based on Multifeature Fusion. Proceedings of 2016 Chinese Intelligent Systems Conference, Springer.
  26 Tejero-de-Pablos, A., et al. (2016). Human action recognition-based video summarization for RGB-D personal sports video. Multimedia and Expo (ICME), 2016 IEEE International Conference on, IEEE.
  27 Sreeraj, M. (2015). Multi-posture Human Detection Based on Hybrid HOG-BO Feature. Advances in Computing and Communications (ICACC), 2015 Fifth International Conference on, IEEE.
  28 Sharif, M., et al. (2017). "A framework of human detection and action recognition based on uniform segmentation and combination of Euclidean distance and joint entropy-based features selection." EURASIP Journal on Image and Video Processing 2017(1): 89.
  29 Qazi, H. A., et al. (2017). Human action recognition using SIFT and HOG method. Information and Communication Technologies (ICICT), 2017 International Conference on, IEEE.
  30 Liu, H., et al. (2017). Study of human action recognition based on improved spatio-temporal features. Human Motion Sensing and Recognition, Springer: 233-250.
  31 Jiang, J., et al. (2015). "Human action recognition via compressive-sensing-based dimensionality reduction." Optik-International Journal for Light and Electron Optics 126(9-10): 882-887.
  32 Cheng, S., et al. (2016). "Action recognition based on spatio-temporal log-Euclidean covariance matrix." International Journal of Signal Processing, Image Processing and Pattern Recognition 9(2): 95-106.
  33 P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie, “Behavior recognition via sparse spatio-temporal features,” in Proc. 14th Int. Conf. Comput. Commun. Netw., 2005, pp. 65–72. 
  34 H. Wang and C. Schmid, ”Action recognition with improved trajectories,” in Proc. IEEE Int. Conf. Comput. Vis., Sydney, Australia, 2013, pp. 3551–3558.
  35 L. Gao, J. Song, X. Liu, J. Shao, J. Liu, and J. Shao, “Learning in high-dimensional multimedia data: the state of the art,” Multimedia Syst., pp. 1–11, 2015.
  36 CARMONA, Josep Maria et CLIMENT, Joan. Human action recognition by means of subtensor projections and dense trajectories. Pattern Recognition, 2018, vol. 81, p. 443-455.
  37  A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and F. Li, “Large-scale video classification with convolutional neural networks,” in Proc. 2014 Comput. Vis. Pattern Recognit., 2014, pp. 1725–1732.
  38 JIN, Cheng-Bin, LI, Shengzhe, et KIM, Hakil. Real-Time Action Detection in Video Surveillance using Sub-Action Descriptor with Multi-CNN. arXiv preprint arXiv:1710.03383, 2017.
  39 L. Sun, K. Jia, D. Yeung, and B. E. Shi, “Human action recognition using factorized spatio-temporal convolutional networks,” 
  40 YANG, Hao, YUAN, Chunfeng, XING, Junliang, et al. SCNN: Sequential convolutional neural network for human action recognition in videos. In : Image Processing (ICIP), 2017 IEEE International Conference on. IEEE, 2017. p. 355-359.
  41 Simonyan, K. and Zisserman, A., 2014. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems (pp. 568-576).
  42   S. Ji, W. Xu, M. Yang, K. Yu, 3d convolutional neural networks for human action recognition, IEEE Trans. Pattern Anal. Mach. Intell. 35 (2013) 221–231.
  43 P. Wang, Y. Cao, C. Shen, L. Liu, Temporal pyramid pooling based convolutional neural network for action recognition, IEEE Trans. Circuits Syst. Video Technol.(2016).
  44 L. Wang, Y. Qiao, X. Tang, Action recognition with trajectory-pooled deep-convolutional descriptors, in: CVPR, 2015, pp. 4305–4314.
  45 WANG, Xuanhan, GAO, Lianli, SONG, Jingkuan, et al. Beyond frame-level CNN: saliency-aware 3-D CNN with LSTM for video action recognition. IEEE Signal Processing Letters, 2017, vol. 24, no 4, p. 510-514.
  46 WANG, Liangliang, GE, Lianzheng, LI, Ruifeng, et al. Three-stream CNNs for action recognition. Pattern Recognition Letters, 2017, vol. 92, p. 33-40.
  47 MA, Miao, MARTURI, Naresh, LI, Yibin, et al. Region-sequence based six-stream CNN features for general and fine-grained human action recognition in videos. Pattern Recognition, 2018, vol. 76, p. 506-521.
  48 TU, Zhigang, XIE, Wei, QIN, Qianqing, et al. Multi-stream CNN: Learning representations based on human-related regions for action recognition. Pattern Recognition, 2018, vol. 79, p. 32-43.
  49 AKULA, Aparna, SHAH, Anuj K., et GHOSH, Ripul. Deep learning approach for human action recognition in infrared images. Cognitive Systems Research, 2018, vol. 50, p. 146-154
  50 YANG, Hao, YUAN, Chunfeng, LI, Bing, et al. Asymmetric 3D Convolutional Neural Networks for action recognition. Pattern Recognition, 2019, vol. 85, p. 1-12.
  51 Tejero-de-Pablos, A., Nakashima, Y., Sato, T., Yokoya, N., Linna, M., & Rahtu, E. (2018). Summarization of user-generated sports video by using deep action recognition features. IEEE Transactions on Multimedia, 20(8), 2000-2011.
  52 Dos Santos Belo, L., et al., Summarizing video sequence using a graphbased hierarchical approach. Neurocomputing, 2016. 173: p. 1001-1016.
  53 Xu, X., T.M. Hospedales, and S. Gong, Discovery of shared semantic spaces for multiscene video query and summarization. IEEE Trans. Circ. Syst. Vid. Technol., 2017. 27(6): p. 1353-1367

  54 Elharrouss, O., Al-Maadeed, N. and Al-Maadeed, S., 2019, June. Video Summarization based on Motion Detection for Surveillance Systems. In 2019 15th International Wireless Communications & Mobile Computing Conference (IWCMC)(pp. 366-371). IEEE.
  55 Kalaivani, P. and S.M.M. Roomi. Towards Comprehensive Understanding of Event Detection and Video Summarization Approaches. in Recent Trends and Challenges in Computational Models (ICRTCCM), 2017 Second International Conference on. 2017. IEEE. 
  56 MERKOW, Jameson, MARSDEN, Alison, KRIEGMAN, David, et al. Dense volume-to-volume vascular boundary detection. In : International conference on medical image computing and computer-assisted intervention. Springer, Cham, 2016. p. 371-379.
  57  He K, Zhang X, Ren S, Sun J. (2015) Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, pp 1026–1034.


  58Jaouedi, N., Boujnah, N. and Bouhlel, M.S., 2020. A new hybrid deep learning model for human action recognition. Journal of King Saud University-Computer and Information Sciences, 32(4), pp.447-453.

  59Aslan, M.F., Durdu, A. and Sabanci, K., 2020. Human action recognition with bag of visual words using different machine learning methods and hyperparameter optimization. Neural Computing and Applications, 32(12), pp.8585-8597.

  60Antonik, P., Marsal, N., Brunner, D. and Rontani, D., 2019. Human action recognition with a large-scale brain-inspired photonic computer. Nature Machine Intelligence, 1(11), pp.530-537.

  61Rashwan, H.A., Garcia, M.A., Abdulwahab, S. and Puig, D., 2020. Action representation and recognition through temporal co-occurrence of flow fields and convolutional neural networks. Multimedia Tools and Applications, pp.1-18.

  62Avola, D., Cascio, M., Cinque, L., Foresti, G.L., Massaroni, C., Rodol`a, E.: 2d skeleton-based action recognition via two-branch stacked lstm-rnns. IEEE Transactions on Multimeaa 9999, 1–1 (2019)

  63Khan, M.A., Javed, K., Khan, S.A., Saba, T., Habib, U., Khan, J.A. and Abbasi, A.A., 2020. Human action recognition using fusion of multiview and deep features: an application to video surveillance. Multimedia Tools and Applications, pp.1-27.

  64Sharif, A., Khan, M.A., Javed, K., Gulfam, H., Iqbal, T., Saba, T., Ali, H. and Nisar, W., 2019. Intelligent human action recognition: a framework of optimal features selection based on Euclidean distance and strong correlation. Journal of Control Engineering and Applied Informatics, 21(3), pp.3-11.

  65 Elharrouss, O., Almaadeed, N., Al-Maadeed, S., Bouridane, A. and Beghdadi, A., 2020. A combined multiple action recognition and summarization for surveillance video sequences. Applied Intelligence, pp.1-23.

  66 Mliki, H., Bouhlel, F. and Hammami, M., 2020. Human activity recognition from UAV-captured video sequences. Pattern Recognition, 100, p.107140.

  67 Jalal, A., Mahmood, M. and Sidduqi, M.A., 2018, December. Robust spatio-temporal features for human interaction recognition via artificial
neural network. In IEEE Conference on FIT.
 68    Sahoo, S.P. and Ari, S., 2019. On an algorithm for human action recognition. Expert Systems with Applications, 115, pp.524-534.


  69 Liu, Y., Lu, Z., Li, J. and Yang, T., 2018. Hierarchically Learned ViewInvariant Representations for Cross-View Action Recognition. IEEE Transactions on Circuits and Systems for Video Technology
  70 Khan, M.A., Sharif, M., Akram, T., Raza, M., Saba, T. and Rehman, A., 2020. Hand-crafted and deep convolutional neural network features fusion and selection strategy: an application to intelligent human action recognition. Applied Soft Computing, 87, p.105986.
  71 Hu, Shizhe, Xiaoqiang Yan, and Yangdong Ye. "Joint Specific and Correlated Information Exploration for Multi-view Action Clustering." Information Sciences (2020).
  72Wang, Boyu, Lihan Huang, and Minh Hoai. "Active Vision for Early Recognition of Human Actions." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1081-1091. 2020.
  73 Nida, N., Yousaf, M.H., Irtaza, A. and Velastin, S.A., 2020. Deep temporal motion descriptor (DTMD) for human action recognition. Turkish Journal of Electrical Engineering & Computer Sciences, 28(3), pp.1371-1385.

  74Asghari-Esfeden, Sadjad, Mario Sznaier, and Octavia Camps. "Dynamic Motion Representation for Human Action Recognition." In The IEEE Winter Conference on Applications of Computer Vision, pp. 557-566. 2020.

  75Majd, M. and Safabakhsh, R., 2020. Correlational Convolutional LSTM for human action recognition. Neurocomputing, 396, pp.224-229.

  





