It calculates the difference between the current frame *f*(*x*, *y*, *t*) and the background frame *f*(*x*, *y*, *t* 0 ) and compares with a threshold *ξ*thr. Examples are given in **Figure 5** where BDIs are utilized for the posture level of the subaction descriptor, for example, *sitting* and *standing*.

In a motion history image, pixel intensity is a function of the temporal history of motion at that point. MHI captures the motion history patterns of the actor, denoted as *h*(*x*, *y*, *t*), and is defined using a simple replacement and a decay operator in Eqs. (4)–(6) [14].

$$\begin{aligned} \text{Find the following:}\\ \text{defined using a simple replacement and a decay operator in Eqs. (4)-(6) [14].}\\ d(\text{x}, y, t) = \begin{cases} 255, & \text{if} f(\text{x}, y, t) - f(\text{x}, y, t - 1) > \xi\_{\text{thr}} \\ 0, & \text{otherwise} \end{cases} \end{aligned} \tag{4}$$

$$\begin{aligned} \{u\_t \, \quad \text{ } \max \text{w.s.}\\\\ h(\mathbf{x}, y, t) &= \begin{cases} \tau\_{\max} & \text{if } d(\mathbf{x}, y, t) = 255 \\\\ \max \{0, h(\mathbf{x}, y, t - 1) - \Delta \mathbf{r} \} & \text{otherwise} \end{cases} \end{aligned} \tag{5}$$

= *τ*max − *τ* \_\_\_\_\_\_\_\_min

Hence, choosing a good *n* can be fairly difficult.

**Figure 7.** Examples of WAI for different subactions.

MHI is calculated from the difference between the current frame *f*(*x*, *y*, *t*) and the previous frame *f*(*x*, *y*, *t*-1) in Eq. (4). MHI is a vector image of motion, where more recently moving regions are brighter (see **Figure 6**). MHIs are used for the locomotion level of the multilevel action descriptor, which comprises *stationary*, *walking*, *running*, and *bicycling*. MHI captures the motion history cue of the subject, where more recently moving pixel regions are brighter. In Eq. (6), hyperparameter *n* is critical in defining the temporal range of an action. An MHI with a large *n* covers a long range of action history; however, it is insensitive to current actions. Similarly, MHI with a small *n* puts the focus on the recent actions and ignores past actions.

Weighted average images (WAIs) are applied at the gesture level of the multilevel action descriptor, which comprises *nothing*, *texting*, *smoking*, *phoning*, and *others*. For recognizing

*n* . (6)

http://dx.doi.org/10.5772/intechopen.76086

67

Real-Time Action Recognition Using Multi-level Action Descriptor and DNN

**Figure 5.** Examples of BDI for different subactions.

**Figure 6.** Examples of MHI for different subactions.

Real-Time Action Recognition Using Multi-level Action Descriptor and DNN http://dx.doi.org/10.5772/intechopen.76086 67

$$
\Delta \tau = \frac{\tau\_{\text{max}} - \tau\_{\text{min}}}{\mathcal{U}} \,. \tag{6}
$$

MHI is calculated from the difference between the current frame *f*(*x*, *y*, *t*) and the previous frame *f*(*x*, *y*, *t*-1) in Eq. (4). MHI is a vector image of motion, where more recently moving regions are brighter (see **Figure 6**). MHIs are used for the locomotion level of the multilevel action descriptor, which comprises *stationary*, *walking*, *running*, and *bicycling*. MHI captures the motion history cue of the subject, where more recently moving pixel regions are brighter. In Eq. (6), hyperparameter *n* is critical in defining the temporal range of an action. An MHI with a large *n* covers a long range of action history; however, it is insensitive to current actions. Similarly, MHI with a small *n* puts the focus on the recent actions and ignores past actions. Hence, choosing a good *n* can be fairly difficult.

Weighted average images (WAIs) are applied at the gesture level of the multilevel action descriptor, which comprises *nothing*, *texting*, *smoking*, *phoning*, and *others*. For recognizing

**Figure 7.** Examples of WAI for different subactions.

**Figure 6.** Examples of MHI for different subactions.

**Figure 5.** Examples of BDI for different subactions.

66 Intelligent Video Surveillance

subtle actions, the easiest way would be to use the shape or motion history of the actor. It is constructed as a linear combination of BDI and MHI, which is given by Eq. (7):

$$s(\mathbf{x}, y, t) = w\_1 \cdot b(\mathbf{x}, y, t) + w\_2 \cdot h(\mathbf{x}, y, t)\_\prime \quad \text{s.t.} \ w\_1 + w\_2 = 1. \tag{7}$$

**4. Experimental results**

**4.1. Evaluation metrics**

effectively recognize actions in surveillance videos.

**4.2. Action recognition on ICVL dataset**

mately 2 s (15 fps in the ICVL videos).

In this section, an ablation study of the appearance temporal features with the CNN-based approach is presented, and the results of the action recognition are shown with the ICVL dataset. The average processing time was computed based on the ICVL test videos. The experimental results showed that appearance-based temporal features with a multi-CNN classifier

Real-Time Action Recognition Using Multi-level Action Descriptor and DNN

http://dx.doi.org/10.5772/intechopen.76086

69

To quantify the results, we use the average precision at the frame-based *frame-AP*. The *frame-AP* was used in other approaches at the frame-based evaluation. Frame-AP: recognition is correct if the intersection-over-union (IOU) with the ground truth and detection area at that

LocomotionNet encodes sequential frames as memory capacity to represent actions. However, deciding the number of frames *n* in Eq. (6) is a highly action-dependent issue. In this work, the number of frames in the MHI was defined by performing a grid search from 5 to 50 frames with an interval of 5. **Figure 9** plots the classification accuracy (mAP) at the frame-based measurement for the subactions at the locomotion level of the multilevel action descriptor. The gray circles are drawn while training LocomotionNet from 100 to 1K iterations with an interval of 100. The circles lie over a 1.96 standard error of the mean and standard deviation in white. The baseline accuracy at *n* = 10 is given by encoding the temporal features. With *n* equal to 25 frames, LocomotionNet was able to get a performance boost from 1 to 2% of the mAP. This evidence indicates that correctly recognizing one action would need approxi-

**Table 1** shows the results of each temporal feature with CNN. An ablation study of the proposed approach at the gesture level is presented by evaluating the performance of the two appearance-based temporal features, BDI and MHI, and their combination. Frame-AP is reported for PostureNet, LocomotionNet, and GestureNet. The leading scores of each label are displayed in bold font. As in Eq. (7), WAI is the weighted average of BDI and MHI. GestureNet performed significantly better than PostureNet and LocomotionNet, showing the significance of the combined cues for the task of gesture-level subaction recognition. The GestureNet com-

**Figure 10** shows the mAP across subactions at the gesture level of the multilevel action descriptor at the frame-based measurement with regard to varying weights on WAI and training iterations of the GestureNet. In the experiment, *w*1 = 0.6 and *w*2 = 0.4 show a significant improvement beyond *w*1 = 0.5 and *w*2 = 0.5. This implies that the shape cue is more important than the motion history cue in WAI and is quite different from the results in **Table 1**. One possible explanation for this finding is that the motion history cue is more informative than the

bines the static and motion history cues to capture specific patterns of the action.

frame is greater than *σ*(*σ* = 0.5), and the action label is correctly predicted.

Here, **w** = {*w*1 , *w* 2 }T is another hyperparameter. **Figure 7** shows some examples of WAI for different subactions. WAIs were applied at the gesture level of the subaction descriptor, which comprises *nothing*, *texting*, *smoking*, *phoning*, and *others*. WAI obtained the combined cues of the shape and the motion history. *Texting* (frequently moving fingers) and *smoking* (repeated hand-to-mouth motion) were captured in WAIs.

#### **3.4. Multi-CNN action classifier**

In order to reduce the computation time, a lightweight CNN architecture is devised for real-time human action recognition, as shown in **Figure 8**. The architectures of PostureNet, LocomotionNet, and GestureNet are identical with two convolutional layers, two subsampling layers, two fully connected layers, and one softmax regression layer. However, they need to be trained based on the different training data of multilevel action descriptor. The architecture of the network is as follows: Input-Convolution-ReLUs-Max pooling-Convolution-ReLUs-Max pooling-Fully connection-Dropout-Fully connection-Dropout-Fully connection-Softmax regression. The output layer consists of the same number of units as the number of subactions at the corresponding level of the descriptor. If the computational efficiency is not critical, one could use more complicated architectures [33, 34]. In our study, Adam optimizer [35] is used with a learning rate of 1e−3 and *β*1 = 0.5 with a batch size of 256 examples and a weight decay of 5e−4. The networks are trained for 1K iterations [36].

**Figure 8.** Architecture of a CNN.

## **4. Experimental results**

In this section, an ablation study of the appearance temporal features with the CNN-based approach is presented, and the results of the action recognition are shown with the ICVL dataset. The average processing time was computed based on the ICVL test videos. The experimental results showed that appearance-based temporal features with a multi-CNN classifier effectively recognize actions in surveillance videos.

#### **4.1. Evaluation metrics**

**Figure 8.** Architecture of a CNN.

Here, **w** = {*w*1

68 Intelligent Video Surveillance

iterations [36].

, *w* 2

**3.4. Multi-CNN action classifier**

hand-to-mouth motion) were captured in WAIs.

subtle actions, the easiest way would be to use the shape or motion history of the actor. It is

*s*(*x*, *y*, *t*) = *w*1 ⋅ *b*(*x*, *y*, *t*) + *w*2 ⋅ *h*(*x*, *y*, *t*), s.t. *w*1 + *w*2 = 1. (7)

different subactions. WAIs were applied at the gesture level of the subaction descriptor, which comprises *nothing*, *texting*, *smoking*, *phoning*, and *others*. WAI obtained the combined cues of the shape and the motion history. *Texting* (frequently moving fingers) and *smoking* (repeated

In order to reduce the computation time, a lightweight CNN architecture is devised for real-time human action recognition, as shown in **Figure 8**. The architectures of PostureNet, LocomotionNet, and GestureNet are identical with two convolutional layers, two subsampling layers, two fully connected layers, and one softmax regression layer. However, they need to be trained based on the different training data of multilevel action descriptor. The architecture of the network is as follows: Input-Convolution-ReLUs-Max pooling-Convolution-ReLUs-Max pooling-Fully connection-Dropout-Fully connection-Dropout-Fully connection-Softmax regression. The output layer consists of the same number of units as the number of subactions at the corresponding level of the descriptor. If the computational efficiency is not critical, one could use more complicated architectures [33, 34]. In our study, Adam optimizer [35] is used with a learning rate of 1e−3 and *β*1 = 0.5 with a batch size of 256 examples and a weight decay of 5e−4. The networks are trained for 1K

}T is another hyperparameter. **Figure 7** shows some examples of WAI for

constructed as a linear combination of BDI and MHI, which is given by Eq. (7):

To quantify the results, we use the average precision at the frame-based *frame-AP*. The *frame-AP* was used in other approaches at the frame-based evaluation. Frame-AP: recognition is correct if the intersection-over-union (IOU) with the ground truth and detection area at that frame is greater than *σ*(*σ* = 0.5), and the action label is correctly predicted.

#### **4.2. Action recognition on ICVL dataset**

LocomotionNet encodes sequential frames as memory capacity to represent actions. However, deciding the number of frames *n* in Eq. (6) is a highly action-dependent issue. In this work, the number of frames in the MHI was defined by performing a grid search from 5 to 50 frames with an interval of 5. **Figure 9** plots the classification accuracy (mAP) at the frame-based measurement for the subactions at the locomotion level of the multilevel action descriptor. The gray circles are drawn while training LocomotionNet from 100 to 1K iterations with an interval of 100. The circles lie over a 1.96 standard error of the mean and standard deviation in white. The baseline accuracy at *n* = 10 is given by encoding the temporal features. With *n* equal to 25 frames, LocomotionNet was able to get a performance boost from 1 to 2% of the mAP. This evidence indicates that correctly recognizing one action would need approximately 2 s (15 fps in the ICVL videos).

**Table 1** shows the results of each temporal feature with CNN. An ablation study of the proposed approach at the gesture level is presented by evaluating the performance of the two appearance-based temporal features, BDI and MHI, and their combination. Frame-AP is reported for PostureNet, LocomotionNet, and GestureNet. The leading scores of each label are displayed in bold font. As in Eq. (7), WAI is the weighted average of BDI and MHI. GestureNet performed significantly better than PostureNet and LocomotionNet, showing the significance of the combined cues for the task of gesture-level subaction recognition. The GestureNet combines the static and motion history cues to capture specific patterns of the action.

**Figure 10** shows the mAP across subactions at the gesture level of the multilevel action descriptor at the frame-based measurement with regard to varying weights on WAI and training iterations of the GestureNet. In the experiment, *w*1 = 0.6 and *w*2 = 0.4 show a significant improvement beyond *w*1 = 0.5 and *w*2 = 0.5. This implies that the shape cue is more important than the motion history cue in WAI and is quite different from the results in **Table 1**. One possible explanation for this finding is that the motion history cue is more informative than the

**Figure 9.** Memory capacity in MHI for the locomotion level of the multilevel action descriptor.

**Table 1.** Results of the ablation study on the gesture level of ICVL dataset.

shape cue if they are used individually. For the remainder of the experimental results, *w*1 = 0.6 and *w*2 = 0.4 were used in WAI.

**Figure 11.** Confusion matrixes of the ICVL dataset at the frame-based measurement for the action-recognition task when

Real-Time Action Recognition Using Multi-level Action Descriptor and DNN

http://dx.doi.org/10.5772/intechopen.76086

71

using appearance-based temporal features with a multi-CNN classifier.

**Figure 12.** Examples of action localization and recognition results from the ICVL dataset.

To evaluate the effectiveness of the action-recognition model, we included the full confusion matrixes as a source of additional insight. **Figure 11** shows that the proposed approach achieved an mAP of 73.2% at the frame-based measurement. The horizontal rows are the ground truth, and the vertical columns are the predictions. Each row was normalized to a sum of 1. The proposed method was able to get most of the subaction categories correct, except for *smoking*. The results of the experiment show that a multilevel action descriptor can eliminate many misclassifications by dividing one action into many subactions that are not at the same levels.

**Figure 10.** Recognition results with regard to varying weights of WAI and training iterations on GestureNet.

Real-Time Action Recognition Using Multi-level Action Descriptor and DNN http://dx.doi.org/10.5772/intechopen.76086 71

**Figure 11.** Confusion matrixes of the ICVL dataset at the frame-based measurement for the action-recognition task when using appearance-based temporal features with a multi-CNN classifier.

**Figure 12.** Examples of action localization and recognition results from the ICVL dataset.

**Figure 10.** Recognition results with regard to varying weights of WAI and training iterations on GestureNet.

shape cue if they are used individually. For the remainder of the experimental results, *w*1 = 0.6

**Frame-AP (%) Nothing Texting Smoking Phoning mAP** PostureNet 51.3 42.0 11.2 37.9 35.6 LocomotionNet **62.4** 53.5 14.7 49.2 45.0 GestureNet 53.4 **83.3 26.7 57.9 55.3**

**Figure 9.** Memory capacity in MHI for the locomotion level of the multilevel action descriptor.

**Table 1.** Results of the ablation study on the gesture level of ICVL dataset.

To evaluate the effectiveness of the action-recognition model, we included the full confusion matrixes as a source of additional insight. **Figure 11** shows that the proposed approach achieved an mAP of 73.2% at the frame-based measurement. The horizontal rows are the ground truth, and the vertical columns are the predictions. Each row was normalized to a sum of 1. The proposed method was able to get most of the subaction categories correct, except for *smoking*. The results of the experiment show that a multilevel action descriptor can eliminate many misclassifications by dividing one action into many subactions that are not at the same levels.

and *w*2 = 0.4 were used in WAI.

70 Intelligent Video Surveillance

**References**

DOI: 10.1016/j.patrec.2017.08.015

28–July 01, 2015. pp. 622-627

IEEE; December 3-6, 2013. pp. 3551-3558

1345-1355. DOI: 1016/j.cviu.2012.09.008

DOI: 10.1109/CVPR.1999.784652

2013;**35**(12):2821-2840. DOI: 10.1109/TPAMI.2012.241

frobt.2015.00028

[1] Yamin H, Peng Z, Zhuo T, et al. Going deeper with two-stream ConvNets for action recognition in video surveillance. Pattern Recognition Letters. 2017 (available online).

Real-Time Action Recognition Using Multi-level Action Descriptor and DNN

http://dx.doi.org/10.5772/intechopen.76086

73

[2] Andreas S, Rainer S. Pedestrian intention recognition using latent-dynamic conditional random fields. In: Intelligent Vehicles Symposium (IV). Seoul, South Korea: IEEE; June

[3] Michalis V, Christophoros N, Loannis K. A review of human activity recognition methods. Frontiers in Robotics and Artificial Intelligence. 2015;**2**(28):1-28. DOI: 10.3389/

[4] Wang H, Klaser A, Schmid C, et al. Action recognition by dense trajectories. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

[5] Wang H, Schmid C. Action recognition with improved trajectories. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV '13). Sydney, Australia:

[6] Jiang Z, Lin Z, Davis L. A unified tree-based framework for joint action localization, recognition and segmentation. Computer Vision and Image Understanding. 2013;**117**(10):

[7] Shotton J, Girshick R, Fitzgibbon A, et al. Efficient human pose estimation from single depth images. IEEE Transitions on Pattern Analysis and Machine Intelligence.

[8] Siddiqui M, Medioni G. Human pose estimation from a single view point, real-time range sensor. In: Proceedings of the IEEE Conference on Computer Vision and Pattern

[9] Plagemann C, Ganapathi V, Koller D, et al. Real-time identification and localization of body parts from depth images. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA '10). Anchorage, USA: IEEE; May 3-7, 2010. pp. 3108-3113

[10] Cutler R, Davis L. Robust real-time periodic motion detection, analysis, and applications. IEEE Transitions on Pattern Analysis and Machine Intelligence. 2000;**22**(8):781-796.

[11] Zhang B, Wang L, Wang Z, et al. Real-time action recognition with enhanced motion vector CNNs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '16). Las Vegas, USA: IEEE; June 26–July 1, 2016. pp. 2718-2726

[12] Jin C, Li S, Do T, et al. Real-time human action recognition using CNN over temporal images for static video surveillance cameras. Lecture Notes in Computer Science

Recognition (CVPR '11). San Francisco, USA: IEEE; June 13-18, 2011. pp. 1-8

(CVPR '11). Colorado Springs, USA: IEEE; June 20-25, 2011. pp. 3169-3176

**Table 2.** Average processing time of the proposed action detection model.

**Figure 12** shows qualitative localization and recognition results using the proposed approach on the test set of the ICVL dataset. Each block corresponds to a video from a different camera. Two frames are shown from each video. The test platform has a PC with an Intel Core i7-4770 CPU at 3.49 GHz with 32 GB memory. The input video was resized to 640 × 480, and the processing time was tested on 72 videos shown in **Table 2**.

## **5. Conclusions**

This work introduced a new approach to real-time action recognition using multilevel action descriptor in video surveillance system. Experimental results demonstrated that a multilevel action descriptor delivers a complete set of information about human actions and significantly eliminates misclassifications by a large number of actions that are built by few independent subactions at different levels. An ablation study showed the effect of each temporal feature when considered separately. Shape and motion history cues are complementary, and the combination of both leads to a significant improvement in action recognition performance. In addition, the proposed action recognition model simultaneously localizes and recognizes the actions of multiple individuals at low computational cost with acceptable accuracy. The model ran at around 25 fps in 640 × 480 frame size, which is suitable for real-time surveillance applications. In future work, we will extend the approach to learn deep motion flow from original frame sequences and combine detecting and recognizing in one network for becoming an end-to-end human action detection framework.

## **Acknowledgements**

This work was supported by the Industrial Technology Innovation Program, "10052982, Development of multiangle front camera system for intersection AEB," funded by the Ministry of Trade, Industry, & Energy (MOTIE, Korea).

## **Author details**

Cheng-Bin Jin, Trung Dung Do, Mingjie Liu and Hakil Kim\*

\*Address all correspondence to<EMAIL_ADDRESS>
Department of Information and Communication Engineering, Inha University, Incheon, South Korea

## **References**

**Figure 12** shows qualitative localization and recognition results using the proposed approach on the test set of the ICVL dataset. Each block corresponds to a video from a different camera. Two frames are shown from each video. The test platform has a PC with an Intel Core i7-4770 CPU at 3.49 GHz with 32 GB memory. The input video was resized to 640 × 480, and the

**Module Motion Detection Tracking BDI MHI WAI CNNs Others Overall** Processing time (ms) 11.33 11.60 0.28 0.26 0.83 0.12 4.66 12.16 42.93

This work introduced a new approach to real-time action recognition using multilevel action descriptor in video surveillance system. Experimental results demonstrated that a multilevel action descriptor delivers a complete set of information about human actions and significantly eliminates misclassifications by a large number of actions that are built by few independent subactions at different levels. An ablation study showed the effect of each temporal feature when considered separately. Shape and motion history cues are complementary, and the combination of both leads to a significant improvement in action recognition performance. In addition, the proposed action recognition model simultaneously localizes and recognizes the actions of multiple individuals at low computational cost with acceptable accuracy. The model ran at around 25 fps in 640 × 480 frame size, which is suitable for real-time surveillance applications. In future work, we will extend the approach to learn deep motion flow from original frame sequences and combine detecting and recognizing in one network for becom-

This work was supported by the Industrial Technology Innovation Program, "10052982, Development of multiangle front camera system for intersection AEB," funded by the

Department of Information and Communication Engineering, Inha University, Incheon,

processing time was tested on 72 videos shown in **Table 2**.

**Table 2.** Average processing time of the proposed action detection model.

ing an end-to-end human action detection framework.

Ministry of Trade, Industry, & Energy (MOTIE, Korea).

Cheng-Bin Jin, Trung Dung Do, Mingjie Liu and Hakil Kim\*

\*Address all correspondence to<EMAIL_ADDRESS>
**5. Conclusions**

72 Intelligent Video Surveillance

**Acknowledgements**

**Author details**

South Korea

(Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2015;**9315**:330-339. DOI: 10.1007/978-3-319-24078-7\_33

[25] Lan Z, Ming L, Xuanchong L, et al. Beyond Gaussian pyramid: Multi-skip feature stacking for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '15). Boston, USA: IEEE; June 7-10, 2015. pp. 204-212 [26] Ni B, Yang X, Gao S. Progressively parsing interactional objects for fine grained action detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '16). Las Vegas, USA: IEEE; June 26–July 1, 2016. pp. 1020-1028 [27] Yeung S, Russakovsky O, Moi G, et al. End-to-end learning of action detection from frame glimpses in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '16). Las Vegas, USA: IEEE; June 26–July 1, 2016.

Real-Time Action Recognition Using Multi-level Action Descriptor and DNN

http://dx.doi.org/10.5772/intechopen.76086

75

[28] Karpathy A, Toderici G, Shetty S, et al. Large-scale video classification with convolutional neural networks, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '14). Columbus, USA: IEEE; June 24-27, 2014. pp. 1725-1732

[29] Gkioxari G, Malik J. Finding action tubes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '15). Boston, USA: IEEE; June 7-10,

[30] Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos. In: Proceedings of the Advances in Neural Information Processing Systems

[31] Ji S, Xu W, Yang M, et al. 3D convolutional neural networks for human action recognition. IEEE Transitions on Pattern Analysis and Machine Intelligence. 2013;**35**(1):221-331.

[32] Li S. Human re-identification using soft biometrics in video surveillance [thesis].

[33] Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. In: Proceedings of the Advances in Neural Information Processing

[34] Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '14). Columbus, USA: IEEE; June 24-27, 2014.

[35] Kingma P, Ba J. Adam: A method for stochastic optimization. 2017. pp. 1-15. arXiv

[36] Jin C-B, Do T, Liu M, et al. Real-time action detection in video surveillance using a sub-action descriptor with multi-convolutional neural networks. Journal of Institute of Control, Robotics and Systems. 2018;**24**(3):298-308. DOI: 10.5302/J.ICROS.2018.17.0243

Systems (NIPS '12). Lake Tahoe, USA: NIPS; December 03-08, 2012. pp. 1-9

(NIPS '14). Montreal, Canada: NIPS; December 08-13, 2014. pp. 568-576

pp. 2678-2687

2015. pp. 759-768

pp. 580-587

Preprint:1412.6980v9

DOI: 10.1109/TPAMI.2012.59

Incheon: Inha University; 2015

[25] Lan Z, Ming L, Xuanchong L, et al. Beyond Gaussian pyramid: Multi-skip feature stacking for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '15). Boston, USA: IEEE; June 7-10, 2015. pp. 204-212

(Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in

[13] Dalal N, Triggs B. Histograms of oriented gradients for human detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '05). San

[14] Yu C, Joachims T. Learning structural SVMs with latent variables. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV '09). Kyoto, Japan: IEEE;

[15] Bobick AF, Davis JW. The recognition of human movement using temporal templates. IEEE Transitions on Pattern Analysis and Machine Intelligence. 2001;**23**(3):257-267. DOI:

[16] Davis JW, Bobick AF. The representation and recognition of human movement using temporal templates. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '97). San Juan, Puerto Rico: IEEE; June 17-19, 1997. pp. 928-934

[17] Ali S, Basharat A, Shah M. Chaotic invariants for human action recognition. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV '07). Rio

[18] Fathi A, Mori G.Action recognition by learning mid-level motion feature. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '08).

[19] Laptev I. On space-time interest points. International Journal of Computer Vision.

[20] Kim I, Oh S, Vahdat A, et al. Segmental multi-way local pooling for video recognition. In: Proceeding of the ACM International Conference on Multimedia (ICM '13). New York,

[21] Klaser A, Marszalek M, Schmid C. A spatio-temporal descriptor based on 3D-gradients. In: Proceedings of the British Machine Conference (BMC '08). Leeds, UK: Inria;

[22] Li Y, Ye Z, Rehg JM. Delving into egocentric actions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '15). Boston, USA:

[23] Vahdat A, Gao B, Ranjbar M, et al. A discriminative key pose sequence model for recognizing human interactions. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV '11). Barcelona, Spain: IEEE; November 6-13, 2011. pp. 1729-1736

[24] Lan T, Wang Y, Yang W, et al. Discriminative latent models for recognizing contextual group activities. IEEE Transitions on Pattern Analysis and Machine Intelligence.

Bioinformatics). 2015;**9315**:330-339. DOI: 10.1007/978-3-319-24078-7\_33

Diego, USA: IEEE; June 20-26, 2005. pp. 886-893

September 29–October 2, 2009. pp. 1169-1176

de Janeiro, Brazil: IEEE; October 14-20, 2007. pp. 1-8

Anchorage, USA: IEEE; June 24-26, 2008. pp. 1-8

2005;**64**:107-123. DOI: 10.1007/s11263-005-1838-7

2012;**34**(8):1549-1562. DOI: 10.1109/TPAMI.2011.228

USA: ACM; October 21-25, 2013. pp. 37-640

September 3, 2008. pp. 275:1-10

IEEE; June 7-10, 2015. pp. 287-295

10.1109/34.910878

74 Intelligent Video Surveillance

**Chapter 5**

Provisional chapter

**Human Activity Recognition without Vision Tracking**

DOI: 10.5772/intechopen.82660

This work describes the recognition of human activity based on the interaction between people and objects in domestic settings, specifically in a kitchen. The difference between this and other proposals is that considers a human activity in a process without vision tracking. Videos are a sequence of photographs. Taking this into account, if you analyze an orderly sequence of images it could be based on the objects present in each scene so that you can understand the possible activity performed. However, it is not enough to consider the objects present in the scene; it is necessary to determine if those objects are employed or not by the humans present. If they are used, it is evident that they are necessary to carry out the activity; if they are not used they would only provide noise to the recognized activity. Therefore, it is necessary to generate a conceptualization of objects in the scene with characteristics (definition of an object, motion detector, object recognition, object position, object action) that allows you to recognize them and to determine the degree of use (unchanged, added, removed, moved, and indeterminate) and influence the

Keywords: human activity, computer vision, human/computer interaction, human/robot interaction, feature extraction, behavior representation

This study is part of a major research project called InHands [1, 2]. The approach specifically analyzes the recognition of human activity without using the traditional proposal that uses the

> © 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and eproduction in any medium, provided the original work is properly cited.

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

Human Activity Recognition without Vision Tracking

Carlos Alberto Flores Vázquez, Joan Aranda,

Carlos Alberto Flores Vázquez, Joan Aranda,

Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

Daniel Icaza, Santiago Pulla, Marcelo Flores-Vázquez and Nelson Federico Cordova

Daniel Icaza, Santiago Pulla, Marcelo Flores-Vázquez and Nelson Federico Cordova

http://dx.doi.org/10.5772/intechopen.82660

possible recognized activity.

Abstract

1. Introduction

#### **Human Activity Recognition without Vision Tracking** Human Activity Recognition without Vision Tracking

DOI: 10.5772/intechopen.82660

Carlos Alberto Flores Vázquez, Joan Aranda, Daniel Icaza, Santiago Pulla, Marcelo Flores-Vázquez and Nelson Federico Cordova Carlos Alberto Flores Vázquez, Joan Aranda, Daniel Icaza, Santiago Pulla, Marcelo Flores-Vázquez and Nelson Federico Cordova

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.82660

#### Abstract

This work describes the recognition of human activity based on the interaction between people and objects in domestic settings, specifically in a kitchen. The difference between this and other proposals is that considers a human activity in a process without vision tracking. Videos are a sequence of photographs. Taking this into account, if you analyze an orderly sequence of images it could be based on the objects present in each scene so that you can understand the possible activity performed. However, it is not enough to consider the objects present in the scene; it is necessary to determine if those objects are employed or not by the humans present. If they are used, it is evident that they are necessary to carry out the activity; if they are not used they would only provide noise to the recognized activity. Therefore, it is necessary to generate a conceptualization of objects in the scene with characteristics (definition of an object, motion detector, object recognition, object position, object action) that allows you to recognize them and to determine the degree of use (unchanged, added, removed, moved, and indeterminate) and influence the possible recognized activity.

Keywords: human activity, computer vision, human/computer interaction, human/robot interaction, feature extraction, behavior representation

#### 1. Introduction

This study is part of a major research project called InHands [1, 2]. The approach specifically analyzes the recognition of human activity without using the traditional proposal that uses the

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and eproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

follow-up of movements of hands and arms. A short version of this research is shown in [3] and this work presents several improvements to the initial proposal presented in [2].

A similar work where the scenario is to employ a Kinect camera is [9]. However, to increase the reliability of this methodology of recognition radio-frequency identification tags are added,

Human Activity Recognition without Vision Tracking http://dx.doi.org/10.5772/intechopen.82660 79

The first information this system considers is hand and object tracking, and later object and action recognition. Regarding the detail of the recognized actions, seven principals are defined:

An example of the results obtained is the preparation of a cake, for which seven objects, 17 actions, about 6000 frames, and approximately 200 seconds are used. For us the definition of

In this research, we consider that the results of activity recognition would be useful to provide proactive assistance. Therefore, the recognition of the activity should be determined while the activity is being carried out and with the aim of facilitating the robotic assistance considered in

Recognition is approached with computer vision methods. In a specific way, it is based on the recognition of objects in the scene and the interaction with these objects based on their manipulation. However, it is necessary to detail that the interaction with the objects does not contemplate the tracking of the objects, arms, or hands of the person who intervenes in the action. The initial and final positions of the objects, their presence or not, are of importance in

In this proposal the conceptualization of an object is explained in five constitutive parts:

Considering the relevance of the objects and interaction with these, it is necessary to develop a definition of the object with parameters that allow the differentiation between them. Therefore,

1. Identification number: this allows us to have a unique number for each object despite having

actions is simpler. Therefore, our actions will be those explained in Section 2.1.5.

which will not be implemented in our proposal.

2. Methodology

the recognition process.

• Definition of an object

2.1.1. Definition of an object

the chosen parameters are:

similar characteristics such as color.

• Motion detector • Object recognition

• Object position • Object action

later stages of the InHands project.

2.1. Conceptualization of an object

place; move; chop; mixing; pouring; spooning; and scooping [9].

To introduce the field of recognition of human activity is of interest in this research and the definition mentioned in [4] on human/object interactions is as follows:

"The most typical human-object interaction recognition approaches are the ones ignoring interplays between object recognition and motion estimation. In those works, objects are generally recognized first, and activities involving them are recognized by analyzing the objects' motion. They have made the object recognition and motion estimation independent or made it so that the motion estimation is strictly dependent on the object recognition" [2, 4].

In addition to the definition quoted here, this chapter defines the structure and classification of human activity recognition, from which we extract the following:

	- Statistical: construct statistical state-based models concatenated hierarchically [2, 4].
	- Syntactic: use a grammar syntax such as stochastic context-free grammar to model sequential activities [2, 4].
	- Description-based: represent human activities by describing subevents of the activities and their temporal, spatial, and logical structures [2, 4].

This research applies the approach of human/object interactions, and included in these are more specific subtopics: syntactic and description-based.

In [5] a proposal for the recognition of activity based on description-based is presented. This methodology consists of motion detection and tracking complemented by event analysis, which is of interest for the detection of movement. On the basis of this the capture of images for our proposal is carried out. As in [5] we use a single camera for capture and for segmentation the background is extracted, but we use image difference.

Among the relevant works to establish an activity recognition procedure [6], each action event is assigned a symbol and then a sequence of actions corresponds to a string of symbols. In our proposal we will use words instead of symbols, and a set of words regardless of their order could form an activity.

A similar way to [6] is the proposal of [7]: a BOW (bag of words). A BOW considers that an image might be similar to a paragraph where repetition of one or more words would allow us to recognize the content or essence of the text. For us it is similar to considering the repetition of objects in the image, and to interact with them would give us indications of the activity that is developing.

For the InHands project, proactive assistance is very important, and with this premise in [8] they present a probabilistic prediction of the actions carried out, which is precisely what is intended to implement our proposal.

A similar work where the scenario is to employ a Kinect camera is [9]. However, to increase the reliability of this methodology of recognition radio-frequency identification tags are added, which will not be implemented in our proposal.

The first information this system considers is hand and object tracking, and later object and action recognition. Regarding the detail of the recognized actions, seven principals are defined: place; move; chop; mixing; pouring; spooning; and scooping [9].

An example of the results obtained is the preparation of a cake, for which seven objects, 17 actions, about 6000 frames, and approximately 200 seconds are used. For us the definition of actions is simpler. Therefore, our actions will be those explained in Section 2.1.5.

## 2. Methodology

follow-up of movements of hands and arms. A short version of this research is shown in [3]

To introduce the field of recognition of human activity is of interest in this research and the

"The most typical human-object interaction recognition approaches are the ones ignoring interplays between object recognition and motion estimation. In those works, objects are generally recognized first, and activities involving them are recognized by analyzing the objects' motion. They have made the object recognition and motion estimation independent or made it so that the motion estimation is strictly dependent on the object recognition" [2, 4]. In addition to the definition quoted here, this chapter defines the structure and classification of

• Single-layered approaches are appropriate for gesture and action recognition by sequen-

• Hierarchical approaches are human activity representations with a high level of abstraction. Within this topic we find a subclassification that is of interest to us [2, 4]:

• Statistical: construct statistical state-based models concatenated hierarchically [2, 4]. • Syntactic: use a grammar syntax such as stochastic context-free grammar to model

• Description-based: represent human activities by describing subevents of the activi-

This research applies the approach of human/object interactions, and included in these are

In [5] a proposal for the recognition of activity based on description-based is presented. This methodology consists of motion detection and tracking complemented by event analysis, which is of interest for the detection of movement. On the basis of this the capture of images for our proposal is carried out. As in [5] we use a single camera for capture and for segmenta-

Among the relevant works to establish an activity recognition procedure [6], each action event is assigned a symbol and then a sequence of actions corresponds to a string of symbols. In our proposal we will use words instead of symbols, and a set of words regardless of their order

A similar way to [6] is the proposal of [7]: a BOW (bag of words). A BOW considers that an image might be similar to a paragraph where repetition of one or more words would allow us to recognize the content or essence of the text. For us it is similar to considering the repetition of objects in the image, and to interact with them would give us indications of the activity that is developing.

For the InHands project, proactive assistance is very important, and with this premise in [8] they present a probabilistic prediction of the actions carried out, which is precisely what is

ties and their temporal, spatial, and logical structures [2, 4].

and this work presents several improvements to the initial proposal presented in [2].

definition mentioned in [4] on human/object interactions is as follows:

human activity recognition, from which we extract the following:

tial characteristics [2, 4].

78 Intelligent Video Surveillance

could form an activity.

intended to implement our proposal.

sequential activities [2, 4].

more specific subtopics: syntactic and description-based.

tion the background is extracted, but we use image difference.

In this research, we consider that the results of activity recognition would be useful to provide proactive assistance. Therefore, the recognition of the activity should be determined while the activity is being carried out and with the aim of facilitating the robotic assistance considered in later stages of the InHands project.

Recognition is approached with computer vision methods. In a specific way, it is based on the recognition of objects in the scene and the interaction with these objects based on their manipulation. However, it is necessary to detail that the interaction with the objects does not contemplate the tracking of the objects, arms, or hands of the person who intervenes in the action. The initial and final positions of the objects, their presence or not, are of importance in the recognition process.

#### 2.1. Conceptualization of an object

In this proposal the conceptualization of an object is explained in five constitutive parts:

#### 2.1.1. Definition of an object

Considering the relevance of the objects and interaction with these, it is necessary to develop a definition of the object with parameters that allow the differentiation between them. Therefore, the chosen parameters are:

1. Identification number: this allows us to have a unique number for each object despite having similar characteristics such as color.

Figure 1(a) graphically shows the conception of an object in this proposal. Figure 1(b) shows the execution of the algorithm in parallel to generate the attributes of each object.

#### 2.1.2. Motion detector

Motion detection is important so that we know that an activity is taking place in the scene. In addition, objects in the scene (OBJECT RECOGNITION) and their position (OBJECT POSITION) are acquired. Since we do not track, is important to recapture the objects in the scene after the movements made by the person present. By making this subsequent capture we avoid occlusions and we can determine an action (OBJECT ACTION) for each object when checking through their presence and position if they were moved, removed, added, or not used in the last action.

As demonstrated in the flow chart of Figure 2, one of the most important methods applied in this part of the system is image difference, specifically "the mixture of the Gaussian method" according to [10]. Figure 2(a) explains the flow chart for motion detection and Figure 2(b) shows the three results after executing the motion detection algorithm.

This image difference allows us to extract the objects from the background, which is dynamically updated while the system is working. The motion detection algorithm performs a continuous comparison of frames by setting a minimum threshold level to consider whether that variation between a frame at t = 0 and the following at t + 1, t + 2, and t + 3 implies an action performed or is only a visual noise.

2.1.3. Object recognition

mask is the region of interest (ROI).

restricted by thresholds the colors black and white.

Figure 2. Flow charts: (a) motion detector, (b) outputs from motion detector.

The object recognition process is developed in Figure 3. The first step is to get the mask by difference of images and apply it to the original image. The result of the application of the

Human Activity Recognition without Vision Tracking http://dx.doi.org/10.5772/intechopen.82660 81

From each ROI we obtain a histogram of 10 BINS RG chromaticity space (two-dimensional color space in which there is no color intensity information). Working with RG chromaticity allows us to avoid the problems of brightness, additionally normalized the images in RGB color space (a pixel is identified by the intensity of red, green and blue values) and was

A comparison of the new histograms obtained from the ROI was made against our database. The selected comparison method was the Bhattacharyya distance. As evidenced in Table 1 this method was selected considering the percentage of errors reduced and low amount of time.

Figure 1. (a) Graphical model of our definition of an object. (b) General flow chart of the definition of an object.

Figure 2. Flow charts: (a) motion detector, (b) outputs from motion detector.

#### 2.1.3. Object recognition

2. Color: this is the main characteristic that allows us to recognize the type of object present in

3. Position: this is confirmed by the coordinates of the centroid for each object, taking as

4. Actions with objects: four basic interactions have been defined with the objects by the user:

Figure 1(a) graphically shows the conception of an object in this proposal. Figure 1(b) shows

Motion detection is important so that we know that an activity is taking place in the scene. In addition, objects in the scene (OBJECT RECOGNITION) and their position (OBJECT POSITION) are acquired. Since we do not track, is important to recapture the objects in the scene after the movements made by the person present. By making this subsequent capture we avoid occlusions and we can determine an action (OBJECT ACTION) for each object when checking through their presence and position if they were moved, removed, added, or not used in the last action. As demonstrated in the flow chart of Figure 2, one of the most important methods applied in this part of the system is image difference, specifically "the mixture of the Gaussian method" according to [10]. Figure 2(a) explains the flow chart for motion detection and Figure 2(b)

This image difference allows us to extract the objects from the background, which is dynamically updated while the system is working. The motion detection algorithm performs a continuous comparison of frames by setting a minimum threshold level to consider whether that variation between a frame at t = 0 and the following at t + 1, t + 2, and t + 3 implies an action

Figure 1. (a) Graphical model of our definition of an object. (b) General flow chart of the definition of an object.

the execution of the algorithm in parallel to generate the attributes of each object.

shows the three results after executing the motion detection algorithm.

origin the lower left corner of the kitchen counter.

add, remove, move, unchanged.

performed or is only a visual noise.

the scene.

80 Intelligent Video Surveillance

2.1.2. Motion detector

The object recognition process is developed in Figure 3. The first step is to get the mask by difference of images and apply it to the original image. The result of the application of the mask is the region of interest (ROI).

From each ROI we obtain a histogram of 10 BINS RG chromaticity space (two-dimensional color space in which there is no color intensity information). Working with RG chromaticity allows us to avoid the problems of brightness, additionally normalized the images in RGB color space (a pixel is identified by the intensity of red, green and blue values) and was restricted by thresholds the colors black and white.

A comparison of the new histograms obtained from the ROI was made against our database. The selected comparison method was the Bhattacharyya distance. As evidenced in Table 1 this method was selected considering the percentage of errors reduced and low amount of time.

Figure 3. Flow chart for object recognition.

of latency. However, the YOLO method is not convenient for localization errors, for example Fast R-CNN has 8.6% of localization errors versus 19% for the YOLO method. This is explained in

Coffee Predicted label Measure Result

Known label Positive 22 4 Recall/sensitivity 84.62%

Glass Predicted label Measure Result

Known label Positive 24 5 Recall/sensitivity 82.76%

Spoon Predicted label Measure Result

Known label Positive 23 2 Recall/sensitivity 92.00%

Positive Negative Precision 91.67%

Positive Negative Precision 100.00%

Positive Negative Precision 100.00%

Accuracy 84.62%

83

Human Activity Recognition without Vision Tracking http://dx.doi.org/10.5772/intechopen.82660

Accuracy 82.76%

Accuracy 92.00%

Negative 2 2 Specificity 50.00%

Negative 0 1 Specificity 100.00%

Negative 0 5 Specificity 100.00%

Other alternatives taking to account reduced time can be [13, 14]. Tensor flow is a flexible system and can be used to express a wide variety of algorithms. For this proposal, the main advantage would be the capacity for distributing the process in many computational devices

In this proposal the position of the object is one of the essential characteristics; this allows us to define the actions resulting from the interaction with it. If the position does not change between images it means that the object was not used (UNCHANGED); if on the other hand

As for the technical details the centroid is a pixel in the image; this pixel is positioned in the coordinate system of the image. Taking into account that the InHands project requires obtaining world coordinates to assist with robots, then these coordinates must be referenced

Homography matrix H is used according to [15]. It is necessary to apply matrix H; the intrinsic and extrinsic matrixes of the calibration of the camera are explained in [16]. The result of homography is expressed in millimeters and the final adjustment is made with rotation and

Section 2.1.4; localization is an important part of this proposal for object definition.

Table 2. Confusion matrix and measurements: (a) coffee, (b) glass, (c) spoon.

it changes it means that the object is necessary in the developed activity (MOVE).

for object recognition.

2.1.4. Object position

to the kitchen counter.

translation matrixes (Figure 4).

Table 1. Comparison of methods.

The database of the objects has five different perspectives for each one. As far as the classifier was concerned, Knn (nearest neighbors) was chosen with K = 1.

Evaluation of the aforementioned classifier was performed in a similar way to the proposal in [11]. In detail, a confusion matrix is employed to measure the recognition of each of the objects. Forty images were used for validation and 30 for each test of a total of 1870 images. Some examples are shown in Table 2.

Probably a better alternative to reduce time in object recognition is [12]. The YOLO Detection System is a really fast detector that can process streaming video in real time with less than 25 ms

Table 2. Confusion matrix and measurements: (a) coffee, (b) glass, (c) spoon.

of latency. However, the YOLO method is not convenient for localization errors, for example Fast R-CNN has 8.6% of localization errors versus 19% for the YOLO method. This is explained in Section 2.1.4; localization is an important part of this proposal for object definition.

Other alternatives taking to account reduced time can be [13, 14]. Tensor flow is a flexible system and can be used to express a wide variety of algorithms. For this proposal, the main advantage would be the capacity for distributing the process in many computational devices for object recognition.

#### 2.1.4. Object position

The database of the objects has five different perspectives for each one. As far as the classifier

[1.0 …–1.0] [0.0…2.0] [1.0…0.0] [0.0…1.0]

Quick—and—dirty matching

Moderately fast—more accurate matches

Method Correlation Chi-square Intersection Bhattacharyya distance

Samples 100 100 100 100 Error 8% 8% 2% 2%

accurate matches

Evaluation of the aforementioned classifier was performed in a similar way to the proposal in [11]. In detail, a confusion matrix is employed to measure the recognition of each of the objects. Forty images were used for validation and 30 for each test of a total of 1870 images. Some

Probably a better alternative to reduce time in object recognition is [12]. The YOLO Detection System is a really fast detector that can process streaming video in real time with less than 25 ms

was concerned, Knn (nearest neighbors) was chosen with K = 1.

examples are shown in Table 2.

Table 1. Comparison of methods.

Figure 3. Flow chart for object recognition.

82 Intelligent Video Surveillance

Range [exact … mismatch]

Observations [5] Quick Moderately fast—more

In this proposal the position of the object is one of the essential characteristics; this allows us to define the actions resulting from the interaction with it. If the position does not change between images it means that the object was not used (UNCHANGED); if on the other hand it changes it means that the object is necessary in the developed activity (MOVE).

As for the technical details the centroid is a pixel in the image; this pixel is positioned in the coordinate system of the image. Taking into account that the InHands project requires obtaining world coordinates to assist with robots, then these coordinates must be referenced to the kitchen counter.

Homography matrix H is used according to [15]. It is necessary to apply matrix H; the intrinsic and extrinsic matrixes of the calibration of the camera are explained in [16]. The result of homography is expressed in millimeters and the final adjustment is made with rotation and translation matrixes (Figure 4).

1. UNCHANGED: The object was present in the previous activity but it has not been moved, which implies that the object is present in the scene but it was not used in the activity. 2. MOVE: This indicates that the object was present in a previous scene and now changes

Human Activity Recognition without Vision Tracking http://dx.doi.org/10.5772/intechopen.82660 85

3. ADD: The object was not present in previous scene, and because it is now added to the

4. REMOVE: An object present in previous scenes is no longer present. This induces the

To avoid detecting false movements caused by occlusions or errors in the calculation of the centroid a tolerance range was established; only movements greater than 5 mm are recorded.

The taxonomy proposed in [4] allows us to illustrate an approximation to the approach that this research has. In detailed form it is typecast in hierarchical approaches and has many coincidences with the vision presented in syntactic and description-based. Specifically, the approach of human/object interactions uses a syntax to define human activity but it is not

An approach whereby it is important to make a comparison of the methodology used is [7]. BOW collects features by assigning the nearest word and the frequency of occurrence of this in the images. In our proposal, each object could be a word and the repetition of these words will be relevant to determine the activity, but unlike [7] it is not necessary to consider the sequence

How do you define an activity? This is one of the questions that arose during the project; in this case a recipe inspires it, so we will use ingredients, kitchen tools, and possible substitutes to

position implying that it was used in the developing activity.

scene it is assumed to be necessary for the developing activity.

necessary to consider order, sequences, and logical structure (Figure 6).

2.2. Human activity recognition

of occurrence of words.

2.2.1. Definition of an activity

define an activity, see Figure 7(a).

Figure 6. The hierarchical approach-based taxonomy [4].

thought that it is now not necessary for the activity that is developing.

Figure 4. (a) Flow chart of object position. (b) View of a planar object as described by homography [15].

#### 2.1.5. Object action

To build the object with the previously obtained characteristics (ID number, color, centroid), in "Action" we assign the state of "UNDETERMINED," see Figure 5(a). The human/object interaction is defined in the feature of the object called Action. This can take four possible options:

Figure 5. (a) Flow charts of object builder. (b) Flow charts of object action.

To avoid detecting false movements caused by occlusions or errors in the calculation of the centroid a tolerance range was established; only movements greater than 5 mm are recorded.

#### 2.2. Human activity recognition

2.1.5. Object action

84 Intelligent Video Surveillance

To build the object with the previously obtained characteristics (ID number, color, centroid), in "Action" we assign the state of "UNDETERMINED," see Figure 5(a). The human/object interaction is defined in the feature of the object called Action. This can take four possible options:

Figure 4. (a) Flow chart of object position. (b) View of a planar object as described by homography [15].

Figure 5. (a) Flow charts of object builder. (b) Flow charts of object action.

The taxonomy proposed in [4] allows us to illustrate an approximation to the approach that this research has. In detailed form it is typecast in hierarchical approaches and has many coincidences with the vision presented in syntactic and description-based. Specifically, the approach of human/object interactions uses a syntax to define human activity but it is not necessary to consider order, sequences, and logical structure (Figure 6).

Figure 6. The hierarchical approach-based taxonomy [4].

An approach whereby it is important to make a comparison of the methodology used is [7]. BOW collects features by assigning the nearest word and the frequency of occurrence of this in the images. In our proposal, each object could be a word and the repetition of these words will be relevant to determine the activity, but unlike [7] it is not necessary to consider the sequence of occurrence of words.

#### 2.2.1. Definition of an activity

How do you define an activity? This is one of the questions that arose during the project; in this case a recipe inspires it, so we will use ingredients, kitchen tools, and possible substitutes to define an activity, see Figure 7(a).

The second evaluation corresponds to calculating the contribution of each ingredient, utensil, and substitute that has been added to the scene. Similar to the first evaluation, the contribution of ingredients, utensils, and substitutes is considered to be weighted by the constants a, b, and c, respectively. The result is the value that these objects provides for each of the probable

The third evaluation is exactly the same in its procedure as the two previously made, with the only difference being that the objects that intervene here are the ones that have remained in the

It is important to emphasize that an object could be a utensil or a substitute depending on the activity, for example a glass would be a utensil if the activity is to prepare juice (activity 1) but

þ b �

þ b �

þ b �

The fourth evaluation corresponds to the addition of the activity lists resulting from Eqs. (1)–(3). Explicitly the result of Eq. (4) (SUM WEIGHTED OF THE LISTS IN THE TIME) corresponds to adding the probable activities that result from moving, adding, or not changing objects in a scene (Eq. 4). This result would correspond to the probable instantaneous activity, i.e. in the

þ β �

VAct\_1 ⋮ VAct\_N 3 7 5 ½ � A

þ γ �

VAct\_1 ⋮ VAct\_N 3 7 5 ½ � Un

2 6 4

2 6 4

TAct\_1 ⋮ TAct\_N

TAct\_1 ⋮ TAct\_N

TAct\_1 ⋮ TAct\_N 3 7 5 ½ � M

3 7 5 ½ � A

3 7 5 ½ � Un

þ c �

þ c �

þ c �

SAct\_1 ⋮ SAct\_N

Human Activity Recognition without Vision Tracking http://dx.doi.org/10.5772/intechopen.82660

> SAct\_1 ⋮ SAct\_N

> > SAct\_1 ⋮ SAct\_N

3 7 5 ½ � M

3 7 5 ½ � A

> 3 7 5 ½ � Un

(1)

87

(2)

(3)

(4)

2 6 4

2 6 4

> 2 6 4

2 6 4

2 6 4

2 6 4

scene without any change, i.e. they have not been moved, added, or removed (Eq. 3).

would be a substitute if the activity is to prepare coffee (activity N). where:

IAct\_1 ⋮ IAct\_N

IAct\_1 ⋮ IAct\_N

IAct\_1 ⋮ IAct\_N 3 7 5 ½ � M

3 7 5 ½ � A

3 7 5 ½ � Un

• IAct = Recognized objects that are considered ingredients for each activity.

• SAct = Recognized objects that are considered substitutes for each activity.

• TAct = Recognized objects that are considered utensils for each activity.

• a, b, c = Constants for tuning contribution, a ¼ 0:5, b ¼ 0:3, c ¼ 0:2 [2].

VAct\_1 ⋮ VAct\_N 3 7 5 ½ � M

2 6 4

2 6 4

> 2 6 4

2 6 4

activities carried out Eq. (2).

VAct\_1 ⋮ VAct\_N

> VAct\_1 ⋮ VAct\_N

VAct\_1 ⋮ VAct\_N 3 7 5 ½ � Un

• VAct = Result for each activity from 1 to N.

• ½ � M; A; Un = MOVE, ADD, UNCHANGED.

2 6 4

last frames (1 to 4 frames).

2 6 4

PVAct\_1 ⋮ PVAct\_N 3 7 5 ¼ α �

3 7 5 ½ � M

> 3 7 5 ½ � A

¼ a �

¼ a �

¼ a �

2 6 4

> 2 6 4

Figure 7. (a) Definition of an activity. (b) Flow chart of activity recognition.

#### 2.2.2. Evaluation function and activity recognition

To start the recognition of activity we made three groupings of objects according to their action, in other words a list for moved objects (MOVE), another one for objects added to the scene (ADD), and a list of objects present in the scene, but they do not have to be moved or withdrawn (UNCHANGED).

The first evaluation corresponds to calculating that the contribution of each ingredient, utensil, and substitute has undergone a movement, taking into account that the contribution of ingredients, utensils, and substitutes is being weighted by the constants a, b, and c, respectively. The result is the value that these objects provide for each of the probable activities carried out Eq. (1). The second evaluation corresponds to calculating the contribution of each ingredient, utensil, and substitute that has been added to the scene. Similar to the first evaluation, the contribution of ingredients, utensils, and substitutes is considered to be weighted by the constants a, b, and c, respectively. The result is the value that these objects provides for each of the probable activities carried out Eq. (2).

The third evaluation is exactly the same in its procedure as the two previously made, with the only difference being that the objects that intervene here are the ones that have remained in the scene without any change, i.e. they have not been moved, added, or removed (Eq. 3).

It is important to emphasize that an object could be a utensil or a substitute depending on the activity, for example a glass would be a utensil if the activity is to prepare juice (activity 1) but would be a substitute if the activity is to prepare coffee (activity N). where:

$$
\begin{bmatrix} V\_{Act\\_1} \\ \vdots \\ V\_{Act\\_N} \end{bmatrix}\_{[\mathcal{M}]} = a \cdot \begin{bmatrix} I\_{Act\\_1} \\ \vdots \\ I\_{Act\\_N} \end{bmatrix}\_{[\mathcal{M}]} + b \cdot \begin{bmatrix} T\_{Act\\_1} \\ \vdots \\ T\_{Act\\_N} \end{bmatrix}\_{[\mathcal{M}]} + c \cdot \begin{bmatrix} \mathcal{S}\_{Act\\_1} \\ \vdots \\ \mathcal{S}\_{Act\\_N} \end{bmatrix}\_{[\mathcal{M}]} \tag{1}
$$

$$
\begin{bmatrix} V\_{Act\\_1} \\ \vdots \\ V\_{Act\\_N} \end{bmatrix}\_{[A]} = a \cdot \begin{bmatrix} I\_{Act\\_1} \\ \vdots \\ I\_{Act\\_N} \end{bmatrix}\_{[A]} + b \cdot \begin{bmatrix} T\_{Act\\_1} \\ \vdots \\ T\_{Act\\_N} \end{bmatrix}\_{[A]} + c \cdot \begin{bmatrix} S\_{Act\\_1} \\ \vdots \\ S\_{Act\\_N} \end{bmatrix}\_{[A]} \tag{2}
$$

$$
\begin{bmatrix} V\_{Act\\_1} \\ \vdots \\ V\_{Act\\_N} \end{bmatrix}\_{[\text{lIn}]} = a \cdot \begin{bmatrix} I\_{Act\\_1} \\ \vdots \\ I\_{Act\\_N} \end{bmatrix}\_{[\text{lIn}]} + b \cdot \begin{bmatrix} T\_{Act\\_1} \\ \vdots \\ T\_{Act\\_N} \end{bmatrix}\_{[\text{lIn}]} + c \cdot \begin{bmatrix} \mathcal{S}\_{Act\\_1} \\ \vdots \\ \mathcal{S}\_{Act\\_N} \end{bmatrix}\_{[\text{lIn}]} \tag{3}
$$

• VAct = Result for each activity from 1 to N.

• INGREDIENTS: This refers to objects considered as ingredients for the preparation of a

• TOOLS: This includes cooking utensils and cutlery necessary for the elaboration of the

• SUBSTITUTES: These are kitchen utensils that could be replaced by others that perform a

To start the recognition of activity we made three groupings of objects according to their action, in other words a list for moved objects (MOVE), another one for objects added to the scene (ADD), and a list of objects present in the scene, but they do not have to be moved or

The first evaluation corresponds to calculating that the contribution of each ingredient, utensil, and substitute has undergone a movement, taking into account that the contribution of ingredients, utensils, and substitutes is being weighted by the constants a, b, and c, respectively. The result is the value that these objects provide for each of the probable activities carried out Eq. (1).

recipe, e.g. for a cereal-activity (cereal, milk).

similar function, e.g. a cup for a glass.

2.2.2. Evaluation function and activity recognition

withdrawn (UNCHANGED).

86 Intelligent Video Surveillance

recipes-activities, e.g. cereal-activity (bowl, spoon).

Figure 7. (a) Definition of an activity. (b) Flow chart of activity recognition.

The fourth evaluation corresponds to the addition of the activity lists resulting from Eqs. (1)–(3). Explicitly the result of Eq. (4) (SUM WEIGHTED OF THE LISTS IN THE TIME) corresponds to adding the probable activities that result from moving, adding, or not changing objects in a scene (Eq. 4). This result would correspond to the probable instantaneous activity, i.e. in the last frames (1 to 4 frames).

$$
\begin{bmatrix}
\sum V\_{Act\\_1} \\
\vdots \\
\sum V\_{Act\\_N}
\end{bmatrix} = \alpha \cdot \begin{bmatrix}
V\_{Act\\_1} \\
\vdots \\
V\_{Act\\_N}
\end{bmatrix}\_{[M]} + \beta \cdot \begin{bmatrix}
V\_{Act\\_1} \\
\vdots \\
V\_{Act\\_N}
\end{bmatrix}\_{[A]} + \gamma \cdot \begin{bmatrix}
V\_{Act\\_1} \\
\vdots \\
V\_{Act\\_N}
\end{bmatrix}\_{[[Ln]]}\tag{4}$$

where:

$$a = \frac{1}{3} + \left(\frac{1}{6} - \gamma\right) \tag{5}$$

$$
\beta = \frac{1}{3} + \left(\frac{1}{6} - \gamma\right) \tag{6}
$$

$$\gamma = \frac{1}{3} \bullet \left( \frac{ElapsedTime}{AverageTime} \right) \tag{7}$$

where:

Factors α, β, γ serve to ponder the contribution of VAct ½ �M, VAct ½ �A, VAct ½ �Un. Note that activities resulting from actions such as moving (MOVE) or adding (ADD) objects to the scene are more relevant over time than the result of just keeping objects in the scene with no position changes (UNCHANGED).

The results of Eq. (4) represent activities recognized in the last four frames, so in no way would represent a global or final result of the activity recognized.

To obtain a more reliable result we must add a considerable group of results of Eq. (4), and by adding these results we obtain a statistically reliable result of the activity recognized activity, Eq. (8) being the maximum value of the activity recognized.

$$\text{Activity\\_Recopmixed} = \max \left\{ \sum\_{1}^{\text{Samples}} \begin{bmatrix} \sum Act\\_1\\ \vdots\\ \sum Act\\_N \end{bmatrix} \right\} \tag{8}$$

In this research a color Kinect camera was employed; the depth of view was not used. People doing the cooking activities in Figure 9(a) did so randomly without any training or prepared script. The system was tested with four typical breakfast activities consisting of: brewing

Human Activity Recognition without Vision Tracking http://dx.doi.org/10.5772/intechopen.82660 89

The ingredients, cooking utensils, and substitutes are shown in Figure 9(b) as follows: coffee,

The first tests of the system of recognition of the proposed activity used five videos for each of the four activities raised. The results were excellent and after 600 frames it could be clearly differentiated which was the activity being executed in Figure 10(b). Partial results of Eq. (4) are illustrated in Figure 10(a). It should be mentioned that the results of 1 to 4 frames illustrated in Figure 10(a) suffer from occlusions and changes in lighting that hinder a more

coffee, juice preparation, cereal preparation, and chocolate preparation.

Figure 9. (a) InHands automated kitchen scenario. (b) Selected objects for the experiment.

Figure 8. Complete implemented system.

sugar, chocolate, juice, cereal, milk, bowl, cup, glass, plate, and spoon.

Tsamples = Total samples of results of Eq. (4) during average time of activity recognized.

#### 3. Results

The complete system that is obtained to perform our proposal of activity recognition is outlined in Figure 8; each of its constitutive parts was explained in the previous sections.

For experiments in a real domestic scenario, we count on the automated kitchen developed under the InHands project in Figure 9(a) [1].

Figure 8. Complete implemented system.

where:

88 Intelligent Video Surveillance

where:

(UNCHANGED).

3. Results

AverageTime, later decreased.

• PVAct = The summation of value by activity (the last 1 to 4 frames).

α ¼ 1 3 þ

β ¼ 1 3 þ

γ ¼ 1 3

• AverageTime ¼ Average time for the execution of any predefined activity.

1 6 � γ � �

1 6 � γ � �

• ElapsedTime AverageTime � �

• ElapsedTime ¼ The elapsed time from the start of an activity. Initial value is set to

Factors α, β, γ serve to ponder the contribution of VAct ½ �M, VAct ½ �A, VAct ½ �Un. Note that activities resulting from actions such as moving (MOVE) or adding (ADD) objects to the scene are more relevant over time than the result of just keeping objects in the scene with no position changes

The results of Eq. (4) represent activities recognized in the last four frames, so in no way would

To obtain a more reliable result we must add a considerable group of results of Eq. (4), and by adding these results we obtain a statistically reliable result of the activity recognized activity,

> Tsamples X 1

8 ><

>:

Tsamples = Total samples of results of Eq. (4) during average time of activity recognized.

The complete system that is obtained to perform our proposal of activity recognition is outlined in Figure 8; each of its constitutive parts was explained in the previous sections.

For experiments in a real domestic scenario, we count on the automated kitchen developed

2 6 4

PAct\_1 ⋮ PAct\_N 3 7 5

9 >=

>;

(5)

(6)

(7)

(8)

• α, β, γ = Variables changing on the time, α þ β þ γ ≤ 1.

represent a global or final result of the activity recognized.

Eq. (8) being the maximum value of the activity recognized.

under the InHands project in Figure 9(a) [1].

Activity\_Recognized ¼ max

Figure 9. (a) InHands automated kitchen scenario. (b) Selected objects for the experiment.

In this research a color Kinect camera was employed; the depth of view was not used. People doing the cooking activities in Figure 9(a) did so randomly without any training or prepared script. The system was tested with four typical breakfast activities consisting of: brewing coffee, juice preparation, cereal preparation, and chocolate preparation.

The ingredients, cooking utensils, and substitutes are shown in Figure 9(b) as follows: coffee, sugar, chocolate, juice, cereal, milk, bowl, cup, glass, plate, and spoon.

The first tests of the system of recognition of the proposed activity used five videos for each of the four activities raised. The results were excellent and after 600 frames it could be clearly differentiated which was the activity being executed in Figure 10(b). Partial results of Eq. (4) are illustrated in Figure 10(a). It should be mentioned that the results of 1 to 4 frames illustrated in Figure 10(a) suffer from occlusions and changes in lighting that hinder a more

accurate recognition of objects; however, thanks to the evaluation of Eq. (8) the erroneous partial results can be filtered to obtain a correct overall result.

Proactive support is one of the objectives of the InHands project, so our system should be able to recognize activities without having to be segmented, i.e. in a normal sequence of events recognize the different activities that are being developed.

As mentioned, the following test phase consisted of placing several activities without segmenting and checking whether the system was capable of recognizing them. Figure 11 illustrates one of the tests performed with the preparation of unsegmented juice, cereal, and coffee without a preconfigured specific order. As evidenced in Figure 11(a) the instantaneous activity recognition system (1 to 4 frames) is unclear in showing the activity developed but after processing it with Eq. (8) the result is clear (Figure 11(b)). Therefore, the functioning of the systems in a continuous way without the need to segment activities is demonstrated. The

final activity is coffee preparation with satisfactory performance from the beginning. Figure 12 shows a sample frame of our video process for the recognition of objects and activities [3].

Figure 12. (a) Sample frame of a processed video sequence. (b) Sample frame of a processed video computer interface.

Human Activity Recognition without Vision Tracking http://dx.doi.org/10.5772/intechopen.82660 91

This proposal for recognition of human activity is different from others based on tracking as shown in [17]. It is based on the preparation of cooking recipes, considering the interaction of the user with the objects present in the scene. In a detailed way, after the recognition of objects we classify them in categories (ingredients, utensils, and substitutes) and for the interaction we define four actions (add, remove, move, and unchanged) weighting the contribution of objects

Among its notable features are: it does not use intrusive methods with the user and requires an average time of 0.25 s for an instant recognition of activity [18–22]. It is also noted that in all the tests the recognitions were met by surpassing problems of brightness and occlusions to allow

The system is flexible and scalable by simply adding more activity definitions (recipes). The

Future work would be to define with statistical methods the weighting constants here desig-

This research was supported by the InHands project (Interactive robotics for Human Assistance in Domestic Scenarios), grant P6-L13-AL.INHAND founded by Fundaci La Caixa, inside

4. Conclusions

nated as a, b, c,α, β, γ.

Acknowledgements

the Recercaixa research program [3].

and interactions to determine possible activity.

completely natural movements of the user.

system works continuously with no default activity segmentation.

Figure 10. (a) Instantaneous activity recognized Pð Þ VAct : CHOCOLATE. Vertical axis = percentage of similarity with the Pð Þ VAct , horizontal axis = number of frames. (b) Final activity recognized Tsamples P 1 !: CHOCOLATE.

Figure 11. (a) Instantaneous activity recognized Pð Þ VAct : JUICE—CEREAL—COFFEE. (b) Final activity recognized Tsamples P 1 !: JUICE—CEREAL—COFFEE.

Figure 12. (a) Sample frame of a processed video sequence. (b) Sample frame of a processed video computer interface.

final activity is coffee preparation with satisfactory performance from the beginning. Figure 12 shows a sample frame of our video process for the recognition of objects and activities [3].

## 4. Conclusions

accurate recognition of objects; however, thanks to the evaluation of Eq. (8) the erroneous

Proactive support is one of the objectives of the InHands project, so our system should be able to recognize activities without having to be segmented, i.e. in a normal sequence of events

As mentioned, the following test phase consisted of placing several activities without segmenting and checking whether the system was capable of recognizing them. Figure 11 illustrates one of the tests performed with the preparation of unsegmented juice, cereal, and coffee without a preconfigured specific order. As evidenced in Figure 11(a) the instantaneous activity recognition system (1 to 4 frames) is unclear in showing the activity developed but after processing it with Eq. (8) the result is clear (Figure 11(b)). Therefore, the functioning of the systems in a continuous way without the need to segment activities is demonstrated. The

Figure 10. (a) Instantaneous activity recognized Pð Þ VAct : CHOCOLATE. Vertical axis = percentage of similarity with the

Figure 11. (a) Instantaneous activity recognized Pð Þ VAct : JUICE—CEREAL—COFFEE. (b) Final activity recognized

Tsamples P 1 !

: CHOCOLATE.

partial results can be filtered to obtain a correct overall result.

90 Intelligent Video Surveillance

recognize the different activities that are being developed.

Pð Þ VAct , horizontal axis = number of frames. (b) Final activity recognized

Tsamples P 1 !

: JUICE—CEREAL—COFFEE.

This proposal for recognition of human activity is different from others based on tracking as shown in [17]. It is based on the preparation of cooking recipes, considering the interaction of the user with the objects present in the scene. In a detailed way, after the recognition of objects we classify them in categories (ingredients, utensils, and substitutes) and for the interaction we define four actions (add, remove, move, and unchanged) weighting the contribution of objects and interactions to determine possible activity.

Among its notable features are: it does not use intrusive methods with the user and requires an average time of 0.25 s for an instant recognition of activity [18–22]. It is also noted that in all the tests the recognitions were met by surpassing problems of brightness and occlusions to allow completely natural movements of the user.

The system is flexible and scalable by simply adding more activity definitions (recipes). The system works continuously with no default activity segmentation.

Future work would be to define with statistical methods the weighting constants here designated as a, b, c,α, β, γ.

## Acknowledgements

This research was supported by the InHands project (Interactive robotics for Human Assistance in Domestic Scenarios), grant P6-L13-AL.INHAND founded by Fundaci La Caixa, inside the Recercaixa research program [3].

C. Flores and J. Aranda are associated with the Institute for Bioengineering of Catalunya and Universitat Politècnica de Catalunya, Barcelona-Tech, Spain [3].

[8] Ryoo M. Human activity prediction: Early recognition of ongoing activities from streaming videos. In: IEEE International Conference on Computer Vision (ICCV); IEEE; 2011.

Human Activity Recognition without Vision Tracking http://dx.doi.org/10.5772/intechopen.82660 93

[9] Lei J, Ren X, Fox D. Fine-grained kitchen activity recognition using rgb-d. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing. Pittsburgh, Pennsylvania: ACM;

[10] Laganière R. OpenCV Computer Vision Application Programming Cookbook. 2nd ed.

[11] Sokolova M, Lapalme G. A systematic analysis of performance measures for classification tasks. In: Information Processing & Management. 2009. vol. 45, no. 4, pp. 427-437. https://

[12] Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Rec-

[13] Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Kudlur M. Tensorflow: A system for

[14] Szegedy C, Reed S, Erhan D, Anguelov D, Ioffe S. Scalable, high-quality object detection.

[15] Bradski G, Kaehler A. Learning OpenCV: Computer Vision with the OpenCV Library.

[16] Mordvintsev A, Abid K. OpenCV-Python Tutorials: Camera Calibration and 3D Reconstruction [Internet]. Available from: http://opencv-python-tutroals.readthedocs.io/en/latest/py\_ tutorials/py\_calib3d/py\_calibration/py\_calibration.html#calibration [Accessed: 10-04-2018]

[17] Stauffer C, Grimson W. Adaptive background mixture models for real-time tracking. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition; IEEE;

[19] Team OpenCV Dev. OpenCV <IP_ADDRESS> Documentation [Internet]. Available from: http://

[20] Creative Commons Attribution, Ros Tutorials [Internet]. Available from: http://wiki.ros.

[21] Python Software Foundation. The Python Tutorial [Internet]. Available from: https://

[22] Dennis D, Tin C, Marou R. Color Image Segmentation [Internet]. Available from: https://

theiszm.wordpress.com/tag/color-segmentation/ [Accessed: 10-04-2018]

2012. pp. 208-211. http://dx.doi.org/10.1145/2370216.2370248

Birmingham, UK: Packt Publishing Ltd; 2014. pp. 272-277

large-scale machine learning. In: OSDI. Vol. 16. 2016. pp. 265-283

California, USA: O'Reilly Media, Inc.; 2008. pp. 201-204, 384-404

docs.opencv.org/2.4/modules/refman.html [Accessed: 10-04-2018]

doi.org/10.1016/j.ipm.2009.03.002

arXiv preprint arXiv:1412.1441. 2014

[18] Goebel R. ROS by Example. Vol. 2. Lulu.com; 2015

org/ROS/Tutorials [Accessed: 10-04-2018]

www.python.org/ [Accessed: 10-04-2018]

ognition; 2016. pp. 779-788

1999. pp. 246-252

pp. 1036-1043

## Author details

Carlos Alberto Flores Vázquez1 \*, Joan Aranda2 , Daniel Icaza1 , Santiago Pulla1 , Marcelo Flores-Vázquez3 and Nelson Federico Cordova1

\*Address all correspondence to<EMAIL_ADDRESS>
1 GIRVyP Research Group, Catholic University of Cuenca, Cuenca, Ecuador

2 IBEC Institute for Bioengineering of Catalonia, Polytechnic University of Catalonia, Barcelona–Tech, Barcelona, Spain

3 Salesian Polytechnic University, Cuenca, Ecuador

## References

C. Flores and J. Aranda are associated with the Institute for Bioengineering of Catalunya and

, Daniel Icaza1

, Santiago Pulla1

,

Universitat Politècnica de Catalunya, Barcelona-Tech, Spain [3].

Marcelo Flores-Vázquez3 and Nelson Federico Cordova1

\*Address all correspondence to<EMAIL_ADDRESS>
3 Salesian Polytechnic University, Cuenca, Ecuador

Persons. Cham: Springer; 2014. pp. 152-155

\*, Joan Aranda2

1 GIRVyP Research Group, Catholic University of Cuenca, Cuenca, Ecuador

%2013062014.pdf?sequence=1&isAllowed=y [Accessed: 10-04-2018]

2 IBEC Institute for Bioengineering of Catalonia, Polytechnic University of Catalonia,

[1] Vinagre M, Aranda J, Casals A. An interactive robotic system for human assistance in domestic environments. In: International Conference on Computers for Handicapped

[2] Flores Vázquez C. Human activity recognition from object interaction in domestic scenarios [MS thesis]. Universitat Politècnica de Catalunya; 2014. Available from: https:// upcommons.upc.edu/bitstream/handle/2099.1/23706/TFM%20Carlos%20Flores%20Vazquez

[3] Flores-Vázquez C, Aranda J. Human activity recognition from object interaction in domestic scenarios. In: Ecuador Technical Chapters Meeting (ETCM); IEEE. IEEE; 2016. pp. 1-6. Available from: https://upcommons.upc.edu/bitstream/handle/2117/100423/v2%2bHUMAN %2bACTIVITY%2bRECOGNITION.pdf?sequence=3&isAllowed=y [Accessed: 10-04-2018]

[4] Aggarwal J, Ryoo M. Human activity analysis: A review. ACM Computing Surveys. 2011;

[5] Hongeng S, Nevatia R, Bremond F. Video-based event recognition: Activity representation and probabilistic recognition methods. Computer Vision and Image Understanding.

[6] Moore D, Essa I. Recognizing multitasked activities from video using stochastic contextfree grammar. In: Proceedings of 18th National Conference on Artificial Intelligence. 2002.

[7] Liefeng B, Sminchisescu C. Efficient match kernel between sets of features for visual recognition. In: Advances in Neural Information Processing Systems (NIPS). 2009. pp. 135-143

Author details

92 Intelligent Video Surveillance

References

43(3):16

2004;96(2):129-162

pp. 770-776

Carlos Alberto Flores Vázquez1

Barcelona–Tech, Barcelona, Spain

**Chapter 6**

Provisional chapter

**Device-Free Localization for Human Activity**

Device-Free Localization for Human Activity Monitoring

DOI: 10.5772/intechopen.79442

Over the past few decades, human activity monitoring has grabbed considerable research attentions due to greater demand for human-centric applications in healthcare and assisted living. For instance, human activity monitoring can be adopted in smart building system to improve the building management as well as the quality of life, especially for the elderly people who are facing health deterioration due to aging factor, without neglecting the important aspects such as safety and energy consumption. The existing human monitoring technology requires additional sensors, such as GPS, PIR sensors, video camera, etc., which incur cost and have several drawbacks. There exist various solutions of using other technologies for human activity monitoring in a smartly controlled environment, either deviceassisted or device-free. A radio frequency (RF)-based device-free indoor localization, known as device-free localization (DFL), has attracted a lot of research effort in recent years due its simplicity, low cost, and compatibility with the existing hardware equipped with RF interface. This chapter introduces the potential of RF signals, commonly adopted for wireless communications, as sensing tools for DFL system in human activity monitoring. DFL is based on the concept of radio irregularity where human existence in wireless communica-

Keywords: device-free localization, indoor localization, human detection and tracking,

Human movement and behavior while performing their daily activities have inherent hierarchical structure. As enabling technology, real-time human activity monitoring plays an important role in many human-centric applications in different areas such as healthcare, security, surveillance, smart building, etc., particularly to protect elder people and children from some

> © 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and eproduction in any medium, provided the original work is properly cited.

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

Shaufikah Shukri, Latifah Munirah Kamarudin and

tion field may interfere and change the wireless characteristics.

Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

Shaufikah Shukri, Latifah Munirah Kamarudin

**Monitoring**

Abstract

Mohd Hafiz Fazalul Rahiman

and Mohd Hafiz Fazalul Rahiman

http://dx.doi.org/10.5772/intechopen.79442

human activity monitoring

1. Introduction

## **Device-Free Localization for Human Activity Monitoring** Device-Free Localization for Human Activity Monitoring

DOI: 10.5772/intechopen.79442

Shaufikah Shukri, Latifah Munirah Kamarudin

Shaufikah Shukri, Latifah Munirah Kamarudin and Mohd Hafiz Fazalul Rahiman and Mohd Hafiz Fazalul Rahiman Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.79442

#### Abstract

Over the past few decades, human activity monitoring has grabbed considerable research attentions due to greater demand for human-centric applications in healthcare and assisted living. For instance, human activity monitoring can be adopted in smart building system to improve the building management as well as the quality of life, especially for the elderly people who are facing health deterioration due to aging factor, without neglecting the important aspects such as safety and energy consumption. The existing human monitoring technology requires additional sensors, such as GPS, PIR sensors, video camera, etc., which incur cost and have several drawbacks. There exist various solutions of using other technologies for human activity monitoring in a smartly controlled environment, either deviceassisted or device-free. A radio frequency (RF)-based device-free indoor localization, known as device-free localization (DFL), has attracted a lot of research effort in recent years due its simplicity, low cost, and compatibility with the existing hardware equipped with RF interface. This chapter introduces the potential of RF signals, commonly adopted for wireless communications, as sensing tools for DFL system in human activity monitoring. DFL is based on the concept of radio irregularity where human existence in wireless communication field may interfere and change the wireless characteristics.

Keywords: device-free localization, indoor localization, human detection and tracking, human activity monitoring

## 1. Introduction

Human movement and behavior while performing their daily activities have inherent hierarchical structure. As enabling technology, real-time human activity monitoring plays an important role in many human-centric applications in different areas such as healthcare, security, surveillance, smart building, etc., particularly to protect elder people and children from some

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and eproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

bad incidents. Due to the advanced development in medical, science and technology, the human average lifespan has increased rapidly where people are getting healthier and having longer lives, thus increased the aging population worldwide. The United States' medical research agency, known as National Institute of Health (NIH), reported that in 2012, 8.0% (or 562 million) of the 7 billion global human population are aged 65 and over, and the percentage has increased by 0.5% (or 55 million) in 2015 [1]. Based on the aging trends, NIH has projected that by 2050, the older population will grow substantially up to 17% (or 1.6 billion) throughout the world [1]. However, most of elderly people spend their extra lifespan in unhealthy manner, with often debilitating illness and disability due to the deterioration of physical or mental functions caused by age-related diseases. In fact, the increase in the older population has a slight impact on the increase in disability rates of world's population [2].

monitoring. This is inconvenient and inappropriate for human usage especially the elders or people with brain-related diseases (Alzheimer, amnesia, dementia, etc.) to remember each day to wear or to activate those sensors. Furthermore, the whole monitoring process is ineffective and futile if the subject forgets to carry the sensor. Besides, the acoustic-based approach is range limited and prone to false detections since it can only be used in a short range and can easily be influenced by other audio signals [15]. The motion-based sensor such as a single accelerometer is not able to provide sufficient information to the system if used alone, hence need to combine with other sensors for more efficient activity monitoring [16]. Nevertheless, both vision- and sensor-based approaches bear with huge costs due to expensive equipment, installation, and maintenance. All the advantages and disadvantages of

Recently, Radio Frequency (RF)-based approaches have received significant research attentions to be employed in the human presence detection and activity monitoring based on different wireless radio technologies such as RFID [17, 18], Wi-Fi [19, 20], ZigBee [21, 22], FM radio [23, 24], microwave [25], etc. According to studies on the impact of human presence and activity on the RF signal strength [26–28], it has been proven that the existence and movement of human body in wireless radio network environment will interfere the wireless signal profiles, either in

() Disadvantages

+ Maintain records – Interfere with privacy – Ineffective in the dark – High computational cost

+ No privacy issue + Lower cost (PIR) + High detection accuracy

– Prone to false detection – Insensitive to very slow motions

+ Very sensitive to motion

– Prone to false detections – Range limited

+ High detection accuracy + No privacy issue – Expensive devices (sensors)

– Work only directionally (ultrasonic)

– Disturb or limit the activities of the users – Required sensors installation and calibration

+ Inexpensive (audio)

– Raise physical discomfort issue (accelerometer and gyroscopes)

Device-Free Localization for Human Activity Monitoring http://dx.doi.org/10.5772/intechopen.79442 97

– No direct linear or angular position information – Low range and line-of-sight restriction (PIR)

+ Objects and distances are typically determined precisely

– Sensitive to temperature and angle of the target (ultrasonic) – Easily be influenced by other audio signals/noise

above approaches are summarized in Table 1.

Category Sensor technology (+) Advantages

Motion-based Accelerometer

Sound-based Ultrasonic

Sensor-based Body-worn sensors

Gyroscopes PIR

Acoustic

Audio (microphone)

(Body sensor networks)

Table 1. Advantages and disadvantages of existing sensor technologies.

Vision-based Video camera + Effective security measure

With the increasing of older and disabled population in most countries and regions across the world, human and activity monitoring has gained substantial attention from the research community for ambient assisted living or elderly care application. As reported in [3], majority of the elderly people are more comfortable to live independently at their own home and community. Nevertheless, in the modern society, the conventional ways of taking care of elders in the family are no longer effective. As a result, there is higher demand in the society for the assistive technologies such as an intelligent monitoring system that can record the elders' daily activities in which family can respectfully monitor their loved ones who live alone at home. Due to the lower income earner after retirement and higher standards of living, many of the elders cannot afford to pay their healthcare cost as well as the expensive healthcare system or private nursing home care services. Nonetheless, various human monitoring technologies have been developed that help elderly people to age in place.

Traditionally, the human activity monitoring technology is a vision-based [4], which requires the use of video camera to monitor the human activity. Although this vision-based is an effective security measure approach as it can retains the records with high resolution, it also has several drawbacks that involves cost-inefficiency for large-scale deployments, energy consumption, and serious user privacy concerns if it is used in inappropriate places such as lavatory or bathroom, bedroom, and even nursing room. However, in several applications like elderly care and assisted living, monitoring human activities in these privacy areas is very crucial and necessary. For instance, lavatory or bathroom is one of the potential places for falling due to its slippery condition, thus activity monitoring in this place is very important for elderly fall detection system in detecting the falling event [5]. Meanwhile, the activity monitoring in the bedroom is very important for patient sleep monitoring system in detecting unusual sleeping behavior. In fact, the video camera requires a good lighting area ineffective in the dark and has limited view angles.

In recent years, thousands of research study on human activity monitoring has been conducted involving the replacement of traditional vision-based approach with various technologies such as acoustic-based [6, 7], motion-based [8, 9], body-worn sensors [10, 11], gyroscope [12], as well as smartphone [13, 14]. While such approaches address the privacy concern issue, they are sensor-based or in other words impose the requirement that special sensors, that is to be attached to, carried or worn by the subject for an effective activity monitoring. This is inconvenient and inappropriate for human usage especially the elders or people with brain-related diseases (Alzheimer, amnesia, dementia, etc.) to remember each day to wear or to activate those sensors. Furthermore, the whole monitoring process is ineffective and futile if the subject forgets to carry the sensor. Besides, the acoustic-based approach is range limited and prone to false detections since it can only be used in a short range and can easily be influenced by other audio signals [15]. The motion-based sensor such as a single accelerometer is not able to provide sufficient information to the system if used alone, hence need to combine with other sensors for more efficient activity monitoring [16]. Nevertheless, both vision- and sensor-based approaches bear with huge costs due to expensive equipment, installation, and maintenance. All the advantages and disadvantages of above approaches are summarized in Table 1.

bad incidents. Due to the advanced development in medical, science and technology, the human average lifespan has increased rapidly where people are getting healthier and having longer lives, thus increased the aging population worldwide. The United States' medical research agency, known as National Institute of Health (NIH), reported that in 2012, 8.0% (or 562 million) of the 7 billion global human population are aged 65 and over, and the percentage has increased by 0.5% (or 55 million) in 2015 [1]. Based on the aging trends, NIH has projected that by 2050, the older population will grow substantially up to 17% (or 1.6 billion) throughout the world [1]. However, most of elderly people spend their extra lifespan in unhealthy manner, with often debilitating illness and disability due to the deterioration of physical or mental functions caused by age-related diseases. In fact, the increase in the older population has a

With the increasing of older and disabled population in most countries and regions across the world, human and activity monitoring has gained substantial attention from the research community for ambient assisted living or elderly care application. As reported in [3], majority of the elderly people are more comfortable to live independently at their own home and community. Nevertheless, in the modern society, the conventional ways of taking care of elders in the family are no longer effective. As a result, there is higher demand in the society for the assistive technologies such as an intelligent monitoring system that can record the elders' daily activities in which family can respectfully monitor their loved ones who live alone at home. Due to the lower income earner after retirement and higher standards of living, many of the elders cannot afford to pay their healthcare cost as well as the expensive healthcare system or private nursing home care services. Nonetheless, various human monitoring technologies have

Traditionally, the human activity monitoring technology is a vision-based [4], which requires the use of video camera to monitor the human activity. Although this vision-based is an effective security measure approach as it can retains the records with high resolution, it also has several drawbacks that involves cost-inefficiency for large-scale deployments, energy consumption, and serious user privacy concerns if it is used in inappropriate places such as lavatory or bathroom, bedroom, and even nursing room. However, in several applications like elderly care and assisted living, monitoring human activities in these privacy areas is very crucial and necessary. For instance, lavatory or bathroom is one of the potential places for falling due to its slippery condition, thus activity monitoring in this place is very important for elderly fall detection system in detecting the falling event [5]. Meanwhile, the activity monitoring in the bedroom is very important for patient sleep monitoring system in detecting unusual sleeping behavior. In fact, the video camera requires a good lighting area ineffective in the dark

In recent years, thousands of research study on human activity monitoring has been conducted involving the replacement of traditional vision-based approach with various technologies such as acoustic-based [6, 7], motion-based [8, 9], body-worn sensors [10, 11], gyroscope [12], as well as smartphone [13, 14]. While such approaches address the privacy concern issue, they are sensor-based or in other words impose the requirement that special sensors, that is to be attached to, carried or worn by the subject for an effective activity

slight impact on the increase in disability rates of world's population [2].

been developed that help elderly people to age in place.

and has limited view angles.

96 Intelligent Video Surveillance

Recently, Radio Frequency (RF)-based approaches have received significant research attentions to be employed in the human presence detection and activity monitoring based on different wireless radio technologies such as RFID [17, 18], Wi-Fi [19, 20], ZigBee [21, 22], FM radio [23, 24], microwave [25], etc. According to studies on the impact of human presence and activity on the RF signal strength [26–28], it has been proven that the existence and movement of human body in wireless radio network environment will interfere the wireless signal profiles, either in

Table 1. Advantages and disadvantages of existing sensor technologies.

constructive or destructive manners, in which will change the RF communication pattern between the wireless transceivers. This phenomenon is called radio irregularity, which often consider as a drawback in RF communication. In RF-based human detection and activity monitoring, researchers have seen the radio irregularity phenomenon as a benefit in which it can be exploited as sensing tools to locate the human presence in the indoor environment and discriminate human activities or gestures. Since RF-based human activity monitoring approaches only exploit the wireless communication features, there is no need for expensive physical sensing equipment and modules, which accordingly reduce the cost, ease the deployment, reduce the energy consumption, and protect user privacy [29].

Refs. [46, 47] are specifically on the Wi-Fi-based approaches. However, we do not focus on the classification approaches of human activity, as there exist several in-depth literatures on

Device-Free Localization for Human Activity Monitoring http://dx.doi.org/10.5772/intechopen.79442 99

The organization of the chapter is as follows: Section 2 "RF-based DFL Technology" briefly discusses the concept of DFL in the perspective of human activity monitoring as understood within this study and provide an extensive review on the existing works. We decompose the taxonomy of the existing RF-based DFL technologies for human activity monitoring into measurement-based categories, regardless the type of wireless radio technologies used. Section 3 "Opportunities and Potential" presents the potential applications based on the state-of-art of RFbased DFL technology. Based hereon, in the last Section 4 "Challenges, Open Issues and Future Directions," we outline the challenges and the possible solutions, discuss the open issues, and comment on the possible future research direction of activity recognition using DFL technology.

Historically, the DFL analogy was firstly introduced by Youssef et al. in 2009 as device-free passive (DfP) for location determination, in which the subject is not equipped with a radio device, or not required to actively participate in the localization system [52]. The concept of DFL relies on the fact that any changes on the radio network environment will fluctuate the received signal profiles, i.e., due to reflection, diffraction, absorption, or scattering phenomena. DFL exploits the potential of ubiquitous deployed Internet of Things (IoT) [53] devices for indoor localization by leveraging the RF fluctuations as an indicator of presence of obstruction, i.e., object or human body. In [54], we have briefly defined the concept of DFL technology in the context of human detection and counting, together with the comprehensive review on the

In correspondence to the tremendous progresses on DFL research, Scholz et al. have expanded the area of DFL technology in the context of activity recognition by introducing the concept of device-free RF-based for human activity monitoring as device-free radio-based activity recognition (DFAR) [55]. Instead of utilizing radio signal analysis for object detection and tracking, it can also be utilized in the DFL technology to recognize specific human movement, and even their activities and gesture. For instance, the fluctuations of ambient and local continuous signals have been exploited in detecting human daily activities such as walking, lying, crawling, or standing [56]. To ease the reader's understanding, we defined the DFL and

DFL: device-free localization system: a system which detects the presence of a passive target and locates the target's position using radio signal information while the target is not equipped

DFAM: device-free activity monitoring system: a system which monitors and recognizes the activity performed by a passive target using radio signal information while the target is not equipped with a wireless device, nor required to actively participate in the localiza-

with a wireless device, nor required to actively participate in the localization system.

human activity classification methods [48–51].

2. RF-based DFL technology

publications related to DFL research.

DFAM systems as:

tion system.

The RF-based approaches can be classified into device-bound and device-free. Like the sensorbased, the device-bound RF-based approach requires the on-body wireless sensors or devices (such as RFID tags or cards, Bluetooth wristbands, smart watches, etc.) to be attached to the subject, which has been known as one of the drawbacks. Hence, the subject is required to actively participate in the activity recognition and monitoring process by always remembering to activate and carry the wearable wireless devices. This device-bound system is also known as active monitoring system and the subjects are usually willing to be monitored by the system. Therefore, we refer the subject in this active monitoring system as an active target. As an example, daily activities, such as walking, sitting, lying, falling, etc., of an active target wearing a simple RFID tag can be tracked using RFID readers [30, 31]. Another example is that an active target carrying mobile phone or other Wi-Fi-embedded devices can be easily tracked by Wi-Fi detectors or monitor [32, 33].

Although the on-body wireless sensors such as RFID tags and RFID cards are commercially available and relatively low cost compared to other wireless technologies, their placement on the target's body may cause physical discomfort [34], especially the elders under long-term monitoring. Recent research works introduce the placement of RFID in the environments and objects instead of target's body for activity monitoring [35, 36]. However, reading multiple RFID tags at once may cause malfunction due to signal collision, thus anticollision algorithms are required in which incur an extra cost [37]. On the contrary, device-free RF-based approach, known as device-free localization (DFL), is a passive monitoring system that can locate and monitor human position and activity without the subject's participation, where the subjects do not need to carry or wear any radio devices. They are usually unaware with the system's existence, and possibly want to avoid being monitored [21]. The subject in this passive monitoring case is referred as passive target.

In this chapter, we review the recent progress of DFL for indoor environment prioritizing on human activity monitoring with a particular focus on the monitoring systems targeting personal health and assisted living applications. Our aims are to provide a comprehensive review on the topic and to quickly update the researchers beyond this field the state of art, potential, opportunities, challenges, opens issues, and future directions of activity recognition using DFL technology. To the best of our knowledge, although there exist surveys on human activity monitoring and recognition using vision-based [4, 38], wearable sensors [10, 39, 40], mobile phones [41, 42], there are only few surveys published in this research field on human activity monitoring using device-free RF-based [29, 43–45], including a general architecture of existing work especially in the context of healthcare and assisted living applications. Surveys as in Refs. [46, 47] are specifically on the Wi-Fi-based approaches. However, we do not focus on the classification approaches of human activity, as there exist several in-depth literatures on human activity classification methods [48–51].

The organization of the chapter is as follows: Section 2 "RF-based DFL Technology" briefly discusses the concept of DFL in the perspective of human activity monitoring as understood within this study and provide an extensive review on the existing works. We decompose the taxonomy of the existing RF-based DFL technologies for human activity monitoring into measurement-based categories, regardless the type of wireless radio technologies used. Section 3 "Opportunities and Potential" presents the potential applications based on the state-of-art of RFbased DFL technology. Based hereon, in the last Section 4 "Challenges, Open Issues and Future Directions," we outline the challenges and the possible solutions, discuss the open issues, and comment on the possible future research direction of activity recognition using DFL technology.

## 2. RF-based DFL technology

constructive or destructive manners, in which will change the RF communication pattern between the wireless transceivers. This phenomenon is called radio irregularity, which often consider as a drawback in RF communication. In RF-based human detection and activity monitoring, researchers have seen the radio irregularity phenomenon as a benefit in which it can be exploited as sensing tools to locate the human presence in the indoor environment and discriminate human activities or gestures. Since RF-based human activity monitoring approaches only exploit the wireless communication features, there is no need for expensive physical sensing equipment and modules, which accordingly reduce the cost, ease the deployment, reduce the

The RF-based approaches can be classified into device-bound and device-free. Like the sensorbased, the device-bound RF-based approach requires the on-body wireless sensors or devices (such as RFID tags or cards, Bluetooth wristbands, smart watches, etc.) to be attached to the subject, which has been known as one of the drawbacks. Hence, the subject is required to actively participate in the activity recognition and monitoring process by always remembering to activate and carry the wearable wireless devices. This device-bound system is also known as active monitoring system and the subjects are usually willing to be monitored by the system. Therefore, we refer the subject in this active monitoring system as an active target. As an example, daily activities, such as walking, sitting, lying, falling, etc., of an active target wearing a simple RFID tag can be tracked using RFID readers [30, 31]. Another example is that an active target carrying mobile phone or other Wi-Fi-embedded devices can be easily tracked by

Although the on-body wireless sensors such as RFID tags and RFID cards are commercially available and relatively low cost compared to other wireless technologies, their placement on the target's body may cause physical discomfort [34], especially the elders under long-term monitoring. Recent research works introduce the placement of RFID in the environments and objects instead of target's body for activity monitoring [35, 36]. However, reading multiple RFID tags at once may cause malfunction due to signal collision, thus anticollision algorithms are required in which incur an extra cost [37]. On the contrary, device-free RF-based approach, known as device-free localization (DFL), is a passive monitoring system that can locate and monitor human position and activity without the subject's participation, where the subjects do not need to carry or wear any radio devices. They are usually unaware with the system's existence, and possibly want to avoid being monitored [21]. The subject in this passive moni-

In this chapter, we review the recent progress of DFL for indoor environment prioritizing on human activity monitoring with a particular focus on the monitoring systems targeting personal health and assisted living applications. Our aims are to provide a comprehensive review on the topic and to quickly update the researchers beyond this field the state of art, potential, opportunities, challenges, opens issues, and future directions of activity recognition using DFL technology. To the best of our knowledge, although there exist surveys on human activity monitoring and recognition using vision-based [4, 38], wearable sensors [10, 39, 40], mobile phones [41, 42], there are only few surveys published in this research field on human activity monitoring using device-free RF-based [29, 43–45], including a general architecture of existing work especially in the context of healthcare and assisted living applications. Surveys as in

energy consumption, and protect user privacy [29].

98 Intelligent Video Surveillance

Wi-Fi detectors or monitor [32, 33].

toring case is referred as passive target.

Historically, the DFL analogy was firstly introduced by Youssef et al. in 2009 as device-free passive (DfP) for location determination, in which the subject is not equipped with a radio device, or not required to actively participate in the localization system [52]. The concept of DFL relies on the fact that any changes on the radio network environment will fluctuate the received signal profiles, i.e., due to reflection, diffraction, absorption, or scattering phenomena. DFL exploits the potential of ubiquitous deployed Internet of Things (IoT) [53] devices for indoor localization by leveraging the RF fluctuations as an indicator of presence of obstruction, i.e., object or human body. In [54], we have briefly defined the concept of DFL technology in the context of human detection and counting, together with the comprehensive review on the publications related to DFL research.

In correspondence to the tremendous progresses on DFL research, Scholz et al. have expanded the area of DFL technology in the context of activity recognition by introducing the concept of device-free RF-based for human activity monitoring as device-free radio-based activity recognition (DFAR) [55]. Instead of utilizing radio signal analysis for object detection and tracking, it can also be utilized in the DFL technology to recognize specific human movement, and even their activities and gesture. For instance, the fluctuations of ambient and local continuous signals have been exploited in detecting human daily activities such as walking, lying, crawling, or standing [56]. To ease the reader's understanding, we defined the DFL and DFAM systems as:

DFL: device-free localization system: a system which detects the presence of a passive target and locates the target's position using radio signal information while the target is not equipped with a wireless device, nor required to actively participate in the localization system.

DFAM: device-free activity monitoring system: a system which monitors and recognizes the activity performed by a passive target using radio signal information while the target is not equipped with a wireless device, nor required to actively participate in the localization system.

We illustrate the overall conceptual framework of RF-based DFL technology for human activity monitoring as in Figure 1, including the three important modules: wireless radio sensor network (WRSN), human detection (HD), and human monitoring (HM). The WRSN is a selfconfigured wireless network consisting of radio devices connected wirelessly, acting as the sensors, for monitoring and recording of the physical or environmental conditions, and organizing the collected information to a predefined central location for processing. The WRSN module works by detecting the availability of radio-embedded devices (sensors) for human presence detection, localization, and activity monitoring, as well as the deployment of the radio sensor networks. WRSN can be deployed in the real-world environments using any radio devices that utilize the similar technology or IEEE standard. For instance, Wi-Fi-based sensor network can be deployed using any devices that utilized the IEEE802.11 wireless local area network (WLAN), while ZigBee-based sensor network can be deployed using devices that utilized the IEEE802.15.4 wireless personal area network (WPAN).

The sensors in WRSN collect the information of current environment and forward the information to be processed by the HD module. HD module consists of detection and localization algorithms which analyze the information and automatically discover the presence of target, the number of target, the location of the target, the body temperature of the target, the activities performed by the target, the humidity of the environment, etc. HM module consists of activity

> classification algorithms connected with the designated context aware-based activity reasoning engines depending on the applications. Once an activity is detected, the HM module will observe, retrieve, and recognize the activities and alert the designated context aware-based activity reasoning engine to interpret ongoing events successfully or initiate actions as needed. For example, a ZigBee-based sensor network is deployed in a single bedroom apartment for elderly care

> Figure 2. ZigBee-based sensor network deployed for elderly care application with the integration of mobile apps

Device-Free Localization for Human Activity Monitoring http://dx.doi.org/10.5772/intechopen.79442 101

Following on the DFAM research in [55], many research works on human motion detection and activity monitoring have been presented utilizing different radio technologies such as RFID [35–37], WiFi [5, 19, 20], ZigBee [21, 22], FM radio [23, 24], microwave [25], etc., adopting different signal descriptors such as Receive Signal Strength (RSS) [57–61], Channel State Information (CSI) [5, 20, 62–64], Doppler effect [25, 65], and Packet Received Rate (PRR) [66], without neglecting the easy-of-use problem and physical discomfort issue. In the following subsection, we decompose the taxonomy of the existing RF-based DFL technologies for human activity monitoring into signal descriptor categories such as RSS-based, CSI-based, amplitudebased, Doppler-based, and PRR-based, regardless the type of wireless radio technologies used.

Similar to human presence detection, activity monitoring using RSSI-based DFL technology exploits the RF-signal fluctuation features, in which the components of the received signal are blocked, absorbed, and reflected by the human while performing an activity, inducing the RF signal in the vicinity of receivers into a specific characteristic pattern. Such pattern can be identified and classified for the corresponding activity by exploiting the changes on the RSS

application as depicted in Figure 2.

2.1. RSS-based

visualization.

of the affected wireless links.

Figure 1. The overall conceptual framework of the RF-based DFL system.

Figure 2. ZigBee-based sensor network deployed for elderly care application with the integration of mobile apps visualization.

classification algorithms connected with the designated context aware-based activity reasoning engines depending on the applications. Once an activity is detected, the HM module will observe, retrieve, and recognize the activities and alert the designated context aware-based activity reasoning engine to interpret ongoing events successfully or initiate actions as needed. For example, a ZigBee-based sensor network is deployed in a single bedroom apartment for elderly care application as depicted in Figure 2.

Following on the DFAM research in [55], many research works on human motion detection and activity monitoring have been presented utilizing different radio technologies such as RFID [35–37], WiFi [5, 19, 20], ZigBee [21, 22], FM radio [23, 24], microwave [25], etc., adopting different signal descriptors such as Receive Signal Strength (RSS) [57–61], Channel State Information (CSI) [5, 20, 62–64], Doppler effect [25, 65], and Packet Received Rate (PRR) [66], without neglecting the easy-of-use problem and physical discomfort issue. In the following subsection, we decompose the taxonomy of the existing RF-based DFL technologies for human activity monitoring into signal descriptor categories such as RSS-based, CSI-based, amplitudebased, Doppler-based, and PRR-based, regardless the type of wireless radio technologies used.

#### 2.1. RSS-based