Still , existing systems fall short of the applications ' needs in real-world scenarios , where the quality of the video is less than optimal and the viewpoint is uncontrolled and often not static . In this paper , we extend the Motion Interchange Patterns ( MIP ) framework for action recognition . This effective framework encodes motion by capturing local changes in motion directions and additionally uses mechanisms to suppress static edges and compensate for global camera motion . For spectral clustering , the distribution of data points in the local region is summarised by a covariance matrix which is viewed as the Mahalanobis distance . For the action recognition problem , we extract subspace information for each action class . The query video sequence is labeled by information regarding its distance to the subspaces of the corresponding video classes . The two components of the TEA block are complementary in temporal modeling . Finally , our approach achieves impressive results at low FLOPs on several action recognition benchmarks , such as Kinetics , Something-Something , HMDB51 , and UCF101 , which confirms its effectiveness and efficiency . Finally , an iterative mining method is employed for learning discriminative action primitive detectors . Extensive experiments on action recognition benchmarks demonstrate a superior recognition performance of the proposed framework . Therefore , action recognition gets more and more extensive attention , becomes the key topic for the related scholars to study [ 5 ] . During action recognition process under the mobile phone visual sensor , the process is vulnerable to the interference of objects with similar skin color and action , in the process of recognizing the action , in order to ensure the speed of recognition , generally distributing as sloppy feature points , so as to express composite action feature needs to be recognized[6,7 ] . However , if the action is composite feature in small area , traditional action recognition method based on sparse representation , when expressing action , is not able to express feature details within small area in high accuracy and achieve action recognition under mobile phone visual sensor accurately and effectively . A number of experiments have been performed in order to evaluate the proposed algorithm . The results revealed that the proposed algorithm is efficient and leads to an improved action recognition process . Moreover , it is suitable for We present an approach to localize generic actions in egocentric videos , called temporal action proposals ( TAPs ) , for accelerating the action recognition step . An egocentric TAP refers to a sequence of frames that may contain a generic action performed by the wearer of a head-mounted camera , e.g. , taking a knife , spreading jam , pouring milk , or cutting carrots . This paper introduces a new framework for human contour tracking and action sequence recognition . Given a gallery of labeled human contour sequences , we define each contour as a ' ' word ' ' and encode all of them into a contour dictionary . In this paper , we propose a framework of action sequence recognition by combining the representation of randomized time warping ( RTW ) with the enhanced Grassmann discriminant Analysis ( eGDA ) . RTW is an extension of Dynamic time warping ( DTW ) , and it has been shown to be effective for motion recognition , as it can effectively retain an actions temporal information by generating a low-dimensional subspace from a set of time elastic ( TE ) features of a video . Existing public domain multi-modal datasets for human action recognition only include actions of interest that have already been segmented from action streams . These datasets can not be used to study a more realistic action recognition scenario where actions of interest occur randomly and continuously among actions of non-interest or no actions . It is more challenging to recognize actions of interest in continuous action streams since the starts and ends of these actions are not known and need to be determined in an on-the-fly manner . Here , we adopt the motion vectors as the action features and Support Vector Machine ( SVM ) as the classifier . The action type recognition is to identify the type of the action within the group which is determined by action group classification . Since each action group has its own particular features , we utilize Pictorial Structures Model ( PSM ) to describe the variation of the human body and extracts suitable recognition features for the input action according to its action group . Supervised by this loss function , our method directly learns a similarity function from data . Employing a customized human action and context detection network , we obtain highly discriminative action image embeddings , which can be used in action image recognition and other tasks . Our approach is evaluated on the still-image action recognition task and the image caption generation task . In this paper , we introduce a discriminative model , two-dimensional Conditional Random Field ( 2D CRF ) which can model the spatial-temporal correlations in actions , and present algorithms for multi-view single person action and two persons interaction recognition based on this model . For the action representation part , we model each action as a bag of visual words based on the spatial-temporal features ; for the action recognition part , we use 2D CRF for multi-view single person action and two persons interaction recognition . We use IXMAS and our Multi-view Human Interaction ( MHI ) datasets for the experiments . An illustration of early fusion is shown in fig : early - fusion for , where the temporal dimension has been colour coded and the output mapping to 2D space is omitted . This design has been studied for video classification and action recognition , and was also one of the architectures proposed in VSRnet . However , VSRnet requires bicubic upsampling as opposed to sub - pixel convolution , making the framework computationally much less efficient in comparison . Attributes provide one such space ; they relate different classes through well - known and shared characteristics of objects . Attributes , which are often collected manually , have shown promising results in various applications , i.e. caption generation , face recognition , image retrieval , action recognition and image classification . The main challenge of attribute - based zero - shot learning arises on more challenging fine - grained data collections , in which categories may visually differ only subtly . FlowNet 2.0 clearly has advantages in terms of being robust to homogeneous regions ( rows 2 and 5 ) , image and compression artifacts ( rows 3 and 4 ) and it yields smooth flow fields with sharp motion boundaries . subsection : Performance on Motion Segmentation and Action Recognition To assess performance of FlowNet 2.0 in real - world applications , we compare the performance of action recognition and motion segmentation . Deep learning is an umbrella term for training neural networks with potentially many hidden layers using new training methods allowing to learn complex feature hierarchies from data . Applied to action unit recognition and smile recognition in particular , a deep convolutional neural network model with an overall accuracy of 99.45 % significantly outperforms existing approaches . The underlying extensive model selection allows to find for each action unit an appropriate architecture in order to maximize test accuracies . This creates additional data - implicit redundancy that can be exploited for video super - resolution as has been shown in . Spatio - temporal networks are popular as they fully utilise the temporal information from videos for human action recognition . In the future , we will investigate extending our ESPCN network into a spatio - temporal network to super - resolve one frame from multiple neighbouring frames using 3D convolutions . Our approach to action classification leverages information from human pose and integrates it with appearance and motion descriptors for improved performance . Our approach to action localization models the temporal evolution of actions in the video with a recurrent network trained on the level of person tracks . Finally , the third method in this thesis aims to avoid a prohibitive cost of video annotation and adopts discriminative clustering to analyze and combine different levels of supervision . Spatio-temporal action localization consists of three levels of tasks : spatial localization , action classification , and temporal segmentation . In this work , we propose a new Progressive Cross-stream Cooperation ( PCSC ) framework to iterative improve action localization results and generate better bounding boxes for one stream ( i.e. , Flow/RGB ) by leveraging both region proposals and features from another stream ( i.e. , RGB/Flow ) in an iterative fashion . Specifically , we first generate a larger set of region proposals by combining the latest region proposals from both streams , from which we can readily obtain a larger set of labelled training samples to help learn better action detection models . It is an important component of numerous real-life applications , such as human-computer interaction , sports analysis , video search , and many others . In this thesis we work on the problem of action recognition and localization , which is a crucial part of human behavior understanding . Action recognition explains what a human is doing in the video , while action localization indicates where and when in the video the action is happening . We propose Hierarchical Space-Time Segments as a new representation for action recognition and localization . This representation has a two-level hierarchy . In this work , we present a method for action localization and recognition using an exemplar-based approach . It starts from local dense yet scale-invariant spatio-temporal features . The selected tubes allow us to effectively learn a spatio-temporal action detector based on dense trajectories or CNNs . We conduct experiments on existing action localization benchmarks : UCF-Sports , J-HMDB and UCF-101 . Our results show that our approach , despite using sparse spatial supervision , performs on par with methods using full supervision , i.e. , one bounding box annotation per frame . Temporal detection is performed using a sliding window scheme inside each tube . Our approach outperforms the state of the art on challenging action localization benchmarks . Second , we introduce a weakly-supervised action localization method , ie , which does not require bounding box annotation . Spatiotemporal action localization in videos is a challenging problem which is also an essential and important part of video understanding . Impressive progress has been reported in recent literature for action localization in videos , however , current state-of-the-art approaches have n’t considered the scenario of broken actions , in which an action in an untrimmed video is not a continuous image series anymore because of occlusion , shot change , etc . So , one action is divided into two or more footages ( sub-actions ) and the existing methods localize each of them as an independent action . Our conclusion generalizes to datasets with very different properties . When combined with several other cost-effective designs including separable spatial/temporal convolution and feature gating , our system results in an effective video classification system that that produces very competitive results on several action classification benchmarks ( Kinetics , Something-something , UCF101 and HMDB ) , as well as two action detection ( localization ) benchmarks ( JHMDB and UCF101 - 24 ) . Our proposed framework consists of two major components . First , for action frame localization , we take advantage of the self-attention mechanism to weight each frame , such that the influence of background frames can be effectively eliminated . Second , considering that there are trimmed videos publicly available and also they contain useful information to leverage , we present an additional module to transfer the knowledge from trimmed videos for improving the classification performance in untrimmed ones . Finally , we exploit the object positions and sizes in the spatial-aware embedding to demonstrate a new spatiotemporal action retrieval scenario with composite queries . Action localization and classification experiments on four contemporary action video datasets support our proposal . Apart from state-of-the-art results in the zero-shot localization and classification settings , our spatial-aware embedding is even competitive with recent supervised action localization alternatives . However , few works exist that address action localization and prediction in videos . The current action localization research primarily focuses on the classification of temporally trimmed videos in which only one action occurs per frame . Moreover , nearly all the current approaches work only offline and are too slow to be useful in real-world environments . Weakly-supervised temporal action localization aims to predict when and what actions occur in untrimmed videos with only video-level class labels . Most current methods make prediction based on global features , while ignoring the classification performance of local descriptions of human body . We address the challenging problem of weakly supervised temporal action localization from unconstrained web videos , where only the video-level action labels are available during training . Inspired by the adversarial erasing strategy in weakly supervised semantic segmentation , we propose a novel iterative-winners-out network . Weakly supervised temporal action detection in untrimmed videos is an important yet challenging task , where only video-level class labels are available for temporally locating actions in the videos during training . In this paper , we propose a novel architecture for this task . Weakly supervised temporal action detection is a Herculean task in understanding untrimmed videos , since no supervisory signal except the video-level category label is available on training data . Under the supervision of category labels , weakly supervised detectors are usually built upon classifiers . We present a novel method that uses graph convolutions to explicitly model similarity between video moments . Our method utilizes similarity graphs that encode appearance and motion , and pushes the state of the art on THUMOS’14 , ActivityNet 1.2 , and Charades for weakly- supervised action localization . One of the biggest dificulties in human action analysis is the temporal complexity and structure of actions . By breaking actions down into smaller temporal pieces , it may be possible to enhance action recognition , or allow unsupervised temporal action clustering . We propose a temporal segmentation system for human action recognition based on person tracking and a novel segmentation algorithm . Most prominent temporal action localization methods are of the fully-supervised type , which rely heavily on frame-level labels , which could be prohibitively expensive to annotate . Thanks to recent developments on the Weakly-supervised Temporal Action Localization ( W-TAL ) , this alternative paradigm requires only video-level labels in training , alleviating such annotation efforts . Specifically , we present Action Coherence Network ( ACN ) for W-TAL , which features a new coherence loss that better supervises action boundary learning and facilitate proposal regression . Author(s ) : Dougherty , Thomas | Advisor(s ) : Roy-Chowdhury , Amit K | Abstract : In this thesis , weakly-supervised temporal activity localization and classification is considered with the use of web videos . Most activity localization methods depend on the availability of frame-wise annotation , which is a burdensome task to collect . We show the effectiveness of our model on two standard action datasets , ActivityNet v1.2 and THUMOS14 . RefineLoc equipped with a segment prediction-based pseudo ground truth generator improves the state-of-the-art in weakly-supervised temporal localization on the challenging and large-scale ActivityNet dataset by 1.5 % in average mAP . We present a novel method that uses graph convolutions to explicitly model similarity between video moments . Our method utilizes similarity graphs that encode appearance and motion , and pushes the state of the art on THUMOS’14 , ActivityNet 1.2 , and Charades for weakly- supervised action localization . Temporal action localization is an important yet challenging research topic due to its various applications . Since the frame-level or segment-level annotations of untrimmed videos require amounts of labor expenditure , studies on the weakly-supervised action detection have been springing up . However , most of existing frameworks rely on Class Activation Sequence ( CAS ) to localize actions by minimizing the video-level classification loss , which exploits the most discriminative parts of actions but ignores the minor regions . An Application to Weakly Supervised Action Classification . We present a novel algorithm for weakly supervised action classification in videos . We apply the proposed method to the problem of weakly supervised learning of actions and actors from movies together with corresponding movie scripts . The scaling up of the learning problem to 66 feature-length movies enables us to significantly improve weakly supervised action recognition . This paper is about weakly supervised action segmentation , where the ground truth specifies only a set of actions present in a training video , but not their true temporal ordering . Prior work typically uses a classifier that independently labels video frames for generating the pseudo ground truth , and multiple instance learning for training the classifier . Because of the high cost of preparing training videos with full supervision for action segmentation , weakly supervised approaches which are able to learn only from transcripts are very appealing . In this paper , we propose a new approach for weakly supervised action segmentation based on a two branch network . The two branches of our network predict two redundant but different representations for action segmentation . This paper addresses the problem of automatic temporal annotation of realistic human actions in video using minimal manual supervision . To this end we consider two associated problems : ( a ) weakly-supervised learning of action models from readily available annotations , and ( b ) temporal localization of human actions in test videos . To avoid the prohibitive cost of manual annotation for training , we use movie scripts as a means of weak supervision . Using the obtained action samples , we train temporal action detectors and apply them to locate actions in the raw video data . Our experiments demonstrate that the proposed method for weakly-supervised learning of action models leads to significant improvement in action detection . We present detection results for three action classes in four feature length movies with challenging and realistic video data . Weakly-supervised action localization problem requires training a model to localize the action segments in the video given only video level action label . It can be solved under the Multiple Instance Learning ( MIL ) framework , where a bag ( video ) contains multiple instances ( action segments ) . We address weakly supervised action alignment and segmentation in videos , where only the order of occurring actions is available during training . We propose Discriminative Differentiable Dynamic Time Warping ( D3TW ) , the first discriminative model using weak ordering supervision . The work on object and action localization in Chapter 4 is based on [ L9 , L10 , L13 , L15 ] and relates to [ L1 , L17 , L19 , L20 ] . The second contribution of this thesis is concerned with weakly-supervised action learning . Chap- ter 3 introduces methods for automatic annotation of action samples in video using readily-available video scripts . The goal of this work is spatio-temporal action localization in videos , using only the supervision from video-level class labels . The state-of-the-art casts this weakly-supervised action localization regime as a Multiple Instance Learning problem , where instances are a priori computed spatio-temporal proposals . Rather than disconnecting the spatio-temporal learning from the training , we propose Spatio-Temporal Instance Learning , which enables action localization directly from box proposals in video frames .
Still , existing systems fall short of the applications ' needs in real-world scenarios , where the quality of the video is less than optimal and the viewpoint is uncontrolled and often not static . In this paper , we extend the Motion Interchange Patterns ( MIP ) framework for action recognition . This effective framework encodes motion by capturing local changes in motion directions and additionally uses mechanisms to suppress static edges and compensate for global camera motion .
For spectral clustering , the distribution of data points in the local region is summarised by a covariance matrix which is viewed as the Mahalanobis distance . For the action recognition problem , we extract subspace information for each action class . The query video sequence is labeled by information regarding its distance to the subspaces of the corresponding video classes .
The two components of the TEA block are complementary in temporal modeling . Finally , our approach achieves impressive results at low FLOPs on several action recognition benchmarks , such as Kinetics , Something-Something , HMDB51 , and UCF101 , which confirms its effectiveness and efficiency .
Finally , an iterative mining method is employed for learning discriminative action primitive detectors . Extensive experiments on action recognition benchmarks demonstrate a superior recognition performance of the proposed framework .
Therefore , action recognition gets more and more extensive attention , becomes the key topic for the related scholars to study [ 5 ] . During action recognition process under the mobile phone visual sensor , the process is vulnerable to the interference of objects with similar skin color and action , in the process of recognizing the action , in order to ensure the speed of recognition , generally distributing as sloppy feature points , so as to express composite action feature needs to be recognized[6,7 ] . However , if the action is composite feature in small area , traditional action recognition method based on sparse representation , when expressing action , is not able to express feature details within small area in high accuracy and achieve action recognition under mobile phone visual sensor accurately and effectively .
A number of experiments have been performed in order to evaluate the proposed algorithm . The results revealed that the proposed algorithm is efficient and leads to an improved action recognition process . Moreover , it is suitable for
We present an approach to localize generic actions in egocentric videos , called temporal action proposals ( TAPs ) , for accelerating the action recognition step . An egocentric TAP refers to a sequence of frames that may contain a generic action performed by the wearer of a head-mounted camera , e.g. , taking a knife , spreading jam , pouring milk , or cutting carrots .
This paper introduces a new framework for human contour tracking and action sequence recognition . Given a gallery of labeled human contour sequences , we define each contour as a ' ' word ' ' and encode all of them into a contour dictionary .
In this paper , we propose a framework of action sequence recognition by combining the representation of randomized time warping ( RTW ) with the enhanced Grassmann discriminant Analysis ( eGDA ) . RTW is an extension of Dynamic time warping ( DTW ) , and it has been shown to be effective for motion recognition , as it can effectively retain an actions temporal information by generating a low-dimensional subspace from a set of time elastic ( TE ) features of a video .
Existing public domain multi-modal datasets for human action recognition only include actions of interest that have already been segmented from action streams . These datasets can not be used to study a more realistic action recognition scenario where actions of interest occur randomly and continuously among actions of non-interest or no actions . It is more challenging to recognize actions of interest in continuous action streams since the starts and ends of these actions are not known and need to be determined in an on-the-fly manner .
Here , we adopt the motion vectors as the action features and Support Vector Machine ( SVM ) as the classifier . The action type recognition is to identify the type of the action within the group which is determined by action group classification . Since each action group has its own particular features , we utilize Pictorial Structures Model ( PSM ) to describe the variation of the human body and extracts suitable recognition features for the input action according to its action group .
Supervised by this loss function , our method directly learns a similarity function from data . Employing a customized human action and context detection network , we obtain highly discriminative action image embeddings , which can be used in action image recognition and other tasks . Our approach is evaluated on the still-image action recognition task and the image caption generation task .
In this paper , we introduce a discriminative model , two-dimensional Conditional Random Field ( 2D CRF ) which can model the spatial-temporal correlations in actions , and present algorithms for multi-view single person action and two persons interaction recognition based on this model . For the action representation part , we model each action as a bag of visual words based on the spatial-temporal features ; for the action recognition part , we use 2D CRF for multi-view single person action and two persons interaction recognition . We use IXMAS and our Multi-view Human Interaction ( MHI ) datasets for the experiments .
An illustration of early fusion is shown in fig : early - fusion for , where the temporal dimension has been colour coded and the output mapping to 2D space is omitted . This design has been studied for video classification and action recognition , and was also one of the architectures proposed in VSRnet . However , VSRnet requires bicubic upsampling as opposed to sub - pixel convolution , making the framework computationally much less efficient in comparison .
Attributes provide one such space ; they relate different classes through well - known and shared characteristics of objects . Attributes , which are often collected manually , have shown promising results in various applications , i.e. caption generation , face recognition , image retrieval , action recognition and image classification . The main challenge of attribute - based zero - shot learning arises on more challenging fine - grained data collections , in which categories may visually differ only subtly .
FlowNet 2.0 clearly has advantages in terms of being robust to homogeneous regions ( rows 2 and 5 ) , image and compression artifacts ( rows 3 and 4 ) and it yields smooth flow fields with sharp motion boundaries . subsection : Performance on Motion Segmentation and Action Recognition To assess performance of FlowNet 2.0 in real - world applications , we compare the performance of action recognition and motion segmentation .
Deep learning is an umbrella term for training neural networks with potentially many hidden layers using new training methods allowing to learn complex feature hierarchies from data . Applied to action unit recognition and smile recognition in particular , a deep convolutional neural network model with an overall accuracy of 99.45 % significantly outperforms existing approaches . The underlying extensive model selection allows to find for each action unit an appropriate architecture in order to maximize test accuracies .
This creates additional data - implicit redundancy that can be exploited for video super - resolution as has been shown in . Spatio - temporal networks are popular as they fully utilise the temporal information from videos for human action recognition . In the future , we will investigate extending our ESPCN network into a spatio - temporal network to super - resolve one frame from multiple neighbouring frames using 3D convolutions .
Our approach to action classification leverages information from human pose and integrates it with appearance and motion descriptors for improved performance . Our approach to action localization models the temporal evolution of actions in the video with a recurrent network trained on the level of person tracks . Finally , the third method in this thesis aims to avoid a prohibitive cost of video annotation and adopts discriminative clustering to analyze and combine different levels of supervision .
Spatio-temporal action localization consists of three levels of tasks : spatial localization , action classification , and temporal segmentation . In this work , we propose a new Progressive Cross-stream Cooperation ( PCSC ) framework to iterative improve action localization results and generate better bounding boxes for one stream ( i.e. , Flow/RGB ) by leveraging both region proposals and features from another stream ( i.e. , RGB/Flow ) in an iterative fashion . Specifically , we first generate a larger set of region proposals by combining the latest region proposals from both streams , from which we can readily obtain a larger set of labelled training samples to help learn better action detection models .
It is an important component of numerous real-life applications , such as human-computer interaction , sports analysis , video search , and many others . In this thesis we work on the problem of action recognition and localization , which is a crucial part of human behavior understanding . Action recognition explains what a human is doing in the video , while action localization indicates where and when in the video the action is happening .
We propose Hierarchical Space-Time Segments as a new representation for action recognition and localization . This representation has a two-level hierarchy .
In this work , we present a method for action localization and recognition using an exemplar-based approach . It starts from local dense yet scale-invariant spatio-temporal features .
The selected tubes allow us to effectively learn a spatio-temporal action detector based on dense trajectories or CNNs . We conduct experiments on existing action localization benchmarks : UCF-Sports , J-HMDB and UCF-101 . Our results show that our approach , despite using sparse spatial supervision , performs on par with methods using full supervision , i.e. , one bounding box annotation per frame .
Temporal detection is performed using a sliding window scheme inside each tube . Our approach outperforms the state of the art on challenging action localization benchmarks . Second , we introduce a weakly-supervised action localization method , ie , which does not require bounding box annotation .
Spatiotemporal action localization in videos is a challenging problem which is also an essential and important part of video understanding . Impressive progress has been reported in recent literature for action localization in videos , however , current state-of-the-art approaches have n’t considered the scenario of broken actions , in which an action in an untrimmed video is not a continuous image series anymore because of occlusion , shot change , etc . So , one action is divided into two or more footages ( sub-actions ) and the existing methods localize each of them as an independent action .
Our conclusion generalizes to datasets with very different properties . When combined with several other cost-effective designs including separable spatial/temporal convolution and feature gating , our system results in an effective video classification system that that produces very competitive results on several action classification benchmarks ( Kinetics , Something-something , UCF101 and HMDB ) , as well as two action detection ( localization ) benchmarks ( JHMDB and UCF101 - 24 ) .
Our proposed framework consists of two major components . First , for action frame localization , we take advantage of the self-attention mechanism to weight each frame , such that the influence of background frames can be effectively eliminated . Second , considering that there are trimmed videos publicly available and also they contain useful information to leverage , we present an additional module to transfer the knowledge from trimmed videos for improving the classification performance in untrimmed ones .
Finally , we exploit the object positions and sizes in the spatial-aware embedding to demonstrate a new spatiotemporal action retrieval scenario with composite queries . Action localization and classification experiments on four contemporary action video datasets support our proposal . Apart from state-of-the-art results in the zero-shot localization and classification settings , our spatial-aware embedding is even competitive with recent supervised action localization alternatives .
However , few works exist that address action localization and prediction in videos . The current action localization research primarily focuses on the classification of temporally trimmed videos in which only one action occurs per frame . Moreover , nearly all the current approaches work only offline and are too slow to be useful in real-world environments .
Weakly-supervised temporal action localization aims to predict when and what actions occur in untrimmed videos with only video-level class labels . Most current methods make prediction based on global features , while ignoring the classification performance of local descriptions of human body .
We address the challenging problem of weakly supervised temporal action localization from unconstrained web videos , where only the video-level action labels are available during training . Inspired by the adversarial erasing strategy in weakly supervised semantic segmentation , we propose a novel iterative-winners-out network .
Weakly supervised temporal action detection in untrimmed videos is an important yet challenging task , where only video-level class labels are available for temporally locating actions in the videos during training . In this paper , we propose a novel architecture for this task .
Weakly supervised temporal action detection is a Herculean task in understanding untrimmed videos , since no supervisory signal except the video-level category label is available on training data . Under the supervision of category labels , weakly supervised detectors are usually built upon classifiers .
We present a novel method that uses graph convolutions to explicitly model similarity between video moments . Our method utilizes similarity graphs that encode appearance and motion , and pushes the state of the art on THUMOS’14 , ActivityNet 1.2 , and Charades for weakly- supervised action localization .
One of the biggest dificulties in human action analysis is the temporal complexity and structure of actions . By breaking actions down into smaller temporal pieces , it may be possible to enhance action recognition , or allow unsupervised temporal action clustering . We propose a temporal segmentation system for human action recognition based on person tracking and a novel segmentation algorithm .
Most prominent temporal action localization methods are of the fully-supervised type , which rely heavily on frame-level labels , which could be prohibitively expensive to annotate . Thanks to recent developments on the Weakly-supervised Temporal Action Localization ( W-TAL ) , this alternative paradigm requires only video-level labels in training , alleviating such annotation efforts . Specifically , we present Action Coherence Network ( ACN ) for W-TAL , which features a new coherence loss that better supervises action boundary learning and facilitate proposal regression .
Author(s ) : Dougherty , Thomas | Advisor(s ) : Roy-Chowdhury , Amit K | Abstract : In this thesis , weakly-supervised temporal activity localization and classification is considered with the use of web videos . Most activity localization methods depend on the availability of frame-wise annotation , which is a burdensome task to collect .
We show the effectiveness of our model on two standard action datasets , ActivityNet v1.2 and THUMOS14 . RefineLoc equipped with a segment prediction-based pseudo ground truth generator improves the state-of-the-art in weakly-supervised temporal localization on the challenging and large-scale ActivityNet dataset by 1.5 % in average mAP .
We present a novel method that uses graph convolutions to explicitly model similarity between video moments . Our method utilizes similarity graphs that encode appearance and motion , and pushes the state of the art on THUMOS’14 , ActivityNet 1.2 , and Charades for weakly- supervised action localization .
Temporal action localization is an important yet challenging research topic due to its various applications . Since the frame-level or segment-level annotations of untrimmed videos require amounts of labor expenditure , studies on the weakly-supervised action detection have been springing up . However , most of existing frameworks rely on Class Activation Sequence ( CAS ) to localize actions by minimizing the video-level classification loss , which exploits the most discriminative parts of actions but ignores the minor regions .
An Application to Weakly Supervised Action Classification . We present a novel algorithm for weakly supervised action classification in videos .
We apply the proposed method to the problem of weakly supervised learning of actions and actors from movies together with corresponding movie scripts . The scaling up of the learning problem to 66 feature-length movies enables us to significantly improve weakly supervised action recognition .
This paper is about weakly supervised action segmentation , where the ground truth specifies only a set of actions present in a training video , but not their true temporal ordering . Prior work typically uses a classifier that independently labels video frames for generating the pseudo ground truth , and multiple instance learning for training the classifier .
Because of the high cost of preparing training videos with full supervision for action segmentation , weakly supervised approaches which are able to learn only from transcripts are very appealing . In this paper , we propose a new approach for weakly supervised action segmentation based on a two branch network . The two branches of our network predict two redundant but different representations for action segmentation .
This paper addresses the problem of automatic temporal annotation of realistic human actions in video using minimal manual supervision . To this end we consider two associated problems : ( a ) weakly-supervised learning of action models from readily available annotations , and ( b ) temporal localization of human actions in test videos . To avoid the prohibitive cost of manual annotation for training , we use movie scripts as a means of weak supervision .
Using the obtained action samples , we train temporal action detectors and apply them to locate actions in the raw video data . Our experiments demonstrate that the proposed method for weakly-supervised learning of action models leads to significant improvement in action detection . We present detection results for three action classes in four feature length movies with challenging and realistic video data .
Weakly-supervised action localization problem requires training a model to localize the action segments in the video given only video level action label . It can be solved under the Multiple Instance Learning ( MIL ) framework , where a bag ( video ) contains multiple instances ( action segments ) .
We address weakly supervised action alignment and segmentation in videos , where only the order of occurring actions is available during training . We propose Discriminative Differentiable Dynamic Time Warping ( D3TW ) , the first discriminative model using weak ordering supervision .
The work on object and action localization in Chapter 4 is based on [ L9 , L10 , L13 , L15 ] and relates to [ L1 , L17 , L19 , L20 ] . The second contribution of this thesis is concerned with weakly-supervised action learning . Chap- ter 3 introduces methods for automatic annotation of action samples in video using readily-available video scripts .
The goal of this work is spatio-temporal action localization in videos , using only the supervision from video-level class labels . The state-of-the-art casts this weakly-supervised action localization regime as a Multiple Instance Learning problem , where instances are a priori computed spatio-temporal proposals . Rather than disconnecting the spatio-temporal learning from the training , we propose Spatio-Temporal Instance Learning , which enables action localization directly from box proposals in video frames .
Evaluating New Variants of Motion Interchange Patterns Exploring geometrical structures in high-dimensional computer vision data TEA: Temporal Excitation and Aggregation for Action Recognition Learning Semantic-Aligned Action Representation Research on action recognition method under mobile phone visual sensor Human Action Recognition based on MSVM and Depth Images Egocentric Temporal Action Proposals Contour graph based human tracking and action sequence recognition Enhancing discriminability of randomized time warping for motion recognition C-MHAD: Continuous Multimodal Human Action Dataset of Simultaneous Video and Inertial Sensing Human Activities Recognition Based on a Multi-level Configuration Learning discriminative action and context representations for action recognition in still images Distributed Human Action Recognition via 2D Conditional Random Field Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation Evaluation of output embeddings for fine-grained image classification FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks Deep Learning For Smile Recognition Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network Structured modeling and recognition of human actions in video. (Modélisation structurée et reconnaissance des actions humaines dans les vidéos) Improving Action Localization by Progressive Cross-Stream Cooperation Towards Action Recognition and Localization in Videos with Weakly Supervised Learning Action Recognition and Localization by Hierarchical Space-Time Segments Exemplar-based Action Recognition in Video Human Action Localization with Sparse Spatial Supervision Motion in action : optical flow estimation and action localization in videos. (Le mouvement en action : estimation du flot optique et localisation d'actions dans les vidéos) Get the Whole Action Event by Action Stage Classification Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification Learning Transferable Self-attentive Representations for Action Recognition in Untrimmed Videos with Weak Supervision Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions Real-time multiple spatiotemporal action localization and prediction approach using deep learning Exploiting Human Pose for Weakly-Supervised Temporal Action Localization Breaking Winner-Takes-All: Iterative-Winners-Out Networks for Weakly Supervised Temporal Action Localization Weakly Supervised Temporal Action Detection with Shot-Based Temporal Pooling Network Step-by-step Erasion, One-by-one Collection: A Weakly Supervised Temporal Action Detector Action Graphs: Weakly-supervised Action Localization with Graph Convolution Networks Linear regression motion analysis for unsupervised temporal segmentation of human actions Action Coherence Network for Weakly Supervised Temporal Action Localization Weakly-Supervised Temporal Activity Localization and Classification With Web Videos RefineLoc: Iterative Refinement for Weakly-Supervised Action Localization Action Graphs: Weakly-supervised Action Localization with Graph Convolution Networks Adversarial Seeded Sequence Growing for Weakly-Supervised Temporal Action Localization Similarity Constrained Latent Support Vector Machine: An Application to Weakly Supervised Action Classification Learning from Video and Text via Large-Scale Discriminative Clustering Set-Constrained Viterbi for Set-Supervised Action Segmentation Weakly Supervised Action Segmentation Using Mutual Consistency Automatic annotation of human actions in video Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation Modeling and visual recognition of human actions and interactions Spatio-Temporal Instance Learning: Action Tubes from Class Supervision
4cca640761c980c77a696a64ad3c1e95b82109be 5bf9023174f3e7548e718eefce38f292817ba8cf 1ebeb84e2b8e1182a2b4821c906200ecc49ae187 e4d7b8eb0a8e6d2bb5b90b027c1bf32bad320ba5 ba73dba0f2f8a746aedb203bcf7b073f4c91f0db 76f8da855f7c5f8a6d53efe024f9aa9182e2583c 7a955ee69174fb97c7017f2e227a5fa26b900537 f19ab33e91c59f5e5f42c6c2d40dca84e799b9b9 52398c7f209338a5465606a612c9c5933c69d0aa 039777cb8babb3a04095ca8adbff7381a685026c 2a57006e8b6d12e96a1bf3769d9dbc6efc72174b 2a98a0042affbc122b355d0b10226dbd63721cbf 4bca76d3cb40551dd386a088a709eb6db905afc1 297d88697a8d7e23327aee245a0767da108ecc26 c3d445c883a396501acf3b5f2cd7680b2b953903 ddf37df62a4f266906d8e361deb2be38b0bce1a2 cf2e2804b0601c88dd843a0e494e41b4a7959d81 78bf2d0fe58f8574fe45bfbd4431e7c60f0c3e3e 3daafe6389d877fe15d8823cdf5ac15fd919676f 7675e7d657b3dfe25022ad918a09b588ea0d2e30 473c74782b0120aa0a8b0e94f49fea6542a6d7da 815aa52cfc02961d82415f080384594639a21984 7e273f4763004d9e224e1e91c6f87e63d6d49daf af2896e36cec58c2acca03001d7f19db48e9e5c3 c561a3a9742d356fbcdfbb5c52310cb2d4dd9486 46c899e661ed9c0bef182b8019035bf5a68cd61d ff4f13d4973cfe74f8ac7ef8384548c22284011e a9922322aae36d845391a9685a5f665e65c7c536 d66b80614f873bab0adbb7b1902fcff39fe63fdd 412efd7ffb214d66b2a4e26ff1bbc805e0196b52 7ec727b0ace1e5344c762e8a57ae7bf2a19f4cc8 badd8b78b8ff2278215a5effe503440508692366 d3dc4e5efe48fccbb738b965931cf7625d426ca2 6681bac4cf6e8acdc7f92542c29c30558d280087 412efd7ffb214d66b2a4e26ff1bbc805e0196b52 cc4ce2dff0b386dbe28a67db78314c00926c79a8 01285f2d2e2ec274470fd49d09191f60d4edb878 f6e65e93566b909c04d2d74bc1ade1592e476adf e7cd1bf06d1a248863f58923505e72c1b55e2261 c0450f376334b8c4bd57d34a35ab060ddc3595ee b67efc8f41fcb3458f6ca5482e73bcc929323a09 a84514fc6b08b93e51432d0539cae4ab7692cc56 00ccecc56ed83945fafb8e2dc48ffc1609618040 9a7f53e41acd4bd7b78613d9253a8f89141f80cd c7e10cee568b9251c894e002c5304fb1205eb4ea
["Computer Science"] ["Computer Science"] ["Computer Science"] ["Medicine","Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Medicine","Mathematics"] ["Computer Science"] ["Computer Science","Mathematics"] ["Engineering","Computer Science","Medicine"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Political Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Philosophy"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Mathematics","Medicine"] ["Computer Science"] ["Mathematics","Computer Science","Medicine"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Mathematics"] ["Computer Science"] ["Computer Science"] ["Computer Science"]
2013 2016 2020 2018 2015 2014 2018 2008 2017 2020 2018 2017 2014 2017 2015 2017 2016 2016 2018 2019 2014 2013 2009 2017 2016 2018 2018 2019 2017 2020 2019 2019 2018 2018 2020 2014 2019 2019 2019 2020 2019 2012 2017 2020 2019 2009 2020 2019 2013 2018
CVPR Workshops MVA ICME ICDSC 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) ArXiv 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) CVPR ICCV BMVC PKAW ECCV (15) PRCV (3) ICONIP (4) ACM Multimedia WACV ICIP ECCV (7) ICCV
https://api.semanticscholar.org/4cca640761c980c77a696a64ad3c1e95b82109be https://api.semanticscholar.org/5bf9023174f3e7548e718eefce38f292817ba8cf https://api.semanticscholar.org/1ebeb84e2b8e1182a2b4821c906200ecc49ae187 https://api.semanticscholar.org/e4d7b8eb0a8e6d2bb5b90b027c1bf32bad320ba5 https://api.semanticscholar.org/ba73dba0f2f8a746aedb203bcf7b073f4c91f0db https://api.semanticscholar.org/76f8da855f7c5f8a6d53efe024f9aa9182e2583c https://api.semanticscholar.org/7a955ee69174fb97c7017f2e227a5fa26b900537 https://api.semanticscholar.org/f19ab33e91c59f5e5f42c6c2d40dca84e799b9b9 https://api.semanticscholar.org/52398c7f209338a5465606a612c9c5933c69d0aa https://api.semanticscholar.org/039777cb8babb3a04095ca8adbff7381a685026c https://api.semanticscholar.org/2a57006e8b6d12e96a1bf3769d9dbc6efc72174b https://api.semanticscholar.org/2a98a0042affbc122b355d0b10226dbd63721cbf https://api.semanticscholar.org/4bca76d3cb40551dd386a088a709eb6db905afc1 https://www.semanticscholar.org/paper/0584f25256c83f10349d6108fde1bccd82908d8f https://www.semanticscholar.org/paper/caccc069e658ea397c9faf673e74c959c734ff53 https://www.semanticscholar.org/paper/edd846e76cacfba5be37da99c006e3ccc9b861b0 https://www.semanticscholar.org/paper/8494fa7b95969d8c0df20ba20083b7e5cf73ebe3 https://www.semanticscholar.org/paper/03a5b2aac53443e6078f0f63b35d4f95d6d54c5d https://api.semanticscholar.org/297d88697a8d7e23327aee245a0767da108ecc26 https://api.semanticscholar.org/c3d445c883a396501acf3b5f2cd7680b2b953903 https://api.semanticscholar.org/ddf37df62a4f266906d8e361deb2be38b0bce1a2 https://api.semanticscholar.org/cf2e2804b0601c88dd843a0e494e41b4a7959d81 https://api.semanticscholar.org/78bf2d0fe58f8574fe45bfbd4431e7c60f0c3e3e https://api.semanticscholar.org/3daafe6389d877fe15d8823cdf5ac15fd919676f https://api.semanticscholar.org/7675e7d657b3dfe25022ad918a09b588ea0d2e30 https://api.semanticscholar.org/473c74782b0120aa0a8b0e94f49fea6542a6d7da https://api.semanticscholar.org/815aa52cfc02961d82415f080384594639a21984 https://api.semanticscholar.org/7e273f4763004d9e224e1e91c6f87e63d6d49daf https://api.semanticscholar.org/af2896e36cec58c2acca03001d7f19db48e9e5c3 https://api.semanticscholar.org/c561a3a9742d356fbcdfbb5c52310cb2d4dd9486 https://api.semanticscholar.org/46c899e661ed9c0bef182b8019035bf5a68cd61d https://api.semanticscholar.org/ff4f13d4973cfe74f8ac7ef8384548c22284011e https://api.semanticscholar.org/a9922322aae36d845391a9685a5f665e65c7c536 https://api.semanticscholar.org/d66b80614f873bab0adbb7b1902fcff39fe63fdd https://api.semanticscholar.org/412efd7ffb214d66b2a4e26ff1bbc805e0196b52 https://api.semanticscholar.org/7ec727b0ace1e5344c762e8a57ae7bf2a19f4cc8 https://api.semanticscholar.org/badd8b78b8ff2278215a5effe503440508692366 https://api.semanticscholar.org/d3dc4e5efe48fccbb738b965931cf7625d426ca2 https://api.semanticscholar.org/6681bac4cf6e8acdc7f92542c29c30558d280087 https://api.semanticscholar.org/412efd7ffb214d66b2a4e26ff1bbc805e0196b52 https://api.semanticscholar.org/cc4ce2dff0b386dbe28a67db78314c00926c79a8 https://api.semanticscholar.org/01285f2d2e2ec274470fd49d09191f60d4edb878 https://api.semanticscholar.org/f6e65e93566b909c04d2d74bc1ade1592e476adf https://api.semanticscholar.org/e7cd1bf06d1a248863f58923505e72c1b55e2261 https://api.semanticscholar.org/c0450f376334b8c4bd57d34a35ab060ddc3595ee https://api.semanticscholar.org/b67efc8f41fcb3458f6ca5482e73bcc929323a09 https://api.semanticscholar.org/a84514fc6b08b93e51432d0539cae4ab7692cc56 https://api.semanticscholar.org/00ccecc56ed83945fafb8e2dc48ffc1609618040 https://api.semanticscholar.org/9a7f53e41acd4bd7b78613d9253a8f89141f80cd https://api.semanticscholar.org/c7e10cee568b9251c894e002c5304fb1205eb4ea