Multi-task Self-Supervised Learning for Human Activity Detection
Aaqib Saeed Tanir Ozcelebi Johan Lukkien
cs.LG stat.ML




0000-0003-1473-0322

  Eindhoven University of Technology
  Eindhoven
  The Netherlands

a.saeed@tue.nl



  Eindhoven University of Technology
  Eindhoven
  The Netherlands

t.ozcelebi@tue.nl



 Eindhoven University of Technology
 Eindhoven
 The Netherlands

j.j.lukkien@tue.nl




Deep learning methods are successfully used in applications pertaining to ubiquitous computing, pervasive intelligence, health, and well-being. Specifically, the area of human activity recognition (HAR) is primarily transformed by the convolutional and recurrent neural networks, thanks to their ability to learn semantic representations directly from raw input. However, in order to extract generalizable features massive amounts of well-curated data are required, which is a notoriously challenging task; hindered by privacy issues and annotation costs. Therefore, unsupervised representation learning (i.e., learning without manually labeling the instances) is of prime importance to leverage the vast amount of unlabeled data produced by smart devices. In this work, we propose a novel self-supervised technique for feature learning from sensory data that does not require access to any form of semantic labels, i.e., activity classes. We learn a multi-task temporal convolutional network to recognize transformations applied on an input signal. By exploiting these transformations, we demonstrate that simple auxiliary tasks of the binary classification result in a strong supervisory signal for extracting useful features for the down-stream task. We extensively evaluate the proposed approach on several publicly available datasets for smartphone-based HAR in unsupervised, semi-supervised and transfer learning settings. Our method achieves performance levels superior to or comparable with fully-supervised networks trained directly with activity labels, and it performs significantly better than unsupervised learning through autoencoders. Notably, for the semi-supervised case, the self-supervised features substantially boost the detection rate by attaining a kappa score between 0.7-0.8 with only 10 labeled examples per class. We get similar impressive performance even if the features are transferred from a different data source. Self-supervision drastically reduces the requirement of labeled activity data, effectively narrowing the gap between supervised and unsupervised techniques for learning meaningful representations. While this paper focuses on HAR as the application domain, the proposed approach is general and could be applied to a wide variety of problems in other areas.



<ccs2012>
<concept>
<concept_id>10003120.10003138.10003140</concept_id>
<concept_desc>Human-centered computing Ubiquitous and mobile computing systems and tools</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10010147.10010257</concept_id>
<concept_desc>Computing methodologies Machine learning</concept_desc>
<concept_significance>500</concept_significance>
</concept>
</ccs2012>


[500]Human-centered computing Ubiquitous and mobile computing systems and tools
[500]Computing methodologies Machine learning



Multi-task Self-Supervised Learning for Human Activity Detection
    Johan Lukkien
    December 30, 2023
================================================================





§ INTRODUCTION


Over the last years, deep neural networks have been widely adopted for time-series and sensory data processing; achieving impressive performance in several application areas pertaining to pervasive sensing, ubiquitous computing, industries, health and well-being <cit.>. In particular, for smartphone-based human activity recognition (HAR), 1D convolutional and recurrent neural networks trained on raw labeled signals significantly improve the detection rate over traditional methods <cit.>. Despite the recent advances in the field of HAR, learning representations from a massive amount of unlabeled data still presents a significant challenge. Obtaining large, well-curated activity recognition datasets is problematic due to a number of issues. First, smartphone data are privacy sensitive, which makes it hard to collect sufficient amounts of user-activity instances in a real-life setting. Second, the annotation cost and the time it takes to generate a large volume of labeled instances are prohibitive. Finally, the diversity of devices, types of embedded sensors, variations in phone-usage, and different environments are further roadblocks in producing massive human-labeled data. To sum up, such expensive and hard to scale process of gathering labeled data generated by smart devices makes it very difficult to apply supervised learning in this domain directly.   

In light of these challenges, we pose the question whether it is possible to learn semantic representations in an unsupervised way to circumvent the manual annotation of the sensor data with strong labels, e.g., activity classes. In particular, the goal is to extract features that are on par with those learned with fully-supervised [The fully-supervised network is the standard deep model that is trained in an end-to-end fashion directly with activity labels without any pre-training.] methods. There is an emerging paradigm for feature learning called self-supervised learning that defines auxiliary (also known as pretext or surrogate) tasks to solve, where labels are readily extractable from the data without any human intervention, i.e., self-supervised. The availability of strong supervisory signals from the surrogate tasks enables us to leverage objective functions as utilized in a standard supervised learning setting <cit.>. For instance, the vision community proposed a considerable number of self-supervised tasks for advancing representation learning[also known as feature learning] from static images, videos, and audio (see Section <ref>). Most prominent among them are: colorization of grayscale images <cit.>, predicting image rotations <cit.>, solving jigsaw puzzles <cit.>, predicting the direction of video playback <cit.>, temporal order verification <cit.>, odd sequence detection <cit.>, audio-visual correspondence <cit.>, and curiosity-driven agents <cit.>. The presented methodology for sensor representation learning takes inspiration from these methods and takes leverage of signal transformations to extract highly generalizable features for the down-stream[or an end-task] task, i.e., HAR.    



Our work is motivated by the success of jointly learning to solve multiple self-supervised tasks <cit.> and we propose to learn accelerometer representations (i.e., features) by training a temporal convolutional neural network (CNN) to recognize the transformations applied to the raw input signal. Particularly, we utilize a set of signal transformations <cit.> that are applied on each input signal in the datasets, which are then fed into the convolutional network along with the original data for learning to differentiate among them. In this simple formulation, a group of binary classification tasks (i.e., to recognize whether a transformation such as permutation, scaling, and channel shuffling was applied on the original signal or not) act as surrogate tasks to provide a rich supervisory signal to the model. In order to extract highly generalizable features for the end-task of interest, it is essential to utilize transformations that exploit versatile invariances of the temporal data (further details are provided in Section <ref>). To this end, we utilize eight transformations to train a multi-task network for simultaneously recognizing each of them. The visual illustration of the proposed approach is given in Figure <ref>. In the pre-training phase, the network consisting of a common trunk with a separate head for each task is trained on self-supervised data, and in the second step, the features learned by the shared layers are utilized by the HAR model. Importantly, we want to emphasize that in order for the convolutional network to recognize the transformations, it must learn to understand the core signal characteristics through acquiring knowledge of underlying differences in the accelerometer signals for various activity categories. We support this claim through an extensive evaluation of our method on six publicly available datasets in unsupervised, semi-supervised and transfer learning settings, where it achieves noticeable improvements in all the cases while not requiring manually labeled data for feature learning. 

The main contributions of this paper are:


  * We propose to utilize self-supervision from large unlabeled data for human activity recognition.

  * We design a signal transformation recognition problem as a surrogate task for annotation-free supervision, which provides a strong training signal to the temporal convolutional network for learning generalizable features.

  * We demonstrate through extensive evaluation that the self-supervised features perform significantly better in the semi-supervised and transfer learning settings on several publicly available datasets. Moreover, we show that these features achieve performance that is superior to or comparable with the features learned via the fully-supervised approach (i.e., trained directly with activity labels).

  * We illustrate with SVCCA <cit.>, saliency mapping <cit.>, and t-SNE <cit.> visualizations that the features extracted via self-supervision are very similar to those learned by the fully-supervised network.

  * Our method substantially reduces the labeled data requirement, effectively narrowing the gap between unsupervised and supervised representation learning. 
 

The paper is organized as follows. Section 2 provides an overview of related paradigms and methodologies as background information. Section 3 introduces the proposed self-supervised representation learning framework for HAR. Section 4 presents an evaluation of our framework on publicly available datasets. Section 5 gives an overview of the related work. Finally, Section 6 concludes the paper and lists future directions for research.



§ PRELIMINARIES

In this section, we provide a brief overview of multiple learning paradigms, including multi-task, transfer, semi-supervised and importantly, representation learning. These either benefit or serve as fundamental building blocks of our self-supervised framework for representation extraction and robust HAR under various settings.



 §.§ Representation Learning

Representation (feature) learning is concerned with automatically extracting useful information from the data that can be effectively used for an impending machine learning problem such as classification. In the past, most of the efforts were spent on developing (and manually engineering) feature extraction methods based on domain expertise to incorporate prior knowledge in the learning process. However, these methods are relatively limited as they rely on human creativity to come up with novel features and lack the power to capture underlying explanatory factors in the milieu of low-level sensory input. To overcome these limitations and to automate the discovery of disentangled features, neural networks based approaches have been widely utilized, such as autoencoders and their variants <cit.>. Deep neural networks are composed of multiple (parameterized) non-linear transformations that are trained through a supervised or unsupervised objective function with the aim of yielding useful representations. These techniques have achieved indisputable empirical success across a broad spectrum of problems <cit.> thanks to the increasing dataset sizes and computing power availability. Nevertheless, representation learning still stands as a fundamental problem in machine intelligence and is an active area of research (see <cit.> for a detailed survey).
 


 §.§ Multi-task Learning

The goal of multi-task learning (MTL) is to enhance the learning efficiency and accuracy through simultaneously optimizing multiple objectives based on shared representations and exploiting relations among the tasks <cit.>. It is widely utilized in several application domains within machine learning such as natural language processing <cit.>, computer vision <cit.>, audio sensing <cit.>, and well-being <cit.>. In this learning setting, T supervised tasks, each with a dataset D^t = {x_i^t, y_i^t}_i=1^m and a separate cost function are made available. The multi-objective loss is then generally created through a weighted linear sum of the individual tasks' losses as:

    ℒ_Aggregated = ∑_t ∈ Tψ_t ℒ_t


where ψ_t is the task weight and ℒ_t is a task-specific loss function. It is important to note that,  MTL itself does not impose any restriction on the loss type of an individual task. Therefore, unsupervised and supervised tasks or tasks having different cost functions can be conveniently combined for learning representations. 



 §.§ Transfer Learning

Transfer learning aims to develop methods for preserving and leveraging previously acquired knowledge to accelerate the learning of novel tasks. In recent years, it has shown remarkable improvement in performance on several very challenging problems, especially in areas, where little-labeled data are available such as in natural language understanding, object recognition, and activity recognition <cit.>. In this paradigm, the goal is to transfer (or reuse) the learned knowledge from a source domain 𝒟_SRC to a target domain 𝒟_TRG. More precisely, consider domains 𝒟_SRC and 𝒟_TRG with learning tasks t_SRC and t_TRG, respectively. The goal is to help improve the learning of a predictive function f(.) in t_TRG using the knowledge extracted from 𝒟_SCR and t_SRC, where 𝒟_SRC≠𝒟_TRG, and/or t_SRC≠ t_TRG, meaning that domains or tasks may be different. This learning formulation enables to develop a high-quality model under different knowledge transfer settings (such as features, instances, weights) from existing labeled data of some related task or domain. For a detailed review of transfer learning, we refer an interested reader to <cit.>.



 §.§ Semi-supervised Learning

Semi-supervised learning provides a compelling framework for leveraging unlabeled data in cases when labeled data collection is expensive. It has been repeatedly shown that given enough computational power and supervised data; deep neural networks can achieve human-level performance on a wide variety of problems <cit.>. However, the curation of large-scale datasets is very costly and time-consuming as it either requires crowdsourcing or domain expertise such as in the case of medical imaging. Likewise, for several practical problems, it is simply not possible to create a large enough labeled dataset (e.g., due to privacy issues) to learn a model of reasonable accuracy. Semi-supervised learning algorithms offer a compelling alternative to fully-supervised methods for jointly learning from few labeled and a large number of unlabeled instances. More specifically, given a labeled training set of input-output pairs (X, Y) ∈ D_L and unlabeled instance set, X ∈ D_U, the broad aim is to produce a predictive function f_θ(X) making use of not only D_L but also the underlying structure in D_U, where θ represents the learnable parameters of the model. For a concise review and realistic evaluation of various deep learning based semi-supervised techniques, see <cit.>. 



 §.§ Towards Self-supervision

Deep learning has been increasingly used for end-to-end HAR with far superior performance that can be achieved through traditional machine learning methods <cit.>. However, learning from very few labeled data, i.e. few-shot and semi-supervised learning is still an issue as large labeled datasets are required to train a model of sufficient quality. Similarly, the utilization of previously learned knowledge from related data (or task) to rapidly solve a comparable problem is not addressed very well by the existing methods (see Section <ref> for more details). In this paper, we explore self-supervised feature learning for HAR that effectively utilizes unlabeled data. The exciting field of self-supervision is concerned with extracting supervisory signals from data without requiring any human intervention. The evolution of feature extraction methods from hand-crafted features towards self-supervised representations is illustrated in Figure <ref>. The input to each of the illustrated approaches is raw data, which is not shown for the sake of brevity. 




§ APPROACH


In this section, we present our self-supervised representation learning framework for HAR. First, we provide an overview of the methodology. Next, we discuss various learning tasks (i.e. transformation classification) and their benefits for generic features extraction from unlabeled data. Finally, we provide a detailed description of the network architecture, its implementation, and the optimization process. 


 §.§ Overview

The objective of our work is to learn general-purpose sensor representations based on a temporal convolutional network in an unsupervised manner. To achieve this goal, we introduce a self-supervised deep network named Transformation Prediction Network (TPN), which simultaneously learns to solve multiple (signal) transformation recognition tasks as shown in Figure <ref>. Specifically, the proposed multi-task TPN M_θ(.) is trained to produce estimates of the transformations applied to the raw input signal. We define a set of |T| distinct transformations (or tasks) T = {J_t(.)}_t ∈ T, where J_t(.) is a function that applies a particular signal alteration technique t to the temporal sequence x ∈ℝ^(N, C) to yield a transformed version of the signal J_t(x). The network M_θ(.) that has a common trunk and individual head for each task, it takes an input sequence and produces a probability of the signal being a transformed version of the original, i.e. P(J_t|x) = M_θ(x). Note, that given a set of unlabeled signals (e.g. of accelerometer), we can automatically construct a self-supervised labeled dataset D = {{(J_t(x_i), True), (x_i, False)}_t ∈ T}_i=1^m. Hence, given this set of m training instances, the multi-task self-supervised training objective that a model must learn to solve is: 

    min_θ∑_t ∈ T^ψ_t [- 1/m_t∑_i=1^m_t (y_i^tlog(M_θ(x_i^t)) + (1 - y_i^t)log(1 - M_θ(x_i^t))) ]


where M_θ(x^t) is the predicted probability of x being a transformed version t and θ are the learnable parameters of the network. m_t represents the number of instances for a task (which can vary but are equal in our case) and ψ_t is the loss-weight of task t.

We emphasize that, although the network has a separate layer to differentiate between original and each of the T transformations it can be extended in a straight-forward manner to recognize multiple transformations applied to the same input signal or for multi-label classification. In the following subsection, we explain the types of signal transformations that are used in this work. 



 §.§ Self-supervised Tasks: Signal Transformations

The aforementioned formulation requires the signal transformations J to define a multi-task classification that enables the convolutional model to learn disentangled semantic representations useful for down-stream tasks, e.g. activity detection. We aimed for conceptually simple, yet diverse tasks to possibly cover several invariances that commonly arise in temporal data <cit.>. Intuitively, a diverse set of tasks should lead to a broad spectrum of features, which are more likely to span the feature-space domain needed for a general understanding of the signal's characteristics. In this work, we propose to utilize eight straight-forward signal transformations (i.e. |T| = 8) <cit.> for the self-supervision of a network. More specifically, when transformations are applied on an input signal x, they result in eight variants of x. As mentioned earlier, the temporal convolutional model is then trained jointly on all the tasks' data to solve a problem of transformation recognition, which allows the model to extract high-level abstractions from the raw input sequence. The transformations utilized in this work are summarized below:

 
     
  * Noised:  Given sensor readings of a fixed length, a possible transformation is the addition of random noise (or jitter) in the original signal. Heterogeneity of device sensors, software, and other hardware can cause variations (noisy samples) in the produced data. A model that is robust against noise will generalize better as it learns features that are invariant to minor corruption in the signal.
    
     
  * Scaled: A transformation that changes the magnitude of the samples within a window through multiplying with a randomly selected scalar. A model capable of handling scaled signals produces better representations as it becomes invariant to amplitude and offset invariances. 
    
     
  * Rotated: Robustness against arbitrary rotations applied on the input signal can achieve sensor-placement (orientation) invariance. This transformation inverts the sample signs (without changing the associated class-label) as frequently happens if the sensor (or device) is, for example, held upside down. 

     
  * Negated: This simple transformation is an instance of both scaled (scaling by -1) and rotated transformations. It negates samples within a time window, resulting in a vertical flip or a mirror image of the input signal. 

     
  * Horizontally Flipped: This transformation reverses the samples along the time-dimension, resulting in a complete mirror image of an original signal as if it were evolved in the opposite time direction. 
    
     
  * Permuted: This transformation randomly perturbs the events within a temporal window through slicing and swapping different segments of the time-series to generate a new one, hence, facilitating the model to develop permutation invariance properties. 
    
     
  * Time-Warped: This transformation locally stretches or warps a time-series through a smooth distortion of time intervals between the values (also known as local scaling).
        
     
  * Channel-Shuffled: For a multi-component signal such as a triaxial accelerometer, this transformation randomly shuffles the axial dimensions.  


There are several benefits of utilizing transformations recognition as auxiliary tasks for feature extraction from unlabeled data.

Enabling the learning of generic representations: The primary motivation is that the above-defined pretext tasks enable the network to capture the core signal characteristics. More specifically, for the TPN to successfully recognize if the signal is transformed or not, it must learn to detect high-level semantics, sensor behavior under different device placements, time-shift of the events, varying amplitudes, and robustness against sensor noise, thus, contributing to solving the ultimate task of HAR.

Task diversification and elimination of low-level input artifacts: A clear advantage of using multiple self-supervised tasks as opposed to a single one is that it will lead to a more diverse set of features that are invariant to low-level artifacts of the signals. Had we chosen to utilize signal reconstruction, e.g. with autoencoders, this would learn to compress the input, but due to a weak supervisory signal (as compared to self-supervision), it may discover trivial features with no practical value for the activity recognition or any other task of interest. We compare our approach against other methods in section <ref>.

Transferring knowledge: Furthermore, with our approach, the unlabeled sensor data that are produced in huge quantity can be effectively utilized with no human intervention to pre-train a network that is suitable for semi-supervised and transfer learning settings. It is particularly of high value for training networks in a real-world setting, where very little or no supervision is available to learn a model of sufficient quality from scratch.

Other benefits: Our self-supervised method has numerous other benefits. It has an equivalent computational cost to supervised learning but with better convergence accuracy, making it a suitable candidate for continuous unsupervised representation learning in-the-wild. Moreover, our technique neither requires a sophisticated pre-processing (apart from z-normalization) nor needs a specialized architecture (which also requires labeled data) to exploit invariances. We will show in Section <ref> through extensive evaluation that the self-supervised models learn useful representations and dramatically improve performance over other learning strategies. Despite the simplicity of the proposed scheme, it allows utilizing data collected through a wide variety of devices from a diverse set of users. 



 §.§ Network Architecture and Implementation


We implement the TPN M_θ(.) as a multi-branch temporal convolutional neural network with a common trunk (shared layers) and a distinct head (private layers) for each task with a separate loss function. Hard parameter sharing is employed between all the task-specific layers to encourage strong weight utilization from the trunk. Figure <ref> illustrates the TPN containing three 1D convolutional layers consisting of 32, 64, and 96 feature maps with kernel sizes of 24, 16 and 8 respectively, and having a stride of 1. Dropout is used after each of the layers with a rate of 0.1, and L2 regularization is applied with a rate of 0.0001. Global max pooling is used after the last convolution layer to aggregate high-level discriminative features. Moreover, each task-specific layer is comprised of a fully-connected layer of 256 hidden units followed by a sigmoidal output layer for binary classification. We use ReLU as non-linearity in all the layers (except the output) and train a network with Adam optimizer <cit.> for a maximum of 30 epochs with a learning rate of 0.0003, unless stated otherwise. Furthermore, the activity recognition model has a similar architecture to the TPN except for a fully-connected layer that consists of 1024 hidden units followed by a softmax output layer with units depending on the activity detection task under consideration. Additionally, during training of this model, we apply early-stopping, if the network fully converges on the training set to avoid overfitting. 


The motivation for keeping the TPN architecture simple arises from the fact that we want to show the performance gain does not come from the number of parameters (or layers) or due to the utilization of other sophisticated techniques such as batch normalization but the improvement is due to self-supervised pre-training. Likewise, the choice of multi-task learning setting, where each task has an additional private layer manifests in letting the model push pretext task-specific features to the last layers and let the initial layers extract generic representations that are important for a wide variety of end-tasks. Moreover, our architectural specification allows for a straightforward extension to add other related tasks, if needed, such as input reconstruction. Although, we do not explore applying multiple transformations to the same sequence or train models for their recognition the network design is intrinsically capable of performing this multi-label classification task.




Our training process is summarized in Algorithm <ref>. For every instance, we first generate transformed versions of a signal for the self-supervised pre-training of the network. At each training iteration of the TPN model, we feed the data from all tasks simultaneously, and the overall loss is calculated as a weighted sum of the losses of different tasks. Once pre-training converges, we transfer the weights of convolutional layers from model M_θ to an activity recognition network C_θ for learning the final supervised task. Here, either all the transferred layers are kept frozen, or the last convolutional layer is fine-tuned depending on the learning paradigm. Figure <ref> depicts this process graphically, where shaded convolutional layers represent frozen weights, while others are either trained from scratch or optimized further on the end-task. To avoid ambiguity, in the experiment section, we explicitly mention when the results are from a fully-supervised or self-supervised (including fine-tuned) network.   



§ EVALUATION

In this section, we conduct an extensive evaluation of our approach on several publicly available datasets for human activity recognition (HAR) in order to determine the quality of learned representations, transferability of the features, and benefits of this in the low-data regime. The self-supervised tasks (i.e., transformation predictions) are utilized for learning rich sensor representations that are suitable for an end-task. We emphasize that achieving high performance on these surrogate tasks is not our focus. 





 §.§ Datasets


We consider six publicly available datasets to cover a wide variety of device types, data collection protocols, and activity recognition tasks performed with smartphones in different environments. Some important aspects of the data are summarized in Table <ref>. Below, we give brief descriptions of every dataset summarizing its key points. 



  §.§.§ HHAR

The Heterogeneity Human Activity Recognition (HHAR) dataset <cit.> contains signals from two sensors (accelerometer and gyroscope) of smartphones and smartwatches for 6 different activities, i.e. biking, sitting, standing, walking, stairs-up and stairs-down. The 9 participants executed a scripted set of activities for 5 minutes to get equal class distribution. The subjects had 8 smartphones in a tight pouch carried around their waist and 4 smartwatches, 2 worn on each arm. In total, they used 36 different smart devices of 13 models from 4 manufacturers to cover a broad range of devices for sampling rate heterogeneity analysis. The sampling rate of signals varied significantly across phones with values between 50-200Hz.  



  §.§.§ UniMiB

This dataset <cit.> contains triaxial accelerometer signals collected from a Samsung Galaxy Nexus smartphone at 50Hz. Thirty subjects participated in the data collection process forming a diverse sample of the population with different height, weight, age, and gender. The subject placed the device in her trouser's front left pocket for a partial duration and in the right pocket for the remainder of the experiment. We utilized the data of 9 activities of daily living (i.e., standing up from sitting, standing up from lying, walking, running, upstairs, jumping, downstairs, lying down from sitting, sitting) in this paper. 



  §.§.§ UCI HAR

The UCI HAR dataset <cit.> is obtained from a group of 30 volunteers with a waist-mounted Samsung Galaxy S2 smartphone. The accelerometer and gyroscope signals are collected at 50Hz when subjects performed the following six activities: standing, sitting, laying down, walking, downstairs and upstairs. 



  §.§.§ MobiAct

The MobiAct[second release] dataset <cit.> contains signals from a smartphone's inertial sensors (accelerometer, gyroscope, and orientation) for 11 different activities of daily living and 4 types of falls. It is collected with a Samsung Galaxy S3 smartphone from 66 participants of different gender, age group, and weight through more than 3200 trials. The device is placed in a trouser's pocket freely selected by the subject in any random orientation to capture everyday usage of the phone. We used the data from 61 participants who have data samples for any of the following 11 activities: sitting, walking, jogging, jumping, stairs up, stairs down, stand to sit, sitting on a chair, sit to stand, car step-in, and car step-out. 



  §.§.§ WISDM

The dataset from the Wireless Sensor and Data Mining (WISDM) project <cit.> was collected in a controlled study from 29 volunteers, who carried the cell phone in their pockets.  The data were recorded for 6 different activities (i.e., sit, stand, walk, jog, ascend stairs, descend stairs) via an app developed for an Android phone. The accelerometer signal was acquired every 50ms (sampling rate of 20Hz). We use the data of all the users available in the raw data file with user ids ranging from 1 to 36. 



  §.§.§ MotionSense

The MotionSense dataset <cit.> comprises an accelerometer, gyroscope, and altitude data from 24 participants of varying age, gender, weight, and height. It was collected using an iPhone6s, which is kept in the user's front pocket. The subjects performed 6 different activities (i.e., walking, jogging, downstairs, upstairs, sitting, and standing.) in 15 trials under similar environments and conditions. The study aimed to infer physical and demographics attributes from time-series data in addition to the detection of activities.       



 §.§ Data Preparation and Assessment Strategy

We applied minimal pre-processing on the accelerometer signals as deep neural networks are very good at learning abstract representations directly from raw data <cit.>. We segmented the signals into fixed size windows that have 400 samples with 50% overlap, for all the datasets under consideration. The appropriate window size is a task-specific parameter and could be tuned or chosen based on prior knowledge for improved performance. Here, we utilize the same window size based on earlier exploration across datasets and to keep experimental evaluation impartial towards the effect of this hyper-parameter. Next, we divide each dataset into training and test sets through randomly selecting 20-30% of the users for testing and the rest for training and validation; depending on the dataset size. We used the ceiling function to select number of users, e.g. from HHAR dataset 3 users are used for evaluation out of 9. The training set users' data are further divided into 80% for training the network and 20% for validation and hyper-parameter tuning. Importantly, we also evaluate our models through user-split based 5-folds cross-validation, wherever it is appropriate. Finally, we normalize the data by applying z-normalization with summary statistics calculated from the training set. We generate self-supervised data from an unlabeled training set that is produced as a result of the processing as mentioned earlier. We utilize the data generation procedure as explained earlier in Section <ref>.

Furthermore, due to the large size of the HHAR dataset and in order to reduce computational load, we randomly sample 4000 instances from each users' data to produce transformed signals. Likewise, in the case of UniMiB because of its relatively small size, we generate 5 times more transformed instances. We evaluate the performance with Cohen's kappa, a weighted version of precision, recall and f-score metrics to be robust against inherent imbalanced nature of the datasets. It is important to highlight that, we use a network architecture with the same configuration across the datasets to evaluate models' performance in order to highlight improvement is indeed due to self-supervision and not due to architectural modifications.  



 §.§ Results




  §.§.§ Quantifying the Quality of Learned Feature Hierarchies

We first evaluate our approach to determine the quality of learned representations versus the model depth (i.e., the layer number from which the features come). This analysis helps in understanding whether the features coming from different layers vary in quality concerning their performance on an end-task and if so, which layer should be utilized for this purpose. To this end, we first pre-train our TPN in a self-supervised manner and learn classifiers on top of ConvA, ConvB, and ConvC layers independently, for several activity recognition datasets. These classifiers (see Figure <ref>) are trained in a supervised way while keeping the learned features fixed during the optimization process. Figure <ref> provides kappa values on test sets averaged across 10-independent runs to be robust against differences in weight initializations of the classifiers. We observe that for a majority of the datasets the model performance improves with increasing depth apart from HHAR, where features from ConvB layer results in improved detection rate with a kappa of 0.774 compared to 0.679 of ConvC. It may be because the representation of the last layer starts to become too specific on the transformation prediction task or it may also be because we did not utilize the entire dataset for the self-supervision. To be consistent, in the subsequent experiments we used features from the last convolutional layer for all the considered datasets. For a new task or recognition problem, we recommend performing a similar analysis to identify layer/block of the network that gives optimal results on the particular dataset. 




  §.§.§ Comparison against Fully-Supervised and Unsupervised Approaches

In this subsection, we assess our self-supervised representations learned with TPN against other unsupervised and fully-supervised techniques for feature learning. Table <ref> summarizes the results with respect to four evaluation metrics (namely, precision, recall, f-score, and kappa) for 10-independent runs on the six datasets described earlier. For the Random Init. entries, we keep the convolutional network layers frozen during optimization and train only a classifier in a supervised manner. Likewise, for an Autoencoder, we keep the network architecture the same and pre-train it in an unsupervised way. Afterward, the weights of the encoder are kept frozen, and a classifier is trained on top as usual. The Self-Supervised entries show the result of the convolutional network pre-trained with our proposed method, where a classifier is trained on top of the frozen network in a supervised fashion. Furthermore, Self-Supervised (FT) entries highlight the performance of the network trained with self-supervision but the last convolution layer, i.e. ConvC is fine-tuned along with a classifier during training on the activity recognition task. Training an activity classification model on top of randomly initialized convolutional layers poorly performs as expected, which is evidence that the performance improvement is not only because of the activity classifier. These results are followed by a widely used unsupervised learning method, i.e. an autoencoder. The self-supervised technique outperforms existing methods and achieves results that are on par with the fully-supervised model. It is important to note that, for our proposed technique, only the classifier layers are randomly initialized and trained with activity specific labels (the rest is transferred from the self-supervised network). We also observe that fine-tuning the last convolutional layer further improves the classification performance of the down-stream tasks on several datasets such as UniMiB, HHAR, MobiAct, and UCI HAR. The results show that TPN can learn highly generalizable representations, thus reducing the performance gap of feature learning with the (end-to-end) supervised case. For a more rigorous evaluation, we also performed 5-folds (user split based) cross-validation for every method on all the datasets. The results are provided in Table <ref> of the appendix, which also shows that the self-supervised method reduces the performance gap with the supervised setting.   





  §.§.§ Assessment of Individual Self-Supervised Tasks in Contrast with Multiple Tasks

In Figure <ref>, we show comparative performance analysis of single self-supervised tasks with each other and importantly with a multi-task setting. This assessment helps us in understanding whether self-supervised features extracted via jointly learning to solve multiple tasks are any better (for activity classification) than independently solving individual tasks and whether multi-task learning helps in learning more useful sensor semantics. To achieve this, we pre-train a TPN on each of the self-supervised tasks and transfer the weights for learning an activity recognition classifier. We observe in all the cases that learning representations via solving multiple tasks lead to far better performance on the end-task. This further highlights that the features learned through various self-supervised tasks have different strengths and weaknesses. Therefore, merging multiple tasks results in an improvement in learning a diverse set of features. However, we notice that some tasks (such as Channel Shuffled, Permuted, and Rotated) consistently performed better compared to others across datasets; achieving a kappa score above 0.60 as evaluated on different activity recognition problems. It highlights an important point that there may exist a group of tasks, which are reasonably sufficient to achieve a model of good quality. Furthermore, in Figure <ref> of the appendix, we plot the kappa score achieved by a multi-task TPN on transformation recognition tasks as a function of the number of training epochs. This analysis highlights that task complexity varies greatly from one dataset to another and may help with the identification of trivial auxiliary tasks that may lead to non-generalizable features. 

In addition to activity classification, for any learning task involving time-series sensor data (e.g., as encountered in a various Internet of Things applications), we recommend extracting features through first solving individual tasks and later focusing on the multi-task scenario; discarding low performing tasks or assigning low-weights to the loss functions of the respective tasks. Another approach could be to auto-tune the task-loss weight by taking homoscedastic uncertainty of each task into account <cit.>.  





  §.§.§ Effectiveness under Semi-Supervised Setting


Our proposed self-supervised feature learning method attains very high performance on different activity recognition datasets. This brings up the question, whether the self-supervised representations can boost performance in the semi-supervised learning setting as well or not. In particular, can we use this to perform activity detection with very little labeled data? Intrigued by this, we also evaluate the effectiveness of our approach to semi-supervised learning. Specifically, we initially train a TPN on an entire training set for transformation prediction. Subsequently, we learn a classifier on top of the last layer's feature maps with only a subset of the available accelerometer samples and their corresponding activity labels. For training an activity classifier, we use for each category (class) 2, 5, 10, 20, 50, and 100 examples. Note that, 2-10 samples per class represent a real-world scenario of acquiring a (small) labeled dataset from human users with minimal interruption to their daily routines, hence, making self-supervision from unlabeled data of great value. Likewise, we believe, our analysis of learning with very few labeled instances across datasets is the first attempt in quantifying the amount of labeled data required to learn an activity recognizer of decent quality. For self-supervised models, as earlier, we either kept the weights frozen or only fine-tune the last ConvC layer.



In Figure <ref>, we plot the average kappa of 10-independent runs as a function of the number of available training examples. For each run, we randomly sample desired training instances and train a model from scratch. Note that, we utilize the same instances for evaluating both supervised baseline and our proposed method. The fully-supervised baseline (blue curve) shows network performance when a model is trained only with the labeled data. The proposed self-supervised pre-training technique, in particular, the version with fine-tuning of the last ConvC layer, tremendously improved the performance. The difference in the performance between supervised and self-supervised feature learning is significant on MotionSense, UCI HAR, MobiAct, and HHAR datasets in low-data regime (i.e. with 2-10 labeled instances per class). More notably, we observe that pre-training helps more in a semi-supervised setting when the data are collected from a wide variety of devices; simulating a real-life setting. Finally, we highlight that a simple convolutional network is used in our experiments to show the feasibility of self-supervision from unlabeled data. We believe a deeper network trained on a bigger unlabeled dataset will further improve the quality of learned representations for the semi-supervised setting.



  §.§.§ Evaluating Knowledge Transferability


We have shown that representations learned by the self-supervised TPN consistently achieve the best performance as compared to other unsupervised/supervised techniques and also in a semi-supervised setting. As we have utilized the unlabeled data from the same data source for self-supervised pre-training, a next logical question that arises is can we utilize a different (yet similar) data source for self-supervised representation extraction and gain a performance improvement on a task of interest (also in a low-data regime)? In Table <ref>, we assess the performance of our unsupervised learned features across datasets and tasks by fine-tuning them on HAHR, UniMiB, UCI HAR, WISDM, and MotionSense datasets. For self-supervised feature learning, we utilized the unlabeled MobiAct dataset as it is collected from a diverse group of users that performed twelve activities; highest among other considered datasets both in terms of the number of users and activities. This makes MobiAct a suitable candidate to perform transfer learning as it encompasses all the activity classes in other datasets. Of course, we do not utilize activity labels in MobiAct for self-supervised representation learning. We begin by pre-training a network on MobiAct dataset and utilize the learned weights for initialization of an activity recognition model. Moreover, the latter model is trained in a fully-supervised manner on an entire training set of a particular dataset (e.g., UniMiB). In comparison with supervised training of the network (from scratch), the weights learned through our technique from a different and completely unlabeled data source improved the performance in all the cases. On WISDM and HHAR our results are 3 percentage points better in terms of kappa score. Similarly, on UniMiB we obtained 4 percentage points improvement over supervised model, i.e. kappa score increase from 0.781 to 0.821. 



Further, we determine the generalization ability in a low-data regime setting, i.e., when very few labeled data are attainable from an end-task of interest. We transfer self-supervised learned representations on the MobiAct dataset as initialization for an activity recognizer. The network is trained in a supervised manner on the available labeled instances of a particular dataset. Figure <ref> shows average kappa score of 10-independent runs of a fully-supervised (learned from scratch) and transferred models for 2, 5, 10, 20, 50, and 100 labeled instances. For each training run, the desired instances are randomly sampled, and for both techniques, the same instances are used for learning the activity classifier. In the majority of the cases, transfer learning improves the recognition performance especially when the number of labeled instances per class are very few, i.e. between 2 to 10. In particular, on HHAR the performance of a model trained with weights transfer is slightly lower in low-data setting but improves significantly as the number of labeled data points increases. We think it may be because of the complex characteristics of the HHAR dataset as it is particularly collected to show heterogeneity of devices (and sensors) having varying sampling rates and its impact on the activity recognition performance.





 §.§ Determining Representational Similarity

The previous experiments establish the effectiveness of self-supervised sensor representations for activity classification that are significantly better than unsupervised and on-par with fully-supervised approaches. The critical question that arises is whether the self-supervised representations are similar to those learned via direct supervision, i.e., with activity labels. The interpretability of the neural networks and deciphering of the learned representations have recently gained significant attention, especially, for images (see <cit.> for an excellent review). Here, to better understand the similarity of the extracted representation from TPN and the supervised network, we utilize singular vector canonical correlation analysis (SVCCA) <cit.>, saliency maps <cit.> and t-distributed stochastic neighbor embedding (t-SNE) <cit.>. 



  §.§.§ Insights on Representational Similarity with Canonical Correlation

The SVCCA allows for a comparison of the learned distributed representations across different networks and layers. It does so through identifying optimal linear relationships between two sets of multidimensional variates (i.e., neuron activation vectors) arising from an underlying process (i.e., a neural network being trained on a specific task) <cit.>. Figure <ref> provides a mean similarity of top 20 SVCCA correlation coefficients for all pairs of layers for a self-supervised (trained to predict transformations) and a fully-supervised network. We averaged 20 coefficients as SVCCA implicitly assumes that all CCA vectors are equally crucial for the representations at a specific layer. However, there is plenty of evidence that high-performing deep networks do not utilize the entire dimensionality of a layer <cit.>. Due to this, averaging over all the coefficients underestimates the degree of representational similarity. To apply SVCCA, we train both the networks as explained earlier and produce activations of each layer. For a layer, where the number of neurons is larger than the layer in comparison, we randomly sample neuron activation vectors to have comparable dimensionality. In Figure <ref> each grid entry represents a mean SVCCA similarity between two layers of different networks. We observe a high correlation among temporal convolution layers trained with two different methods across all the evaluated datasets. In particular, a strong grid-like structure emerges between the last layers of the networks, which is because those layers are learned from scratch with activity labeled data and result in identical representations. 





  §.§.§ Visualizing Salient Regions

To further understand the predictions produced by both models, we visualize saliency maps <cit.> for the highest-scoring class on randomly selected instances from the MotionSense dataset. Saliency maps highlight which time steps largely affect the output through computing gradient of the loss function with respect to each input time step. More formally, let x = [x_1, …, x_N] be an accelerometer sample of length N and C_θ(x) be the class probability produced by a network C_θ(.). The saliency score of each input element x_k indicating its influence on the prediction is calculated as: 

    S_k =  |∂ℒ/∂ x_k|

where ℒ is the negative log-likelihood loss of an activity classification network for an input example x.

Figure <ref> provides a saliency mapping of the same input produced by the two networks for a class with the highest score. To aid interpretability of the saliency score, we calculate a magnitude of each tri-axial accelerometer sample, effectively combining all three channels. The actual input is given in the top-most pane, the magnitudes with varying color intensity are shown in the bottom panes. The dark color illustrates the regions that contribute most to the network's prediction. We observe that the saliency maps of both self-supervised and fully-supervised networks hint towards similar regions that are crucial for deciding on the class label.

Interestingly, for the Sitting class instance both network mainly focus on a smaller region of the input with slightly more variation in the values. We think it could be because one thing that a network learns is to find periodic variations in the signal (such as peaks and slopes). Hence, it pays more attention even to slightest fluctuation, but it decides on the Sitting label as the signal remains constant (before and after minor changes) which is an entirely different pattern as compared to the instances of other classes. This analysis further validates the point that our self-supervised network learns generalizable features for activity classification.






  §.§.§ Visualization of High-Level Feature Space through t-SNE

t-SNE is a non-linear technique for exploring and visualizing multi-dimensional data <cit.>. It approximates a low-dimensional manifold of a high-dimensional counterpart through minimizing Kullback-Leibler divergence between them with a gradient-based optimization method. More specifically, it maps multi-dimensional data onto a lower dimensional space and discovers patterns in the input through identifying clusters based on the similarity of the data points. Here, the activations from global max-pooling layers (of both self-supervised and fully-supervised networks) with 96 hidden units are projected on to a 2D space. Figure <ref> provides the t-SNE embeddings showing high semantic relevance of the learned features for various activity classes. We notice that the self-supervised features largely correspond to those learned with the labeled activity data. Importantly, the clusters of data points across two feature learning strategies are similar, e.g. in UCI HAR, the activity classes like Upstairs, Downstairs and Walking are grouped. Likewise, in HHAR, the data points for Walking, Upstairs, and Downstairs are close-by as opposed to others in the embeddings of both networks. Finally, it is important to note that t-SNE is an unsupervised technique which does not use class labels; the activity labels are just used for final visualization. 






§ RELATED WORK


Deep learning methods have been successfully used in several applications of ubiquitous computing,  pervasive intelligence, health, and well-being <cit.> and eliminate the need of hand-crafted feature engineering. Convolutional and recurrent neural networks have shown dominant performance in solving numerous high-level recognition tasks from temporal data such as activity detection and stress recognition <cit.>. In particular, CNNs are becoming increasingly popular in sequence (or time-series) modeling due to their ability of weight sharing, translation invariance, scale separation and localization of filters in space and time <cit.>. In fact, (1D) temporal CNNs are now widely used in the area of HAR (see <cit.> for a detailed review), but the prior works are mostly concerned with supervised learning approaches. The training of deep networks requires a huge (carefully) curated dataset of labeled instances, which in several domains is infeasible due to required manual labeling effort or can only be possible on a small-scale in a controlled lab environment. This inherent limitation of the fully-supervised learning paradigm emphasizes the importance of unsupervised learning to leverage a large amount of unlabeled data for representation learning <cit.> that can be easily acquired in a real-world setting. 

Unsupervised learning has been well-studied in the literature over the past years. Before the era of end-to-end learning, manual feature design strategies <cit.> such as those that employ statistical measures have been used with clustering algorithms to discover a latent group of activities <cit.>. Although deep learning techniques have almost entirely replaced hand-crafted feature extraction with directly learning rich features from data, representation learning still stands as a fundamental problem in machine learning (see <cit.> for an in-depth review). The classical approaches for unsupervised learning include autoencoders <cit.>, restricted Boltzmann machines <cit.>, and convolutional deep belief networks <cit.>. Another emerging line of research for unsupervised feature learning (also studied in this work), which has shown promising results and does not require manual annotations, is self-supervised learning <cit.>. These methods exploit the inherent structure of the data to acquire a supervisory signal for solving a pretext task with reliable and widely used supervised learning schemes. 

Self-supervision has been actively studied recently in the vision domain, and several surrogate tasks have been proposed for learning representations from static images, videos, sound, and in robotics <cit.>. For example, in images and videos,  spatial and temporal contexts, respectively, provide forms of rich supervision to learn features. Similarly, colorization of gray-scale images <cit.>, rotation classification <cit.>, odd sequence detection <cit.>, frame order prediction <cit.>, learning the arrow of time <cit.>, audio-visual correspondence <cit.> and synchronization <cit.> are some of the recently explored directions of self-supervised techniques. Furthermore, multiple such tasks are utilized together in a multi-task learning setting for solving diverse visual recognition problems <cit.>. These self-supervised learning paradigms have shown to extract high-level representations that are on par with those acquired through fully-supervised pre-training techniques (e.g., with ImageNet labels) and they tremendously help with transfer and semi-supervised learning scenarios. Inspired from this research direction, we explore multi-task self-supervision for learning representations from sensory data through utilizing transformations of the signals. 

Some earlier works on time-series analysis have explored transformations to exploit invariances either through architectural modifications (to automatically learn task-relevant variations) or less commonly with augmentation and synthesis. In <cit.> task-specific transformations (such as added noise and rotation) are applied to wearable sensor data to augment and improve the performance of Parkinson's disease monitoring systems. Saeed et al. <cit.> utilized an adversarial autoencoder for class-conditional (multimodal) synthetic data generation for the behavioral context in a real-life setting. Moreover, Oh et al. <cit.> focused on learning invariances directly from clinical time-series data with specialized neural network architecture. Razavian et al. <cit.> used convolution layers of varying size filters to capture different resolutions of temporal patterns. Similarly, through additional pre-processing of the original data Cui et al. <cit.> used transformed signals as extra channels to the model for learning multiscale features. To summarize, these works are geared towards learning supervised networks for specific tasks through exploiting invariances, but they do not address the topics of semi-supervised and unsupervised learning.

To the best of our knowledge, the work presented here is the first attempt of self-supervision for sensor representation learning, in particular for HAR. Our work differs from the aforementioned works in several ways as we learn representations with self-supervision from completely unlabeled data and without using any specialized architecture. We show that when training a CNN to predict generally known (time-series) transformations <cit.> as a surrogate task, the model can learn features that are on a par with a fully-supervised network and far better than unsupervised pre-training with an autoencoder. We also demonstrate that the learned representations from a different (but related) unlabeled data source can be successfully transferred to improve the performance of diverse tasks even in the case of semi-supervised learning. In terms of transfer learning, our approach also differs significantly from some earlier attempts <cit.> that were concerned with features transferability from a fully-supervised model learned from inertial measurement units data, as our approach utilizes widely available smartphones and does not require labeled data. Finally, the proposed technique is also different from previously studied unsupervised pre-training methods such as autoencoders <cit.>, restricted Boltzmann machines <cit.> and sparse coding <cit.> as we employ an end-to-end (self) supervised learning paradigm on multiple surrogate tasks to extract features. 



§ CONCLUSIONS AND FUTURE WORK

We present a novel approach for self-supervised sensor representation learning from unlabeled data with a focus on smartphone-based human activity recognition (HAR). We train a multi-task temporal convolutional network to recognize potential transformations that may have been applied to the raw input signal. Despite the simplicity of the proposed self-supervised technique (and the network architecture), we show that it enables the convolutional model to learn high-level features that are useful for the end-task of HAR. We exhaustively evaluate our approach under unsupervised learning, semi-supervised learning and transfer learning settings on several publicly available datasets. The performance we achieve is consistently superior to or comparable with fully-supervised methods, and it is significantly better than traditional unsupervised learning methods such as an autoencoder. Specifically, our self-supervised framework drastically improved the detection rate under semi-supervised learning setting, i.e., when very few labeled instances are available for learning. Likewise, the transferred features learned from a different but related unlabeled dataset (MobiAct in our case), further improves the performance in comparison with merely training a model from scratch. Notably, these transferred representations even boost the performance of an activity recognizer in semi-supervised learning from a dataset (or task) of interest. Finally, canonical correlation analysis, saliency mapping, and t-SNE visualizations show that the representations of the self-supervised network are very similar to those learned by a fully-supervised model that is trained in an end-to-end fashion with activity labels. We believe that, through utilizing more sophisticated layers and deep architectures, the presented approach can further reduce the gap between unsupervised and supervised feature learning. 

In this work, we provided the basis for self-supervision of HAR with smartphones through a few labeled data. In the Internet of Things era, there are many exciting opportunities for future works in related areas, such as in industrial manufacturing, electrical grid, smart wearable technologies, and home automation. In particular, we believe that self-supervision is of immense value for automatically extracting generalizable representations in domains, where labeled data are challenging to acquire, but unlabeled data are available in vast quantities. We hope that the presented perspective of self-supervision inspires the development of additional approaches, specifically for the selection of appropriate auxiliary tasks (based on domain expertise) that enables the network to learn useful features to solve a particular problem. Likewise, combining self-supervision with network architecture search is another crucial area of improvement that will automate the process of optimal model discovery. Another exciting avenue for future research is evaluating self-supervised representations on an imbalanced activity dataset, where, the number of classes are high and collecting a few labeled data points for each activity class is not feasible. Finally, evaluation in a real-world setting (application deployed on real devices) is of prime importance to further understand the aspects that need improvement concerning computational, energy and, labeled data requirements.



This work is funded by SCOTT (www.scott-project.eu) project. It has received funding from the Electronic Component Systems for European Leadership Joint Undertaking under grant agreement No 737422. This Joint Undertaking receives support from the European Union's Horizon 2020 research and innovation programme and Austria, Spain, Finland, Ireland, Sweden, Germany, Poland, Portugal, Netherlands, Belgium, Norway. 

We thank Prof. Peter Baltus for a helpful discussion and anonymous reviewers for their insightful comments and suggestions. Various icons used in the figures are created by Anuar Zhumaev, Korokoro, Gregor Cresnar, Becris, Hea Poh Lin, AdbA Icons, Universal Icons, and Baboon designs from the Noun Project. 



plainnat





§ APPENDIX










