Self-supervision, in which a target task is improved without external supervision, has primarily been explored in settings that assume the availability of additional data. However, in many cases, particularly in healthcare, one may not have access to additional data (labeled or otherwise). In such settings, we hypothesize that self-supervision based solely on the structure of the data at-hand can help. We explore a novel self-supervision framework for time-series data, in which multiple auxiliary tasks (e.g., forecasting) are included to improve overall performance on a sequence-level target task without additional training data. We call this approach limited self-supervision, as we limit ourselves to only the data at-hand. We demonstrate the utility of limited self-supervision on three sequence-level classification tasks, two pertaining to real clinical data and one using synthetic data. Within this framework, we introduce novel forms of self-supervision and demonstrate their utility in improving performance on the target task. Our results indicate that limited self-supervision leads to a consistent improvement over a supervised baseline, across a range of domains. In particular, for the task of identifying atrial fibrillation from small amounts of electrocardiogram data, we observe a nearly 13% improvement in the area under the receiver operating characteristics curve (AUC-ROC) relative to the baseline (AUC-ROC=0.55 vs. AUC-ROC=0.62). Limited self-supervision applied to sequential data can aid in learning intermediate representations, making it particularly applicable in settings where data collection is difficult. Many problems involving sequential data, such as machine translation, sentiment analysis, and mortality prediction, are naturally framed as sequence-level tasks (Harutyunyan et al., 2017; Hassan et al., 2018; Radford et al., 2017) . Sequence-level tasks map a sequence of observations x 0:T to a single label y. Learning this mapping is often made challenging due to a high-D (dimension) low-N (number of samples) setting (Nasrabadi, 2007) . Such problems are particularly prevalent in healthcare tasks, which often involve limited quantities of labeled data captured at a high temporal resolution (e.g., electrocardiogram waveforms). In high-D low-N settings, researchers have had success with transfer learning techniques, by leveraging additional data to learn intermediate representations that are then used in the target task. When additional data are unavailable, it may be possible to improve the intermediate learned representation of the data with respect to the target task by considering additional tasks intrinsic to the data. In particular, we hypothesize that the structure of sequential data provides a rich source of innate supervision. For example, signal reconstruction or forecasting could improve the intermediate representation by capturing the underlying data-generating process. Such approaches are examples of self-supervision, where labels are derived from the input (as opposed to external sources). In this paper, we show that leveraging the sequential structure of the data at-hand can lead to improved performance on sequence-level tasks (i.e., the target task). More specifically, by considering self-supervised auxiliary tasks (e.g., signal reconstruction), in addition to the sequence-level task, one can learn useful intermediate representations of the data. Past work investigating self-supervision for sequential data has focused on full-signal reconstruction (Dai & Le, 2015) , and to a lesser extent forecasting (Ramachandran et al., 2016) . Building on past work, we examine the utility of self-supervision on sequential data when additional data are unavailable, and we propose new types of self-supervision tasks. We refer to this approach as 'limited self-supervision. ' We limit the self-supervision to the data at-hand, and focus on self-supervised auxiliary tasks relevant to sequential data ordered by time (i.e., time-series data). Our main contributions are as follows: • We demonstrate the efficacy of the proposed limited self-supervision framework for improving performance across datasets/tasks with no additional data. • We compare the utility of several different existing forms of self-supervision in our limiteddata setting, identify consistent trends across supervision types, and demonstrate the utility of combining multiple different forms of self-supervision. • We propose a new form of self-supervision, piecewise-linear autoencoding, that trades off fine-grained signal modeling and long-term dependency propagation. We demonstrate that this is the best form of limited self-supervision across all tasks. Our work suggests that there is a wide range of time-series and sequence classification tasks where limited self-supervision could improve performance. It also shows the value of including multiple, simultaneous streams of auxiliary self-supervision. Our findings present a methodological contribution, in the form of a useful new type of self-supervision, piecewise-linear autoencoding. Further, our empirical findings on when and how auxiliary tasks help can inform future work in developing self-supervision techniques. In this paper, we introduced a limited self-supervised framework, in which we sought to improve sequence-level task performance without additional data. By jointly training our target task with auxiliary self-supervised tasks, we demonstrated small but consistent improvements across three different sequence classification tasks. Our novel piecewise-linear autoencoding task emerged as the most useful auxiliary task across all datasets. In contrast, forecasting, which presents an intuitively appealing form of self-supervision, led to the smallest improvements. , continuing untilx 0:T is fully generated. To provide a shorter path for gradient flow, we decode in reverse order, generatinĝ x T :0 instead ofx 0:T (Goodfellow et al., 2016) .
We show that extra unlabeled data is not required for self-supervised auxiliary tasks to be useful for time series classification, and present new and effective auxiliary tasks. This paper proposes a self-supervised method for learning from time series data in healthcare settings via designing auxilliary tasks based on data's internal structure to create more labeled auxilliary training tasks. This paper propose an approach for self-supervised learning on time series.