This work addresses the long-standing problem of robust event localization in the presence of temporally of misaligned labels in the training data. We propose a novel versatile loss function that generalizes a number of training regimes from standard fully-supervised cross-entropy to count-based weakly-supervised learning. Unlike classical models which are constrained to strictly fit the annotations during training, our soft localization learning approach relaxes the reliance on the exact position of labels instead. Training with this new loss function exhibits strong robustness to temporal misalignment of labels, thus alleviating the burden of precise annotation of temporal sequences. We demonstrate state-of-the-art performance against standard benchmarks in a number of challenging experiments and further show that robustness to label noise is not achieved at the expense of raw performance. Figure 1: Temporal localization under label misalignment. Models are trained with noisy labels that differ from the actual ground-truth, while the final inference objective is the precise localization of events. The surge of deep neural networks Schmidhuber, 2015) has accentuated the evergrowing need for large corpora of data (Banko & Brill, 2001; Halevy et al., 2009) . The main bottleneck for the efficient creation of datasets remains the annotation process. Over the years, while new labeling paradigms have emerged to alleviate this issue (e.g., crowdsourcing (Deng et al., 2009) or external information sources (Abu-El-Haija et al., 2016) ), these methods have also highlighted, and emphasized, the prevalence of label noise. Deep neural networks are unfortunately not immune to these perturbations as their intrinsic ability to memorize and learn label noise (Zhang et al., 2017) can be the cause of training robustness issues and poor generalization performance. In this context, the development of models robust to label noise is essential. This work tackles the problem of precise temporal localization of events (i.e., determining when and which events occur) in sequential data (e.g. time series, video or audio sequences) despite only having access to poorly aligned annotations for training (see Figure 1 ). This task is characterized by the discrepency between the precision required of the predictions during inference and the noisiness of the training labels. Indeed, while models are trained on inaccurate data, they are evaluated on their ability to predict event occurences as precisely as possible with respect to the ground-truth. In such a setting, effective models have to infer event locations more accurately than the labels they relied on for training. This requirement is particularly challenging for most classical approaches that are designed to learn localization by strictly mimicking the provided annotations. Indeed, as the training labels themselves do not accurately reflect the event location, focusing on replicating these unreliable patterns is incompatible with the overall objective of learning the actual ground-truth. These challenges highlight the need for more relaxed learning approaches that are less dependent on the exact location of labels for training. The presence of temporal noise in localization tasks is ubiquitous given the continuous nature of the perturbation, in contrast to classification noise where only a fraction of the samples are misclassified. Temporal labeling is further characterized by an inevitable trade-off between annotation precision and time investment. For instance, while a coarse manual transcription of a minute of complex piano music might be achieved within a moderate time frame, a millisecond precision requirement -a common assumption for deep learning models -significantly increases the annotation burden. In this respect, models alleviating the need for costly annotations are key for a wide and efficient deployment of deep learning models in temporal localization applications. This work introduces a novel model-agnostic loss function that relaxes the reliance of the learning process on the exact temporal location of the annotations. This softer learning approach inherently makes the model more robust to temporally misaligned labels. Contributions This work: a) proposes a novel loss function for robust temporal localization under label misalignment, b) presents a succinct analysis of the loss' properties, c) evaluates the robustness of state-of-the-art localization models to label misalignment, and d) demonstrates the effectiveness of the proposed approach in various experiments. In this work, we have shown how relaxing annotation requirements (i.e., weakening the model's reliance on the exact location of events) not only has the practical benefit of alleviating annotation efforts but, more importantly, leads to a model that is robust to temporal noise without compromising performance on clean training data. This contrasts with traditional approaches which attempt to strictly mimic the annotations, leading to poor predictions when training with noisy labels. We have demonstrated these claims on a number of classical challenging tasks, in which our SoftLoc loss exhibits state-of-the-art performance. The proposed loss function is agnostic to the underlying network and hence can be used as a loss replacement in almost any recurrent architecture. The versatility of the model can find applications in a wide array of tasks, even beyond temporal localization.
This work introduces a novel loss function for the robust training of temporal localization DNN in the presence of misaligned labels. A new loss for training models that predict where events occur in a training sequence with noisy labels by comparing smoothed label and prediction sequence.