In this paper, we find that by designing a novel loss function entitled, ''tracking loss'', Convolutional Neural Network (CNN) based object detectors can be successfully converted to well-performed visual trackers without any extra computational cost. This property is preferable to visual tracking where annotated video sequences for training are always absent, because rich features learned by detectors from still images could be utilized by dynamic trackers. It also avoids extra machinery such as feature engineering and feature aggregation proposed in previous studies. Tracking loss achieves this property by exploiting the internal structure of feature maps within the detection network and treating different feature points discriminatively. Such structure allows us to simultaneously consider discrimination quality and bounding box accuracy which is found to be crucial to the success. We also propose a network compression method to accelerate tracking speed without performance reduction. That also verifies tracking loss will remain highly effective even if the network is drastically compressed. Furthermore, if we employ a carefully designed tracking loss ensemble, the tracker would be much more robust and accurate. Evaluation results show that our trackers (including the ensemble tracker and two baseline trackers), outperform all state-of-the-art methods on VOT 2016 Challenge in terms of Expected Average Overlap (EAO) and robustness. We will make the code publicly available. Visual tracking is a fundamental computer vision task, and can be used to predict the trajectory of objects in a video sequence. It is the building block for applications in self-driving vehicles, robotics and automatic surveillance.The richness of feature representations is crucial to the success of Convolutional Neural Networks (CNNs) in many computer vision tasks, such as image classification and object detection. This property also motivates researchers to adopt CNNs as strong feature extractors in the setting of visual tracking BID21 BID14 BID10 BID15 BID16 . MDNet BID15 ) is a famous CNN based tracker which achieved tremendous success on VOT 2015 BID11 . One major drawback is that MDNet is trained with annotated video sequences provided by previous challenges. Therefore, MDNet is lack of generalizability to a diversity of tracking targets. Actually, there is no such dataset with enough labeled video sequences specialized to train trackers.Previously, a series of trackers BID21 BID14 BID10 attempted to convert CNN based classifiers trained on large scale image classification datasets (for example ImageNet (Russakovsky et al., 2015) ) to tracking. In our opinion, these approaches relied heavily on feature engineering and feature aggregation. That would result in more time and computational cost. Another feasible idea might be to convert pre-trained object detectors to trackers. Region Proposal Network (RPN) BID17 is a state-of-the-art object detector and achieves tremendous success. It could simultaneously provide strong features for classification and bounding box regression. After a careful exploration, we find the internal structure of RPN is highly relevant and possesses strong potential for discriminative trackers.Visual trackers usually require to be trained online to learn specific appearances of targets. However, ground truths are quite limited. Data augmentation is widely employed to enlarge training samples.As to positive training samples, they are a batch of randomly cropped patches, which are subject to a two-dimensional Gaussian distribution, from the entire image around the ground truth. However, this sampling strategy usually contains some background pixels at the border of sampled images. These background pixels are noisy to train a well-performed tracker. With time elapse, accumulative noises will make the tracker get even worse. Due to above problem, if RPN is directly employed in tracking without any modification, different pixels will be treated equally, and the tracker will perform poor. In our opinion, we think pixels in different positions should be treated discriminately. Generally speaking, centric pixels of sampled images are more confident to cover the target object than border ones. Also corresponding feature points of centric pixels are more likely to be positive. In order to realize treating different feature points discriminatively, we explore the top layer feature maps of RPN, design a series of matching strategies, and evaluate them quantitatively and qualitatively. Due to none of the matching strategies is the best, we propose the tracking loss composed of two better performed matching strategies. Such method proves to be effective to take advantage of pros and offset cons of each matching strategy. Tracking loss bridges the gap between object detection and visual tracking in an unconventional loss viewpoint which will not bring extra computation.RPN is a relative large network which would limit tracking speed. Basing on knowledge distillation theory BID9 , we also propose a network compression method to trim the RPN. Experiments show that tracking loss would remain highly effective even if the network is drastically compressed.Furthermore, we adopt a carefully designed tracking loss ensemble which is consist of four types of loss functions. Evaluation results show that tracking loss ensemble could perform much better. Our trackers (including the ensemble tracker and two baseline trackers) outperform all state-of-the-art methods on VOT 2016 Challenge in terms of Expected Average Overlap (EAO) and robustness.The contributions can be summarized as follows,• We propose a novel tracking loss which successfully converts a pre-trained object detector RPN to a state-of-the-art visual tracker without extra computational cost. It shed new lights on transferring pre-trained detection network to new tasks where labeled data is very scarce.• We propose a network compression method to speed up our tracker. Meanwhile , it proves that tracking loss is a robust way to convert detection to tracking and independent of network variations. Furthermore , we implement a tracking loss ensemble with four types of loss functions to further promote tracking performance.• Our two baseline trackers and the ensemble tracker outperform all state-of-the-art trackers on VOT 2016 BID12 in terms of EAO and robustness. In the paper, we propose a novel tracking loss to convert an object detector to a well-performed robust tracker without extra time or computational consuming modifications (e.g. feature engineering and feature aggregation). On the basis of inaccuracy of sampling, tracking loss fully exploits the internal structure of top layer features of the detection network to treat feature points discriminatively. Such structure could provide high-quality discrimination and tight bounding boxes in tracking. Our network compression yields 4 times speedup. That also proves tracking loss is robust to network variations. We further employ tracking loss ensemble to promote the performance. Evaluation results on VOT 2016 show that two baseline tracking loss trackers and the tracking loss ensemble tracker outperform all state-of-the-art trackers in terms of EAO and robustness.
We successfully convert a popular detector RPN to a well-performed tracker from the viewpoint of loss function.