# Visual Object Tracking with Discriminative Filters and Siamese Networks: A
Survey and Outlook

Sajid Javed, Martin Danelljan, Fahad Shahbaz Khan, , Muhammad Haris Khan,
Michael Felsberg, , and Jiri Matas S. Javed is with the EECS department,
Khalifa University of Science and Technology, P.O Box : 127788, Abu Dhabi,
UAE. (email: sajid.javed@ku.ac.ae). M. Danelljan is with the Computer Vision
Lab, Dept. of Information Technology and Electrical Engineering, ETH Zürich,
Switzerland. F. S. Khan and M. H. Khan are with computer vision department,
MBZUAI, Abu Dhabi, UAE. M. Felsberg and F. S. Khan are with computer vision
laboratory, Linköping University, Sweden. J. Matas is with Center for Machine
Perception, Czech Technical University, Prague.

###### Abstract

Accurate and robust visual object tracking is one of the most challenging and
fundamental computer vision problems. It entails estimating the trajectory of
the target in an image sequence, given only its initial location, and
segmentation, or its rough approximation in the form of a bounding box.
Discriminative Correlation Filters (DCFs) and deep Siamese Networks (SNs) have
emerged as dominating tracking paradigms, which have led to significant
progress. Following the rapid evolution of visual object tracking in the last
decade, this survey presents a systematic and thorough review of more than 90
DCFs and Siamese trackers, based on results in nine tracking benchmarks.
First, we present the background theory of both the DCF and Siamese tracking
core formulations. Then, we distinguish and comprehensively review the shared
as well as specific open research challenges in both these tracking paradigms.
Furthermore, we thoroughly analyze the performance of DCF and Siamese trackers
on nine benchmarks, covering different experimental aspects of visual
tracking: datasets, evaluation metrics, performance, and speed comparisons. We
finish the survey by presenting recommendations and suggestions for
distinguished open challenges based on our analysis.

###### Index Terms:

Visual Object Tracking, Discriminative Correlation Filters, Siamese Networks.

##  1 Introduction

Visual Object Tracking (VOT) is one of the fundamental open problems in
computer vision. The task is to estimate the trajectory and state of a target
in an image sequence. VOT has a wide range of applications, including
autonomous driving, robotics, intelligent video surveillance, sports analytics
and medical imaging, where it typically plays an important role within large
intelligent systems. Given the initial state of any arbitrary target object,
the main challenge in VOT is to learn an appearance model to be used when
searching for the target object in subsequent frames. In recent years, VOT has
received considerable attention, much thanks to the introduction of a variety
of tracking benchmarks such as, TackingNet [104], VOT2018 [68], and GOT-10K
[60]. Despite the recent progress, VOT is still an open research problem and
is perhaps more active than ever [40, 79].

Figure 1: Performance improvements from 2014 to 2020 on the popular VOT
benchmark series. Tracking quality is measured in terms of Expected Average
Overlap (EAO). We show DCF and Siamese trackers that significantly advanced
the SOTA. The representative trackers are: SAMF [81], DSST [26], KCF [53],
SRDCF [29], SiamFC [4], CCOT [22], CSR-DCF [95], ECO [24], DaSiamRPN [171],
DRT [120], SiamMask [140], DiMP [5], D3S [94] and SiamAttn [158].

The core challenge in generic object tracking111Generic object tracking refers
to the problem, where the tracked object is not known a priori and it is not
constrained to be from a specific class. is to learn an appearance model of an
arbitrary target object online, given only its initial state. Several real-
world factors, complicates learning an accurate appearance model. For
instance, the target object may undergo a partial or full occlusion, scale
variation, and deformation. Moreover, there are environmental factors,
including illumination changes and motion blur, which influence the appearance
of the target. Another factor is that scenes often include objects or
background structures having similar appearance that can easily be confused
with the target itself [142]. To address these challenges, a plethora of
trackers have been proposed in the literature that have contributed to advance
the State-Of-The-Art (SOTA) in tracking.

In the past decade, Discriminative Correlation Filters (DCFs) and deep Siamese
Networks (SNs) have been the two most prominent paradigms for VOT [40, 4, 127,
52]. In DCF-based tracking, a correlation filter is trained online on the
region of interest by minimizing a least squares loss. The target is then
detected in consecutive frames by convolving the trained filter via the fast
Fourier transform (FFT) [10, 53]. In the deep Siamese tracking framework, an
embedding space is learned offline by maximizing the distance between the
target and background appearance, while minimizing the distance between the
two patches from the target itself. The SNs consists of two identical sub-
branches: one for the target template and the other for the target search
region. The network takes both template and search regions as inputs and
outputs the local similarity with the target for each location in the search
region. The SNs were initially proposed for the signature verification task
[12] and later explored for visual tracking by Tao et al. [127] and Bertinetto
et al. [4]. With the design of DCFs [54, 160, 30] and SNs [4, 52, 127], the
tracking community has been largely focusing on these two paradigms in recent
years as these two frameworks have significantly improved the tracking
performance on several datasets such as the performance improvements on VOT
datasets [47, 68] as shown in Fig. 1.

In this work, we present a systematic review of popular DCF and Siamese-based
tracking paradigms. Both paradigms share the same objective, that is to learn
an accurate target appearance model that can effectively discriminate the
target object from the background. While emerging from different underlying
paradigms for addressing the aforementioned objective, the advent of deep
learning has brought several important similarities and common challenges to
these two paradigms. For instance, (i) Feature Representation: Both paradigms
exploit different features representations to estimate target translations and
scale variations. Leveraging deep feature representations extracted from pre-
trained networks is a recent trend shared by both paradigms. However, the
choice of deep architecture and feature hierarchies is still an open problem
in both tracking paradigms. (ii) Target State Estimation: The core formulation
of both DCF and Siamese trackers only addresses how to estimate the
translation of the target object. Hence, neither paradigm provides an explicit
method for estimating the full target state, parametrized by e.g. a bounding
box, which is crucial in most applications. (iii) Offline Training: While
initially only Siamese trackers benefited from end-to-end offline training,
recent DCF trackers [23, 5, 25] also leverage large-scale offline learning,
integrating it with efficient and differentiable online learning modules for
robust and accurate tracking.

While having common attributes, these two popular paradigms also have specific
issues. For instance, (i) Boundary Artifacts: DCF-based trackers generally
exploit the periodic assumption of training samples for learning an online
classifier, which introduces undesirable boundary effects that severely
degrade the quality of the target model. (ii) Optimization: The loss function
minimization also introduces challenges in DCF-based trackers, especially when
target-specific constraints, such as spatial or temporal, etc., are
regularized within the regression loss. (iii) Online Model Adaptability: When
the target appearance undergoes changes due to variation in lighting
conditions or fast motion etc., the learned model is expected to cope with
these variations. The DCF trackers possess the ability to update the
appearance model over time through the loss function. On the other hand,
Siamese trackers do not inherent such a mechanism for online model update.
Therefore, online adaptability is an important issue in Siamese trackers. In
the paper, we systematically review the popular DCF and Siamese tracking
paradigms with the following contributions:

  * •

We distinguish and thoroughly review the shared as well as specific open
challenges in both tracking paradigms.

  * •

We present an extensive background theory of both DCF and Siamese tracking
core formulations.

  * •

We present a comprehensive overview of more than 90 DCF and Siamese-based
trackers in the literature, presenting a comparison on nine tracking
benchmarks.

  * •

Based on our analysis, we present our recommendations and suggestions for
specific open challenges.

To the best of our knowledge, we are the first to present a comprehensive
survey of the two most popular tracking paradigms (DCF and Siamese) in the
last decade, describing their respective background theory, specifying their
shared as well as specific open research issues along with providing a set of
recommendations and future directions.

The rest of this paper is organized as follows: Section 2 focuses on other
surveys on tracking. In Sections 3 and 4, we describe the core DCF and Siamese
tracking formulation, respectively. Then, we present a comprehensive overview
of DCF and Siamese-based trackers, respectively, by distinguishing open key
research issues. Afterwards, we provide an overview of the evolution of these
two paradigms into segmentation-based tracking frameworks. Experimental
evaluations are presented in Sections 5. We conclude our survey and provide
further research directions in Section 6.

##  2 Literature Review

In the literature, numerous survey studies on VOT have been published in the
past two decades [156, 152, 117, 78, 40, 132, 79, 38, 163]. In the first
survey, Yilmaz et al. presented systematic analysis of the whole tracking
procedure [156]. The tracking methods were categorized into: based on the
point or features correspondence, primitive geometric models, and contour
methods. Smeulders et al. presented an experimental survey of 19 different
online trackers that emerged from 1999 to 2012 with multiple evaluation
metrics on a newly proposed ALOV++ dataset [117]. Zhang et al. presented
sparse coding-based trackers survey in which the tracking methods were
categorized into sparse coding and sparse representation-based trackers [163].
Recently, Li et al. presented a survey of deep trackers and provided an
experimental comparison [78]. The trackers were classified and evaluated in
terms of network structure, network function, and network training. Fiaz et
al. reviewed SOTA DCFs and non DCF-based trackers, providing comparative study
based on the feature extraction methods [40].

The main differences between this survey and previous VOT surveys [156, 78,
40, 163] are as follows. Unlike previous VOT surveys, our work focuses solely
on the two best-performing tracking paradigms, DCFs and SNs, in recent years.
We present an extensive background theory of both DCF and Siamese tracking
core formulations. We then provide an extensive overview of more than 90 DCF
and Siamese trackers and the evolution of these two paradigms to segmentation-
based tracking. While previous surveys are based on attributes-based taxonomy,
deep network structure and training or introducing a new dataset, we present a
comprehensive overview of DCF and Siamese trackers by distinguishing shared as
well as specific open research challenges in these two popular tracking
paradigms. Furthermore, to the best of our knowledge, we are the first to
extensively compare the performance of DCF and Siamese trackers on nine
popular visual tracking benchmarks.

##  3 Discriminative Correlation Filters

Discriminative correlation filters (DCFs) is a supervised technique for
learning a linear regressor. In recent years, DCF-based trackers have
demonstrated excellent performance on multiple tracking benchmarks. The key to
the DCF success is the computationally efficient approximation to dense
sampling achieved by circularly shifting the training samples, which allows
the fast Fourier transform (FFT) to be employed when learning and applying the
correlation filter. By utilizing the properties of the Fourier transform, a
DCF learns a correlation filter online to localize the target object in
consecutive frames by efficiently minimizing a least-squares output error. In
order to estimate the target location in the next frame, the learned filter is
then applied to the region of interest in which the location of the maximum
response estimates the target location. The filter is then updated in an
iterative manner by annotating a new sample with this estimate.

###  3.1 Standard Single-Channel DCF Formulation

In the standard single-channel DCF formulation, the aim is to learn a single
channel convolution or correlation filter $w$ from a set of training samples
$\\{(x_{j},y_{j})\\}_{j=1}^{m}$. In its most basic form [10], each training
sample $x_{j}$ consists of a grayscale feature map extracted from an image
region. All samples are assumed to have the same spatial size $N_{1}\times
N_{2}$. At each spatial location
$(n_{1},n_{2})\in\Omega:=\\{0,...,N_{1}-1\\}\times\\{0,...,N_{2}-1\\}$, we
thus have an intensity value $x_{j}(n_{1},n_{2})=x_{j}(n)\in\mathbb{R}$, where
$n=(n_{1},n_{2})$. The desired output $y_{j}$ is a scalar valued function over
the domain $\Omega$, which contains a label for each location in the sample
$x_{j}$. Typically, $y_{j}$ is set to a sampled Gaussian function with a
narrow peak that is centered on the target.

The DCF aims to learn a filter $w$, of the same spatial size $N_{1}\times
N_{2}$, such that $x_{j}*w\approx y_{j}$ for all samples $j$ in the dataset.
Importantly, the standard DCF formulation employs circular convolution,222Some
authors instead employ circular correlation, obtained by simply reflecting the
convolution filter $w$.

$x*w(n_{1},n_{2})=\sum_{l_{1}=0}^{N_{1}-1}\sum_{l_{2}=0}^{N_{2}-1}x\left((n_{1}-l_{1})_{N_{1}},(n_{2}-l_{2})_{N_{2}}\right)w(l_{1},l_{2}).$
(1)

It is computed by cyclically shifting the filter $w$ (or equivalently the
sample $x$) as ensured by the modulo operation $(a)_{N}=a\mod N$. Circular
convolution (1) can also be seen as first padding the sample $x$ by periodic
repetition, followed by regular convolution with the filter $w$.

Since the convolution operation is linear, we can express it in matrix form
$x*w=\mathbfcal{C}(x)w$. With slight abuse of notation in order to avoid
clutter, $w$ is interpreted as a vectorization of the original filter on the
right-hand side of the equation. The matrix $\mathbfcal{C}(x)$ contains a
highly regular structure, containing circulant block matrices, where the
blocks themselves are also circulant [57]. Each row in $\mathbfcal{C}(x)$
consists of a cyclicly shifted version of the original patch $x$. The
convolution theorem implies that a circulant matrix can be diagonalized as,

$\mathbfcal{C}(x)=\mathbfcal{F}\textrm{diag}(\hat{x})\mathbfcal{F}^{H}~{}\textrm{and}~{}\mathbfcal{C}(x)^{\top}=\mathbfcal{F}\textrm{diag}(\bar{\hat{x}})\mathbfcal{F}^{H},$
(2)

where $\mathbfcal{F}$ represents the Discrete Fourier Transform (DFT) matrix.
We denote the conjugate of $x$ by $\bar{x}$ and its Fourier transform
$\mathbfcal{F}^{H}x$ by $\hat{x}$, where H denotes the conjugate transpose.

In order to find the optimal filter $w$, we formulate a linear least-squares
regression problem. For convenience, we first concatenate all samples $j$ in
the dataset to get the data matrix $X$ and target vector y as,

$X=\begin{pmatrix}\mathbfcal{C}(x_{1})\\\ \vdots\\\
\mathbfcal{C}(x_{m})\end{pmatrix},\textbf{y}=\begin{pmatrix}y_{1}\\\ \vdots\\\
y_{m}\end{pmatrix}\,.$ (3)

The DCF is learned by minimizing the squared error objective $L(w)$ over the
samples of $X$ and their regression targets y as,

$L(w)=\lvert\lvert Xw-\textbf{y}\rvert\rvert^{2}+\lambda\lvert\lvert
w\rvert\rvert^{2},$ (4)

Here, $\lambda$ is a regularization parameter. As this is a linear least
squares problem, the solution is given by the following closed form expression

$w=(X^{\top}X+\lambda I)^{-1}X^{\top}\textbf{y}\,.$ (5)

To compute weights of the filter $w$, we simply substitute (3) into (5)

$w_{m}=\Big{(}\sum_{j=1}^{m}\mathbfcal{C}(x_{j})^{H}\mathbfcal{C}(x_{j})+\lambda
I\Big{)}^{-1}\sum_{j=1}^{m}\mathbfcal{C}(x_{j})^{H}y_{j}.$ (6)

Figure 2: The tracking pipeline of a classical discriminative correlation
filter.

Since the matrix $\mathbfcal{C}(x_{j})$ is circulant, it can be efficiently
diagonalized using Eq. (2) as,

$w_{m}=\mathbfcal{F}\Big{(}\sum_{j=1}^{m}\textrm{diag}(\hat{x}_{j})^{H}\textrm{diag}(\hat{x}_{j})+\lambda
I\Big{)}^{-1}\sum_{j=1}^{m}\textrm{diag}(\hat{x}_{j})^{H}\hat{y}_{j}.$ (7)

Note that the matrix to be inverted in (7) is diagonal, which can be computed
by a simple element-wise division. We can further simplify the expression in
terms of its Fourier coefficients of the filter by multiplying with
$\mathbfcal{F}^{H}$ from the left, leading to a particularly simple and
efficient formula

$\hat{w}_{m}=\frac{\sum_{j=1}^{m}\bar{\hat{x}}_{j}\hat{y}_{j}}{\sum_{j=1}^{m}\bar{\hat{x}}_{j}\hat{x}_{j}+\lambda}.$
(8)

The resulting formula contains only element-wise operations and DFTs. It is
hence computed in $\mathcal{O}(N\log N)$ time by exploiting the FFT algorithm,
where $N=N_{1}N_{2}$ denotes total number of pixels within a sample.

###  3.2 Standard Multi-channel DCF Formulation

In DCFs, multi-channel features, such as the RGB image representation of
sample $x_{j}$, offer a much richer representation compared to a single-
channel feature (e.g. grayscale intensity). Here, we describe the multi-
channel DCFs formulation. In the literature, there are two general strategies
to integrate multi-channel features, here termed early and late fusion.

Early Fusion: The first common scheme is known as early fusion in which the
multiple channels are directly concatenated and then a multi-channel DCF
classifier is trained. To generalize (8) to multi-channel DCFs, we let $x$ be
composed of several feature channels $x^{d}:\Omega\rightarrow\mathbb{R}$,
where $d\in\\{1,...,D\\}$ is the channel index. In other words, at each
spatial location
$(n_{1},n_{2})\in\Omega:=\\{0,...,N_{1}-1\\}\times\\{0,...,N_{2}-1\\}$ we have
a $D$-dimensional feature vector
$x_{j}(n_{1},n_{2})=x_{j}(n)\in\mathbb{R}^{D}$. Similarly, we extend the
filter $w$ to be $D$-dimensional, denoting the individual filter channels with
$w^{d}$. The convolution between two multi-dimensional functions is defined by
simply summing the convolution responses for each individual channel. We can
learn the filter $w$ by minimizing the generalized objective,

$L(w)=\sum_{j=1}^{m}\alpha_{j}\left\|\sum_{d=1}^{D}x_{j}^{d}*w^{d}-y_{j}\right\|^{2}+\lambda\sum_{d=1}^{D}\lvert\lvert
w^{d}\rvert\rvert^{2}.$ (9)

Here, we have additionally introduced the per-sample importance weights
$\alpha_{j}\geq 0$.

The above multi-channel formulation of DCF is still a linear least squares
problem. It cannot be fully diagonalized by the DFT in the general case. There
are however two exceptions, where diagonal solutions are found. The first case
is the original MOSSE model in (8) and the other case occurs when there is
only one single training sample $m=1$, resulting in the following optimal
filter,

$\hat{w}^{d}=\frac{\bar{\hat{x}}^{d}\hat{y}}{\sum_{d=1}^{D}\bar{\hat{x}}^{d}\hat{x}^{d}+\lambda}.$
(10)

By again exploiting the FFT, the solution to the general multi-channel
objective (9) can be obtained by solving $N$ number of $D$-dimensional linear
systems. This has a computational complexity of $O(DN\log N+ND^{3})$, which is
substantially smaller than the direct solution in the spatial domain, which
requires $O(D^{3}N^{3})$ operations [55].

Late Fusion: Here, the aim is to compute the DCF on each single channel of the
sample $x_{k}$ and then aggregate all the filters to obtain the resulting
classifier. Let $\hat{w}^{d}$ be the estimated filter of the $d$-th feature
layer using (8), then $w=\sum_{d=1}^{D}\beta_{d}w_{d}$ is aggregated filter
obtained by summing up each DCF, where $\beta$ is the scaling factor used to
give relative importance to each of the feature layer.

###  3.3 Standard DCF Tracking Pipeline

For tracking, the DCF first learns the filter $w$ online and then it performs
tracking-by-detection. Once the target is tracked in the current frame, the
model is then learnt recursively. The block diagram of the DCF tracking
pipeline is shown in Fig. 2.

Target Detection: Let $m$ be the number of the current image frame, in which
we strive to localize the target. From the previous frame we are given the
filter $\hat{w}_{m-1}$ that has been recursively updated since the initial
frame. We extract an image patch $z$ centered at the predicted target
location. Here, $z$ has the same size $N_{1}\times N_{2}$ as the training
patches $x_{j}$. Using the convolution theorem, we then predict target scores
$s(n)$ at each location $n\in\Omega$ in $z$ by applying the learned filter,

$s=\sum_{d=1}^{D}z^{d}*w^{d}_{m-1}=\mathbfcal{F}^{-1}\Big{\\{}\sum_{d=1}^{D}\hat{z}^{d}\hat{w}_{m-1}^{d}\Big{\\}}\,.$
(11)

This computes the score function at all locations $n\in\Omega$. We can then
estimate the target location in frame $m$ as the maximizer of the target score
function $n^{*}=\operatorname*{argmax}_{n}s(n)$. In case we are confident
about our estimate, we can construct a new training sample $(x_{m},y_{m})$ by
extracting a patch $x_{m}$ centered at the estimated location $n^{*}$.

Model update: Another feature of (8) is that the filter can be easily updated
with new training samples. Let us regard the sample index $j$ as the frame
number. That is, sample $x_{j}$ was extracted from frame number $j$. In online
learning, it is common to employ a learning rate parameter $\gamma\in[0,1]$
which controls the speed at which the model is adapted to the new data. Eq.
(8) suggests updating the numerator $\\{w^{num}_{m}\\}$ and denominator
$\\{w^{den}_{m}\\}$ of the filter
$\hat{w}_{m}=\frac{w^{num}_{m}}{w^{den}_{m}}$ recursively as,

$\begin{split}\hat{w}_{m}^{num}=(1-\gamma)\hat{w}_{m-1}^{num}+\gamma\bar{\hat{x}}_{j}\hat{y}_{j},\\\
\hat{w}_{m}^{den}=(1-\gamma)\hat{w}_{m-1}^{den}+\gamma(\bar{\hat{x}}_{j}\hat{x}_{j}+\lambda).\end{split}$
(12)

To achieve the solution given by the recursive formula (12), we need to choose
the weights $\alpha_{j}=\gamma(1-\gamma)^{m-j}$ for $j>1$ and
$\alpha_{1}=(1-\gamma)^{m-1}$ in (9).

###  3.4 Open Issues in the Standard DCF Tracking Pipeline

While having important promising properties, the standard DCF framework
presents several challenges including feature representations, boundary
artifacts, optimization, and target state estimation when applied for the task
of generic object tracking. In the subsections below, we identify and discuss
these important challenges developing a DCF based tracking pipeline.

####  3.4.1 Feature Representations

In object tracking, a variety of visual features have been investigated in the
literature. Finding discriminative, yet invariant features is particularly
important when applying linear discriminative models, such as DCF, which are
restricted to finding a linear decision boundary. Handcrafted features [21,
39, 129], deep features [116, 13, 51], hybrid features, and end-to-end
learning of features [23] have been explored within the DCF-based tracking
framework. Next, we describe details about different types of features used in
the DCF-based trackers.

Handcrafted Features: Early DCF trackers such as, MOSSE [10] and CSK [54] have
exploited intensity features for object tracking. Other than intensity
features, local color and intensity histograms features are also utilized in
DCF trackers such as, RPAC [88], LCT+ [98], LCT [99], and CACF [103]. Simple
color representations, including RGB and LAB have been used for DCFs-based
trackers such as, STAPLE [3] (RGB), SCT [17] (RGB $+$ LAB), and ACFN [18] (RGB
$+$ LAB).

To achieve a more discriminative image representation, ACA [30] investigated
different color descriptors and proposed to use the Color Names (CN) features
[129] along with the intensity channel. The ACA tracker further introduced an
adaptive dimensionality reduction technique to compress the CN features,
thereby providing a tradeoff between speed and tracking performance. CN
features have also been employed in several subsequent DCFs-based trackers
such as, MCCT [136], MKCF [126], MUSTer [56], CSR-DCF [95], CCOT [22], ECO
[24], UPDT [7], AutoTrack [83], ARCF [61], GFS-DCF [145], RPCF [123], and DRT
[120].

Another popular handcrafted feature employed in DCF-based trackers is
Histogram of Oriented Gradients (HOG) [21]. HOG captures shape information by
collecting statistics of the image gradients. HOGs are formed in a dense image
grid of cells. Within the DCF paradigm, KCF [53] was the first tracker to
utilize HOG features for tracking. Several DCF trackers such as, MCCF [64],
CFLB [65], BACF [41], SRDCF [29], STRCF [76], RPCF [123], GFS-DCF [145], RPT
[82], RCF [119], LMCF [133], PTAV [36], StruckCF [87], CFAT [9], and LSART
[121] have utilized HOG features. These features have been the preferred
alternative among handcrafted methods due to its speed and effectiveness.
Further, HOG features have also been effectively combined with CN features to
utilize both shape and color information.

Deep Features: In recent years, deep learning has revolutionized many areas in
computer vision. Deep Convolutional Neural Networks (CNNs) have shown to be
particularly well suited for image related tasks [44]. They apply a sequence
of learnable convolutions and non-linear operations onto the image. However,
employing deep features for object tracking has proved challenging. This was
primarily due to the scarcity of training data for tracking in the initial
years of deep learning, as well as the high dimensionality of the features.
Therefore, many DCF-based trackers such as, HCF [96], HDT [106], CCOT [22],
ECO [24], ASRCF [20], and RPCF [123], employ deep CNNs that are pre-trained on
the ImageNet dataset [31] for image classification. Despite being trained for
classification, such deep representations are applicable to a wide range of
vision tasks [115].

Within the DCF framework, the deep features are extracted from the
convolutional layers. Some popular pre-trained deep networks, including VGG-19
[116], imagenet-vgg-m-2048 [13], VGG-16 [116], ResNet50 [51], and GoogleNet
[124] are used to extract deep feature representations. For instance, Ma et
al., incorporated the hierarchical deep convolutional features for visual
tracking [96]. The HDT tracker also employed deep features from six
convolutional layers of the same network [106]. The DeepSRDCF tracker used the
imagenet-vgg-m-2048 network and performed an analysis of the convolutional
feature maps for tracking [28], indicating the importance of the shallow
layer. Shallow layers contain low-level information at a high spatial
resolution, important for accurate target localization.

On the other hand, deeper-layer feature maps possess high-level invariance to
complex appearance changes, such as deformations and out-of-plane rotations.
Thus, deeper layers have the potential of improving tracking robustness, while
largely invariant to small translation and scale changes. Therefore, an
accurate strategy of fusing shallow and deeper convolutional layers within the
DCF framework has been a topic of interest. In CCOT, a continuous-domain
formulation of the DCF framework is proposed that enables the integration of
multi-resolution features [22]. The ECO investigates strategies to reduce the
computational cost of the CCOT and mitigates the risk of overfitting [24].
Other trackers, such as HDT [106], HCFTs [97], MCCT [136], MCPF [164], MCPFs
[165], LMCF [133], STRCF [76], TRACA [16], DRT [120], UPDT [7], and GFS-DCF
[145] integrate deep features using late fusion strategy. This strategy is to
train a classifier on each individual feature representation and then
aggregate feature response map.

End-to-End Features Learning: This scheme within the DCF framework has also
been explored. Here, the backbone networks, such as AlexNet [1], VGG-16 [116]
or ResNet50 [51], are used to optimize deep features on tracking datasets
[162, 128, 18]. Instead of relying on pre-trained networks, the task-specific
deep feature learning promises improved representations for the tracking
problem itself. Valmadre et al. proposed a CFNET that follows end-to-end
learning of correlation filters in an offline manner [128]. Other trackers
such as, CREST [118] and ACFN [18] also followed the same strategy in an
online manner. In these trackers, the objective is to improve the target
regression. These methods demonstrated comparable performance, as compared to
trackers that utilized deep features extracted from the pre-trained networks.
Recently, an end-to-end target scale estimation component is incorporated in
ATOM [23] while the discriminative strength of the classical DCF model is
improved in DiMP [5] and PrDiMP [25]. This recent trend of end-to-end features
learning in DCF trackers [23, 5, 25] has resulted in excellent tracking
performance on multiple benchmarks, paving the way to explore more
sophisticated end-to-end feature learning in DCF paradigm.

####  3.4.2 Boundary Artifacts

The standard convolution is effectively replaced by circular convolution (1)
in the DCF formulation to ensure the applicability of the DFT, resulting in
the formula (11) for evaluating the target predictions. This change may seem
minor and might even have gone unnoticed to some readers. However, the
circular convolution introduces unwanted boundary artifacts.

The most severe consequence of the circular convolution assumption arises in
the learning formulation (9). The fundamental notion of the DCF paradigm is to
train a filter $w$ that can discriminate the target from background image
regions. Due to the periodic effects, most of the original background content
is replaced by synthetic repetition of a smaller image patch. The model thus
sees fewer background samples during training, severely limiting its
discriminative power. Furthermore, due to the distortions caused by the
periodic repetitions, the predicted target scores are only accurate near the
center of the image patch. The size of the search area is therefore limited.
As traditionally performed in signal processing, DCF methods typically pre-
process the samples $x$ by multiplying them with a window function [10].
However, this technique does not attempt to solve the aforementioned problems
and only serves to smooth out the discontinuities at border regions. Several
solutions have been proposed in the literature to overcome the aforementioned
boundary artifacts problems. To this end, several approaches are proposed that
incorporate target-specific spatial, spatiotemporal, and smoothness
constraints within the DCF objective function [65, 29, 41, 76, 20]. Below, we
summarize some of these major developments.

Spatial regularization: In the SRDCF, Danelljan et al. proposed a spatially
regularized framework to control the spatial extend of the filter in order to
alleviate the boundary problem [29]. A spatial regularization component is
integrated into the multi-channel DCF formulation (9) as,

$L(w)=\sum_{j=1}^{m}\alpha_{j}\lvert\lvert\sum_{d=1}^{D}x_{j}^{d}*w^{d}-y_{j}\rvert\rvert^{2}+\lambda\sum_{d=1}^{D}\lvert\lvert
fw^{d}\rvert\rvert^{2},$ (13)

The spatial weight function $f:\Omega\rightarrow\mathbb{R}$ takes positive
values $f(n)>0$, penalizes the filter coefficients $w^{d}(n)$ based on their
spatial location $n$. By letting $f$ takes large values at background pixels
and small values inside the target region, background filter coefficients are
penalized. As a result, a compact filter $w$ focusing on the target region can
be learned, even for large image sample sizes. The spatial regularization
strategy has been employed in a variety of trackers, including ARCF [61],
ASRCF [20], and AutoTrack [83]. In order to improve the efficiency of the
SRDCF formulation, Li et al. proposed the STRCF [76] which only employs a
single training sample, and instead introduces a temporal regularization term
to integrate historic information. It has also become a popular baseline for
numerous works [83].

Constraint optimization: While the SRDCF [29] aims to penalize the filter
coefficients outside the target region, Kiani et al. propose to introduce hard
constraints [65, 41]. This strategy enforces that the filter coefficients
$w(n)$ are zero outside the target region. The resulting DCF formulation can
be expressed as follows by introducing a binary mask $P$,

$L(w)=\frac{1}{2}\sum_{j=1}^{m}\lvert\lvert y_{j}-(P\cdot
w)*x_{j}\rvert\rvert^{2}_{2}+\frac{\lambda}{2}\lvert\lvert
w\rvert\rvert^{2}_{2},$ (14)

Here, the element-wise product $P\cdot w$ effectively masks out the influence
of filter weights corresponding to background features. The resulting
optimization problem can be efficiently solved through iterative techniques
such as ADMM [65, 41]. The aforementioned target-specific constraints
investigated in (13) and (14) are usually fixed for different objects and they
do not vary during the tracking process. Recently, Dai et al. extended BACF
and SRDCF by introducing an adaptive regularization term [20].

Implicit methods: With the GFS-DCF, Xu et al. proposed a joint group feature
selection model that simultaneously learns three regularization terms
including spatial regularization for feature selection, channel regularization
for feature channel selection, and low-rank temporal regularization term to
enforce smoothness on the filter weights [145]. Mueller et al. proposed to
regularize the contextual information of each target patch [103]. In every
frame, CACF samples several context patches, which serve as negative samples.

Spatial formulation: Danelljan et al. and Bhat et al. proposed ATOM [23] and
DiMP [5] trackers. Both these trackers employ deep features in low resolution
(stride 16) in order to first coarsely, yet robustly, localize the target
object. Due to the coarse resolution and the resulting small target filter
size ($4\times 4$), Danelljan et al. found that the the filter can be directly
learned in the spatial domain using dedicated and efficient iterative solvers
[23]. This approach allows both ATOM and DiMP to completely circumvent the
boundary artifacts problem, since no periodic extension of the training
samples are performed.

Both the regularization-based (SRDCF [29], STRCF [76]) and constraint-based
(CFLB [65]/BACF [41]) formulations have seen large success and been employed
in a wide range of trackers. However, more recent deep learning methods (ATOM
[23]/DiMP [5]), have completely circumvented the boundary artifact problem by
directly optimizing the filter in the spatial domain. Thus, while the Fourier
domain is computationally attractive for high-resolution feature maps,
efficient spatial-domain optimization methods prevail for online learning when
using powerful low-resolution deep features. Recent works constituting the
current SOTA in DCF-based tracking [5, 25] therefore employ a purely spatial
formulation, which does not require additional strategies for alleviating
boundary effects. By further extending the filter to a multi-channel output,
the latter strategy has also demonstrated its use for segmentation [8].

####  3.4.3 Optimization

In the standard DCF formulation, inference is performed by computing the DFT
coefficients $\hat{w}$ using the least squares solution (8). However, when the
model grows more complex and advanced e.g., by introducing multiresolution
feature maps and target-specific constraints, such as spatial regularization,
and temporal regularizations, the model inference cannot be performed using
the simple least-square solution, as provided in (14). Since computational
efficiency is a crucial factor in most applications, these modifications
require alternative inference methods. Therefore, finding efficient and robust
inference schemes is a key problem in DCF-based tracking.

Model inference is performed by minimizing the multi-channel loss (9), which
forms the basis of the DCF framework. However, it does not allow for any
efficient closed-form solution. Therefore, many DCF trackers such as CACF
[103], CSK [54], KCF [53], MUSTer [56], SP-KCF [122], and CFAT [9] employ the
diagonalizable cases in both primal or dual domains to derive approximate
model inference schemes. These loss functions rely on the very restrictive
assumptions of a single feature channel $D=1$ and a single training sample
$m=1$, respectively. Moreover, these solutions cannot benefit from the
aforementioned additional regularizations. Several efficient optimization
methods for model inference have been introduced to minimize the loss
functions in the literature.

Gauss-Seidel Method [48]: Minimizing the DCF loss functions with its spatially
regularized variants online is a highly challenging problem, since the filter
$w$ contains tens or hundreds of thousand parameters to be optimized. In [29],
an optimization approach based on the iterative Gauss-Seidel method is
proposed to minimize the spatially regularized loss function (13). The same
strategy is also considered in its variant DeepSRDCF [28] that employs deep
features. By employing the Gauss-Seidel-based optimization, the tracker
achieves a tracking speed of few frames per second. While not yet real-time,
it demonstrated superior robustness and accuracy compared to previous
approaches, yet faster than many of its competitors.

Conjugate Gradient Based Method [105]: To pave the way for the use of deep
features and to further improve the computational efficiency, a Conjugate
Gradient (CG)-based strategy is utilized in CCOT [22]. CG can be applied to
any set of normal equations $A\widetilde{w}=b$ of full rank. It is based on
finding a set of conjugate directions $p^{(i)}$ and optimal step lengths
$\beta^{(i)}$ used for updating the filter
$\widetilde{w}^{(i)}=\widetilde{w}^{(i-1)}+\beta^{(i)}p^{(i)}$. Theoretically,
the algorithm converges to the solution in a finite number of iterations $i$.
In practice however, the algorithm is stopped after a fixed number of
iterations or when the error has decreased to a satisfactory level.

The computational bottleneck of CG is the evaluation of the matrix-vector
product $Ap^{(i)}$ in each iteration $i$. This is in fact the key advantage of
CG. The Gauss-Seidel method requires solving a triangular system in each
iteration. On the contrary, CG can exploit the particular sparsity structure
of $A$ in the computation $Ap^{(i)}$. Moreover, $Ap^{(i)}$ can be implemented
as a series of simple block-wise dense matrix-vector products, convolutions
and point-wise multiplications. This reduces the quadratic $O(D^{2})$
complexity in the feature dimension to linear $O(D)$, a crucial improvement
enabling tractable integration of high-dimensional deep features.

To address non-linear least squares problems, the Gauss-Newton optimization is
also used in many trackers, including ECO [24], ATOM [23], and UPDT [7]. The
method linearizes the error residuals using a Taylor series expansion around
the current estimate to find a quadratic approximation of the objective. The
resulting quadratic problem can then be tackled with iterative methods, for
instance the CG approach described above. In the ECO [24] and ATOM [23], the
Gauss Newton combined with CG is employed to jointly optimize the filter $w$
and the dimensionality reduction matrix. DiMP employs Gauss Newton together
with Steepest Descent iterations to learn the filter $w$ using a non-linear
robust loss function [5]. The optimization steps are themselves
differentiable, which further enables end-to-end learning of the underlying
deep features. The PrDiMP further employs the more general Newton
approximation to address the convex and non-linear KL-divergence objective
[25].

Alternating Direction Method of Multipliers (ADMM) Method [11]: When a
classical DCF formulation grows in terms of additional regularizations such as
those presented in models (13)-(14), the formulation becomes a constraint
optimization problem which can be solved using efficient convex or non-convex
solvers such as an ADMM-based optimization method. The ADMM method has
recently been used in many DCF-based trackers to efficiently solve the DCF
loss functions specially when additional regularizations are introduced. The
ADMM-based optimization approach provides closed-form solutions for each sub-
problem and empirically converges within a very few iterations. In ADMM, the
model is solved by breaking it into smaller pieces, each of which is then
easier to handle. The augmented lagrangian formulation is always used to
convert the constrained optimization model into unconstrained model with a
lagrangian penalty as an additional variable. Each sub-problem of the
unconstrained model is then solved in an iterative manners. Trackers such as
BACF [41], DRT [120], AutoTrack [83], ARCF [61], and RPCF [123] have employed
ADMM for efficient solutions. Other optimization methods such as Gradient
descent is used by ACFN [18], CREST [118], and DSLT+ [92] trackers for network
optimization.

Current Choices: Both Conjugate Gradient [24, 22, 23, 7] and ADMM (STRCF [76],
BACF [41], RPCF [123], ASRCF [20], GFS-DCF [145], and MCCT [136]) have been
popular choices for DCF trackers. Both support the more complex DCF
formulations required to mitigate the boundary artifacts, using e.g. spatial
regularization or constraints. The most recent approaches DiMP employed a
steepest descent in a spatial-domain formulation [5], while PrDiMP combined it
with a Newton approximation to minimize the non-linear KL-divergence objective
[25]. In addition to efficiency, the adoption of end-to-end learning have
brought the importance of differentiability and simplicity of the optimizer,
motivating the latter choice. It has also opened the opportunity to train
components and hyper-parameters of the optimizer in a meta-learning fashion,
setting the stage for future research directions.

####  3.4.4 Target State Estimation

The DCF models have demonstrated promising results in terms of accuracy and
robustness. However, when a target moves its template size (also known as
bounding box size) also changes. The DCF model employs the bounding box with
the fixed size that often leads to severe model drift, since it is not capable
to handle target scale variations. Accurate scale estimation poses a great
challenge in the classical DCF formulation. The problem of handling bounding
box size for accurate target scale estimation is an established research
direction. Here, we briefly describe the most popular strategies.

Multiple Resolution Scale Search Method: One straight-forward strategy is to
apply learned translation filter $w$ at different image scales. That is, the
image is first resized by different scale factors, followed by feature
extraction. The feature map at each scale is then convolved with learned
filter $w$ to compute the target scores. The change in target location and
scale can then be estimated by finding maximum score across all scales. This
is a common strategy, often applied in both tracking and detection [39].

Li et al. proposed SAMF tracker in which the translation and scale filters
were jointly trained using the standard DCF formulation [81]. The results
demonstrated significant performance gain compared to the standard DCF. This
scale adaptive component has been utilized in a number of DCF-based trackers
such as, CACF [103], CFAT [9], and FD-KCF [169]. However, this approach
suffers from a higher computational cost, since the translation filter has to
be applied at several resolutions to achieve sufficient scale accuracy.

Discriminative Scale Space Search Method: Danelljan et al. proposed an
alternative strategy for accurate scale estimation [26]. Unlike [81], the
target estimation is performed in two steps in order to avoid an exhaustive
search across translation and scales. Since the scale changes between two
frames is usually small or moderate, the target translation is first found by
applying the normal translation filter $w$ at the current scale estimate.
Then, a separate one-dimensional filter is applied in the scale dimension to
update the target size. The scale filter is trained analogously to the
translation filter, but operates in the scale dimension by extracting samples
of the target appearance from a set of different scales.

The advantage of the aforementioned scale-filter approach [26] is two-fold.
First, computational efficiency is gained by reducing the search space.
Second, the scale filter is trained to discriminate between the appearance of
the target at different scales, which can lead to a more accurate estimation.
The proposed scale filter component has been utilized in a multitude of
trackers, including STAPLE [3], MUSTer [56], ASRCF [20], CACF [103], BACF
[41], CSR-DCF [95], MCCT [136], and LCT [99]. Moreover, the follow-up fDSST
tracker reduces the computational cost of DSST by applying the PCA and sub-
grid interpolation [27].

Deep Bounding Box Regression Method: The aforementioned methods show improved
performance. However, they depend on the scaling factor parameters and online
accurate correlation filter response. These methods do not exploit the
powerful deep feature representations in an offline manner. Moreover, these
online methods do not perform any bounding box regression. As a result, these
methods show performance degradation in the presence of sudden scale
variations. Accurate estimation of the target object bounding box is a complex
task, requiring high-level a priori knowledge. The bounding box depends on the
pose and viewpoint of the target, which cannot be modeled as a simple image
transformation (e.g. uniform image scaling). It is therefore extremely
difficult to learn accurate target estimation online from scratch.

In object detection methods, bounding box regression has been widely employed
for precise localization of object bounding boxes [42, 62, 43, 59, 109, 108].
Conventional $L_{1}$ or $L_{2}$ loss is used between the predicted and ground-
truth parameters of the bounding box for regression. To exploit the strength
of end-to-end deep features learning for target scale estimation, this
component has recenty been utilized in [23, 5]. In ATOM [23], inspired by the
IoU-Net [62], the targets-specific features are trained. Since the original
IoU-Net is class-specific, and hence not suitable for generic tracking, a
novel architecture is proposed for integrating target-specific information
into the IoU prediction [23]. This is achieved by introducing a modulation-
based network component that incorporates the target appearance in the
reference image to obtain target-specific IoU estimates. This further enables
the target estimation component to be trained offline on large-scale datasets.
During tracking, the target bounding box is found by simply maximizing the
predicted IoU overlap in each frame. Results demonstrated an excellent
performance boost as compared to classical multi-scale search methods.

Several recent trackers, including DiMP [5], PrDiMP [25], and KYS [6] have
also utilized this strategy for state estimation. In PrDiMP, instead of
predicting the IoU, it employs energy-based model that predicts the un-
normalized probability density of the bounding box. This is trained by
minimizing the KL-divergence to a Gaussian model of the label noise. Fig. 4
shows influential DCF trackers in literature.

###  3.5 Evolution of DCFs to Segmentation-based Trackers

Precise and accurate object segmentation provides reliable object observations
for tracking, which can solve several tracking problems including rotated
bounding box issues, occlusion, deformation, and scaling, etc., and
fundamentally avoid tracking failures. In literature, segmentation-based
approaches have been incorporated within the DCFs-based trackers for improved
filter learning in the presence of non-rectangular targets [95, 63, 3].

Bertinetto et al., used a color histogram-based segmentation method to improve
tracking under varying illumination changes, motion blur, and target
deformation [3]. Lukezic et al., proposed a spatial reliability map using a
color-based segmentation method to regularize filter learning [95]. A real-
time tracker is proposed using handcrafted features and achieved comparable
performance using deep features. Kart et al. [63], extended the CSR-DCF
tracker for RGB-depth tracking based on both color and depth segmentation as
depth cues provide more reliable segmentation map. Lukezic et al., proposed a
single shot segmentation tracker to address VOT and video object segmentation
problems within a joint framework [94]. The target is encoded with two
discriminative models for the joint tracking and segmentation task. The
results are reported in many tracking and segmentation benchmarks and
demonstrated the benefits. Recently, Robinson et al., employed a powerful
discriminative model using the fast optmization scheme borrowed from the ATOM
[23] for video object segmentation task [110]. Bhat et al., also used the
target model discriminative capabilites for more robust video object
segmentation [8].

##  4 Siamese Trackers

Deep learning models have revolutionized many machine learning applications.
The key to the success of these models is the offline learning capabilities of
features on a large volume of data. Such offline training models have a
capability to learn complex and rich relationships from large amount of
annotated data.

End-to-end offline training models have also been employed in the generic
object tracking by posing it as a similarity learning problem [4, 52, 127].
Deep SNs have been widely used to learn a similarity between the target image
and the search image region [4]. SNs were first used for signature
verification task [12] and then adapted for other applications including
fingerprint recognition [19, 125], stereo matching [159], ground-to-aerial
image matching [85], and local patch descriptor learning [49].

In VOT, an offline deep network is trained on a large amount of pairs of
target images to learn a matching function during training and then this
network as a function is evaluated online during tracking. Bertinetto et al.
unveiled the power of SNs for VOT [4]. The siamese tracker consists of two
branches, the template branch and the detection branch. The template branch
receives the target image patch in the previous frame as input while the
detection branch receives the target image patch in the current frame as
input. Both of these branches share CNN parameters so that the two image
patches encode the same transformation which is suitable for tracking. Fig. 3
presents the tracking pipeline of a standard SN. The main aim of SN is to
overcome the limitations of pre-trained deep CNNs and take full advantage of
end-to-end learning for real-time applications. The offline training videos
are used to instruct the tracker to handle rotations, changes in viewpoint,
lighting changes, and other complex challenges. With the use of a SN, the
tracker is able to learn the generic relationship between the object motion
and appearance and can be used to locate unseen targets which were not
utilized in training.

Figure 3: The Siamese tracking pipeline for generic object tracking. Figure 4:
Influential Siamese and DCF trackers proposed in the 2014-2020 period.

Training Pipeline: In the Fully Convolutional SN (SiamFC) [4], we consider a
pairs of training images $(x,z)$. Here, $x$ denotes the object of interest
(for instance an image patch cropped from the center of the target image in
the first frame) and testing image $z$ represents a larger search area in the
next frame. We input these pairs $(x,z)$ into a CNN to obtain two feature
maps, which are then matched using a cross-correlation,

$g_{\rho}(x,z)=f_{\rho}(x)\star f_{\rho}(z)+b,$ (15)

where $\star$ denotes cross-correlation operation, $f_{\rho}(.)$ is a deep
CNN, e.g. AlexNet [1], with learnable parameters $\rho$, and $g_{\rho}(x,z)$
is a response map denoting the similarity between $x$ and $z$, and
$b\in\mathbb{R}$ denotes a scalar offset value. The cross-correlation
similarity function (15) performs an exhaustive search of the features
extracted from sample $x$ over the features extracted from $z$.

The goal is for the maximum value of the response map $g_{\rho}(x,z)$ to
correspond to the target location. To achieve it, the network is trained
offline on millions of random pairs $(x_{i},z_{i})$ extracted from a
collection of videos to track the generic objects. The mean of the logistic
loss is typically employed to train the network as,

$\ell(c,v)=\frac{1}{N}\sum_{i=1}^{N}\log(1+\exp(-c_{i}v_{i})),$ (16)

where $v$ represents real-valued score of a single target-test sample and
labels $c_{i}\in\\{-1,1\\}$, with the true object location belonging to the
positive class and all others to the negative class. Training proceeds by
minimizing an element-wise logistic loss $\ell$ over the training set as:

$\operatorname*{argmin}_{\rho}\mathbb{E}_{(x,z,c)}\ell(c_{i},g_{\rho}(x_{i},z_{i};\rho)).$
(17)

Tao et al. proposed SINT in which Euclidean distance is employed as a
similarity measure instead of cross-correlation [127]. Held et al. proposed
the GOTURN in which a bounding box regression is employed [52]. Similarly,
Valmadre et al. proposed CFNET in which correlation filter is added in the $x$
as a separate block in the matching function (15) and makes this network
shallower but more efficient [128].

Testing Pipeline: The network only provides a function to measure the
similarity of two image patches. It is necessary to combine this network for
VOT with a procedure that describes the logic. To assess the utility of the
similarity function, a simplistic tracking algorithm is employed by first
extracting the feature representation of the $x$ and $z$ in the new frame. The
feature representation of $x$ is then compared to that of $z$, which is
obtained in each new frame by extracting a window centred at the previously
estimated position, with an area that is four times the size of the object.

The new position of the target is predicted using the location with the
highest score. The original SiamFC network simply compares every frame to the
initial appearance of the object and tracks the object real time at 140FPS on
a GPU. SNs are computationally efficient in both inference and in offline
learning. SNs have demonstrated SOTA tracking performance and therefore
siamese trackers are receiving a lot of attention in the tracking community
nowadays.

###  4.1 Open Issues in Standard Siamese Tracking Pipeline

The classical SN outperforms DCFs trackers in both accuracy and efficiency.
However, SN also suffers from several limitations in terms of the backbone
features extraction network, need for a large annotated image pairs in offline
training, lack of online adaptability, loss function formulation, and the
target state estimation. In the subsections below, we identify and discuss
these important challenges for developing a robust Siamese-based tracker.
Below, we briefly describe the details of these problems in siamese tracking
and their potential solutions developed in recent years.

####  4.1.1 Backbone Architectures

In offline training, the backbone feature extraction network plays a dominant
role to capture high-level semantic information of the target. Siamese
trackers have demonstrated encouraging performance using powerful backbone
networks. In early siamese trackers (SiamFC [4], GOTURN [52] and SINT [127]),
a modified pre-trained AlexNet is fine-tuned [1]. A variety of trackers
(FlowTrack [172], MemTrack [153], EAST [58], and SiamRPN [75]) have used
AlexNet. However, it is observed that these trackers are still limited in
performance because AlexNet is a relatively shallow network and does not
produce very strong feature representations.

With the advent of wider and modern deeper networks, the tracking community
has also designed SNs based on ResNet [51], VGG-19 [116], and Inception [124].
All these networks are pre-trained on the ImageNet dataset and fine-tuned to
learn the generic matching function. Tao et al. analysed VGG-16 and AlexNet
models in SINT [127]. Results show performance gap in tracking between these
two pre-trained networks. It is also observed that direct replacement of
powerful deep architectures does not bring performance improvement in the
classical SNs-based tracking methods [74].

To address this issue, Li et al. investigated the main reason behind this
issue and proposed a ResNet-driven SiamRPN++ tracker [74]. In SNs, when
modified AlexNet is employed without zero-padding [4], the learnt spatial
feature representations of the target does not satisfy the spatial
translational invariance constraint. Therefore, an effective sampling
technique is developed to fulfill this spatial invariance constraint.
Leveraging the powerful deep ResNet architecture, the performance of many
siamese trackers are improved. Zhang et al. also investigated the same problem
and proposed SiamDW in which a shallow backbone AlexNet is replaced with deep
networks including Inception, VGG-19, and ResNet [167]. It is investigated
that apart from features padding, receptive fields of neurons and network
strides are also main reasons why such a deeper network cannot directly
replace shallow networks. The results presented in both studies demonstrated
an excellent performance compared to classical SNs-based trackers. With these
foundations, recent trackers including SiamCAR [45], Ocean [168], and SiamBAN
[15] etc., have also employed powerful deep architectures. The ResNet backbone
has become the established and preferred alternative for Siamese tracking,
thanks to its simplicity and strong performance. However, recent advances in
vision transformer networks [34, 90] are expected to have a substantial impact
in the tracking community in the coming years.

####  4.1.2 Offline Training

As discussed above, training data is crucial for VOT therefore it is very
difficult to learn a robust matching function in SNs (15).

To handle this issue, the tracking community has made outstanding progress by
leveraging images and video datasets to learn the generic relationships
between the objects. Object detection, image classification, and object
segmentation datasets including ImageNet ILSVRC2014 [111], ILSVRC2015 [112],
COCO [86], YouTube-BB [107], and YouTube-VOS [144], have been widely used in
SNs. These datasets sufficiently cover a good amount of semantics and do not
focus on particular objects, otherwise, the tuned network parameters will
overfit to particular object categories in siamese training. The datasets are
typically annotated with bounding boxes of a target object in every frame.
Recently, the tracking community has compiled more diverse small and large-
scale tracking benchmark datasets containing tens of thousands of annotated
bounding boxes. These include the LaSOT [35], GOT-10K [60], and TrackingNet
[104] which are also used for offline training by recent trackers [15, 130,
149]. Interested readers can find more details about different training
datasets utilized by different Siamese trackers in Table I of the
supplementary material.

Unlike DCF paradigm, the standard Siamese formulation cannot exploit the
appearance of known distractor objects during tracking. Siamese approaches
therefore often struggle when objects similar to the target itself present.
This occurs when, for instance, other objects of the same semantic class are
in the view. Early Siamese trackers (SiamFC [4] and SiamRPN [75] particularly)
only sample pairs of training images from the same video during training. This
sampling strategy does not focus on challenging cases, with sementically
similar distractor objects. To address this issue, hard negative mining
techniques have been developed in the literature. For instance, Zhu et al.
introduced hard negative mining technique in DaSiamRPN [171] to overcome data
imbalance issue by including more semantic negative pairs into the training
process. The constructed negative pairs consist of labelled targets both in
the same and different categories. This technique assisted DaSiamRPN to
overcome drifting by focusing more on fine grained representations. With the
same spirit, Voigtlaender et al. proposed another hard negative mining
technique using an embedding network and a nearest neighbor approximation
[130]. For every ground truth target bounding box, the embedding vector is
extracted for a similar target appearance using pre-trained network. The
indexing structure is then employed to estimate the approximate nearest
neighbors and use them to estimate the nearest neighbors of the target object
in the embedding space.

The recent trend of utilizing more training data and designing data mining
techniques have demonstrated excellent tracking performance on multiple
benchmarks, opening many doors to explore more sophisticated techniques in
Siamese training.

####  4.1.3 Online Model Update

In SiamFC [4], the target template is initialized in the first frame and then
kept fixed during the remainder of the video. The tracker does not perform any
model update and therefore, the performance totally relies on the general
matching ability of the SN. However, appearance changes in the presence of
tracking challenges are often large and failing to update the model leads to
failure of the tracker. In such scenarios, it is important to adapt the model
to the current target appearance. In the literature, the tracking community
has also proposed potential solutions in this direction

Moving Average Update Method: Many recent SOTA trackers including GOTURN [52],
SINT [127] and SiamAttn [158] etc., employ a simple linear update strategy
using a running average with a fixed learning rate. While it provides a simple
means of integrating new information, the trackers cannot recover from drift
due to constant update rate and simple linear combination of previous
appearance templates.

Learning Dynamic SN Method: Guo et al. proposed the DSiam tracker and designed
dynamic transformation matrices [46]. Two distinct online transformation
matrices including target appearance variation and background suppression are
incorporated within the classical SN. Both matrices are solved in the Fourier
domain with a closed-form solutions. DSiam provides effective online learning
however it ignores the historical target variations which is important for a
smoother adaptation of the exemplar template.

Dynamic Memory Network Method: Yang et al. proposed the MemTrack that
dynamically writes and reads previous templates to cope with target appearance
variations [153]. A long term short term memory is used as a memory
controller. The input to this network is the search feature map and the
network outputs the control signals for the reading and writing process of the
memory block. An attention mechanism is also applied with a gated residual
template to control the amount of retrieved memory that is used to combine
with the initial template. This method enables the tracker to memorize long-
term target appearance. However, it only focuses on combining the previous
target features, ignoring the discriminative information in background
clutter, which leads to an accuracy gap in the presence of drastic target
variations.

Gradient-Guided Method: Li et al. proposed GradNet in which gradient
information is encoded for updating the target template via feedforward and
backward operations [77]. The tracker utilizes the information from the
gradient to update the template in the current frame and then incorporates the
adaptation process to simplify the process of gradient-based optimization.
Unlike aforementioned methods, this method makes full use of the
discriminative information in backward gradients instead of just integrating
previous templates. This results in a performance improvement as compared to
other methods however, computing gradient in a backpropagation manner
introduces a computational burden.

UpdateNet Method: Zhang et al., proposed the UpdateNet [161] which learns a
generic function $\theta$ according to
$\tilde{z}_{t}=\theta(\tilde{z}_{gt},\tilde{z}_{t-1},z_{t})$ The learned
function $\theta$ computes the updated template based on initial ground-truth
template $z_{gt}$, the last accumulated template $\tilde{z}_{t-1}$ and the
template $z_{t}$ extracted from the predicted target location in the current
frame. In general, the function updates the previous accumulated template
$\tilde{z}_{t-1}$ by integrating the new information given by the current
frame $z_{t}$. Therefore, $\theta$ can be adapted to the specific updating
requirements of the current frame, based on the difference between the current
and accumulated templates. Moreover, it also considers the initial template
$z_{gt}$ in every frame, which provides highly reliable information and
increases robustness against model drift. The function $\theta$ is implemented
as a CNN, which grants great expressive power and the ability to learn from
large amounts of data. The results demonstrate excellent performance compared
to SiamFC [4] and DaSiamRPN [171] and it has also recently been adopted by CSA
tracker [149].

While a number of techniques for model update have been proposed, simply using
no update has remain a surprisingly robust and popular alternative [130, 74].
Further research in this direction is required to develop simple, general, and
end-to-end trainable techniques, which could further improve the robustness of
Siamese tracking.

####  4.1.4 Loss Functions

The tracking performance also relies on the loss functions employed within the
SNs. Different loss functions have been used in the SNs either for regression,
classification or for both tasks. Below, we summarize these developments in
more detail.

Logistic Loss: The classical SiamFC employed logistic loss defined in Eqs.
(16)-(17) [4]. A variety of other trackers including DSiam [46], RASNET [139],
SA-SIAM [50], CFNET [128], SiamDW [167], and GradNet [77] etc., have used
logistic loss to train their models built upon SiamFC. This training method
utilizes the pairwise relationship on image pairs by maximizing the similarity
scores on target-positive pairs and minimizing them on target-negative pairs.

Contrastive Loss: The margin contrastive loss is defined as [19]:

$\begin{split}\mathcal{L}(x_{i},z_{i},y_{xz})=\frac{1}{2}D^{2}+\frac{1}{2}(1-y_{xz})\max(0,\epsilon-D^{2}),\\\
D=||f_{\rho}(x_{i})-f_{\rho}(z_{i})||_{2},\end{split}$ (18)

where $\epsilon$ is the minimum distance margin that pairs depicting different
objects should satisfy, $D$ is the Euclidean distance of $l_{2}$-normalized
feature representations, and $y_{xz}\in\\{0,1\\}$ indicates whether $x_{i}$
and $z_{i}$ are the same object or not. The SINT tracker [127] employed the
contrative loss while GOTURN [52] employed $L_{1}$ loss between the predicted
and the ground-truth bounding boxes.

Triplet Loss: The above loss exploits the pairwise relationship between images
only and ignores the underlying structural connections between the positive
and negative instances of the target. Yan et al. proposed SPLT tracker [151]
in which the triplet loss [141, 114] is employed during training. The triplet
loss is defined as

$\begin{split}\mathcal{L}(x,x^{p}_{i},x^{n}_{i})=\sum_{i}^{M}\Big{[}||f_{\rho}(x_{i})-f_{\rho}(x^{p}_{i})||_{2}^{2}-\\\
||f_{\rho}(x_{i})-f_{\rho}(x^{n}_{i})||_{2}^{2}+\epsilon\Big{]}_{+},\end{split}$
(19)

where $x^{p}_{i}$ denotes the positive patch of the target image $x$ i.e., one
of other images of the target and $x^{n}_{i}$ is a negative patch of any other
target or background. $\mathcal{T}$ is the set of all possible triplets in the
training set and has cardinality $M$. The triplet loss not only can further
mine the potential relationship among target, positive and negative instances,
but also contains more robust similarity structure. Dong et al., proposed
SiamFC-Tri in which the probabilistic triplet loss is used [32]. The Siam
R-CNN tracker has also been trained using the same loss [130].

Cross Entropy Loss: The classification component in the SNs are normally
borrowed from object detection methods [42]. To incorporate this branch, a
cross-entropy loss ($\mathcal{L}_{cls}$) is used which is defined as:

$\begin{split}\mathcal{L}_{cls}=-\sum_{j}p_{o}^{*}\log(p_{o})+(1-p_{o}^{*})\log(1-p_{o}),\end{split}$
(20)

where $p_{o}$ is a predicted label and $p_{o}^{*}$ denotes the groundtruth
label. Li et al., proposed the SiamRPN tracker in which cross entropy loss is
employed [75]. Other trackers such as SiamRPN++ [74], SiamAttn [158], Ocean
[168], CLNET [33], SPM [131], C-RPN [37] etc., have also been built upon the
SiamRPN tracker by training classification branch using the cross entropy
loss.

Regression Loss: To train a regression network, three types of loss functions
are employed including the smooth $L_{1}$ norm [42], the Intersection over
Union (IoU) loss [157], and regularized linear regression [113]. The smooth
$L_{1}$ loss is used in Faster R-CNN for bounding box regression [42]. In the
SiamRPN tracker [75], this norm is used to train the regression branch.
Following this study, other trackers including SiamRPN++ [74], SiamAttn [158],
CLNET [33], SPM [131], and C-RPN [37] have also trained the regression branch
of the tracker using the $smooth_{L1}$ loss. The IoU loss for regression
$\mathcal{L}_{reg}^{IoU}$ is defined as [157]:

$\mathcal{L}_{reg}^{IoU}=-\sum_{j}\ln(\text{IoU}(p,r)),$ (21)

where $p$ and $r$ denote the predicted and groundtruth bounding box
corrdinates. Chen et al. proposed the SiamBAN tracker in which the regression
branch of the network is trained using $\mathcal{L}_{reg}^{IoU}$ [15]. The
Ocean [168] and SiaMFC++ [146] trackers have also utilized this loss during
training.

Multi-Task Loss: For the joint training of classification and regression
branches, multi-task loss has also been used in SNs. For instance, a sum of
the cross-entropy loss (20) and the regression loss (21) (i.e.,
$\mathcal{L}_{cls}+\mathcal{L}_{reg}$) is used in SiamRPN [75]. Trackers such
as SiamRPN++ [74], SiamAttn [158], CLNET [33], and SPM [131] have also
employed the multi-task training method. Moreover, a sum of cross entropy loss
(20) and IoU loss for regression (21) (i.e.,
$\mathcal{L}_{cls}+\mathcal{L}^{IoU}_{reg}$) has also been utilized by Ocean
[168], SiamBAN [15], and SiamFC++ [146].

Regularized Linear Regression: To regularize SNs with a correlation filter as
a separate layer, a linear regression loss defined in Eqs. (9)-(10) is
employed in many Siamese trackers including CFNET [128], TADT [80], RTINET
[155], DSiam [46], FlowTrack [172], UDT [134], and UDT++ [135] etc. The ridge
regression problem is then solved by a closed-form solution and the filter is
trained in an end-to-end fashion.

Currently, there is no general consensus in the literature regarding the
employed loss function. Instead, recent SOTA methods adopt different
alternatives. Among the aforementioned approaches, the Cross Entropy loss have
remained a popular choice, also for recent trackers [74].

####  4.1.5 Target State Estimation

Similar to DCFs-based trackers, the SNs also suffer from severe scale
variations challenges. The similarity function only learns the deep structural
generic relationship between the images and it does not take into account the
problem of scale changes. The tracking community has also shown very good
progress in this direction and proposed potential solutions to handle it. In
the subsections below, we discuss the proposed methods.

Multiple Resolution Scale Search Method: Similar to the DCFs-based trackers,
this method has also been employed in many siamese trackers to cope with scale
variation challenges. In classical SiamFC, multiple scales are searched in a
single forward-pass by assembling a mini-batch of scaled images and then the
maximum response is computed. Early SN-based trackers including RASNET [139],
SA-Siam [50], StructSiam [166], UDT [134], UDT++ [135], TADT [80], GradeNet
[77], RTINET [155], and FlowTrack [172] have employed this method for scale
estimation.

Deep Anchor-based Bounding Box Regression Method: To estimate the target
bounding boxes more accurately, the object detection capabilities are
introduced within the SN [4]. Anchor-based bounding box regression using a
Regional Proposal Network (RPN) [42] efficiently predicts region proposals
with a wide variety of scales and aspect ratios. It takes an input image and
estimates a set of rectangular object proposals, each with an objectness
score. To do so, a small network is slided over the convolutional feature map
output by the last convolutional layer. At each sliding-window location, the
multiple region proposals are simultaneously predicted. RPN is a fully
convolutional network that simultaneously predicts object bounds and
objectness scores at each position. The RPN is trained in an end-to-end manner
to generate high-quality region proposals [42].

Li et al. proposed the SiamRPN tracker [75] which incorporates an RPN
component. The outputs of SiamRPN include one classification
($\mathcal{L}_{cls}$) and one regression ($\mathcal{L}_{reg}$) branches to
regress the target bounding box for both position and scale estimation.
Results demonstrated superior tracking performance compared to classical
trackers in the presence of RPN. Many recent trackers such as DaSiamRPN [171],
SiamRPN++ [74], SiamDW [167], SPLT [151], C-RPN [37], SiamAttn [158], CSA
[149], and SPM [131], etc., are also built upon the same notion.

Deep Anchor-free Bounding Box Regression Method: Anchor-based bounding box
regression assists Siamese trackers [75, 74] to handle changes in scale and
aspect ratio and shows encouraging results. However, it needs to carefully
design anchor boxes based on heuristic knowledge, which introduces many hyper-
parameters and computational complexity. Therefore, it is essential to
carefully design and fix the parameters of the anchor boxes.

Anchor-free regression has also been proposed in object detection, which
avoids hyper-parameters associated with the anchor boxes and is more flexible
and general [72, 170]. Anchor-free detectors directly find objects without
preset anchors using keypoint-based methods [72] and center-based methods
[170]. The keypoint-based methods first locate the pre-defined keypoints and
then perform bounding box regression on the objects. In center-based methods,
the four distances from positives to the object boundary are predicted using
the center on the region of object. The anchor-free detectors are able to
eliminate those hyperparameters related to anchors and have achieved similar
performance with anchor-based detectors, making them more powerful in terms of
generalization ability.

Chen et al. proposed the SiamBAN tracker [15] in which anchor-free box
regression is employed to estimate the target scale. The tracker avoids hyper-
parameters associated with target bounding boxes without any preset anchor
boxes. The tracker exploits the expressive power of the fully convolutional
network to classify objects and regresses their bounding boxes in a unified
manner. Similar to SiamRPN [74], SiamBAN includes a classification module
which performs foreground-background classification on each point of the
correlation layer and a regression module performs bounding box prediction on
the corresponding position [15]. Ocean [168] and SiamCAR [45] trackers have
also utilized the same method for scale estimation. The object detection
capabilities have demonstrated outstanding progress in the target state
estimation component. The recent trends of using RPNs and anchor-free bounding
box regression have revealed to further explore these techniques in an end-to-
end paradigm. Fig. 4 shows influential Siamese trackers in literature.

###  4.2 Evolution of Siamese Networks to Segmentation-based Tracker

The Siamese trackers also face severe challenges in the presence of deformable
objects such as a person with spread out hands, rotated or axis aligned
bounding boxes [140]. These tracking challenges can be addressed by assisting
trackers using the most accurate and well-defined target location models.
Siamese trackers are quite fast and provide real-time performance while video
segmentation approaches are slow and not real-times therefore, combining these
two problems provide efficient solution for both tracking and segmentation.

Recently, SNs have also been extended to perform both video object
segmentation and tracking [140]. Wang et al. present a SN to simultaneously
estimate binary mask, bounding box, and the corresponding background-
foreground scores. This multi-stage deep network lacks the opportunity to
process the visual tracking and target segmentation jointly to increase
robustness. Lu et al., employed unsupervised video object segmentation task in
which a novel architecture is proposed based on the co-attention mechanism
within the SN [93].

Figure 5: Sample images from the nine tracking benchmark datasets. The ground-
truth bounding box annotation is overlaid.

##  5 Experimental Comparison

Here, we thoroughly analyze the performance of 59 DCFs and 33 Siamese-based
trackers. The performance of these trackers have been quantitatively compared
on nine tracking benchmarks: Online Tracking Benchmark 100 (OTB100) [142],
Temple Color 128 (TC128) [84], Unmanned Aerial Vehicle 123 (UAV) [102], Visual
Object Tracking 2014 (VOT) [66], VOT2016 [47], VOT2018 [68], TackingNet [104],
Large-scale Single Object Tracking (LaSOT) [35], and Generic Object Tracking
10,000 (GOT-10K) [60]. Fig. 5 shows example frames from different tracking
benchmarks. The quantitative results of the compared trackers are either taken
directly from respective papers or from other papers.

TABLE I: Details of the nine benchmarks used in our experimental comparison.

Description OTB100 TC128 UAV123 UAV20L VOT2014 VOT2016 VOT2018 VOT2018-LT
LaSOT GOT-10K TrackingNet Publication PAMI2015 [142] TIP2015 [84] ECCV2016
[102] ECCV2016 [102] ECCV-W2014 [66] ECCV-W2016 [47] ICCV-W2017 [68]
ECCV-W2018 [68] CVPR2019 [35] PAMI2019 [60] ECCV2018 [104] Sequences Total:
1400 Total: 10k Total: 30.643k 100 128 123 20 25 60 60 35 Training: 1120
Training: 9.34k Training: 30.132k Testing: 280 Testing: 420 Testing: 511 Boxes
Total: 3.52M Total: 1.5M Total: 14M 58.61k 55.652k 113.476k 58.67k 10k 21.455k
21.356k 146.847k Training: 2.8M Training: 1.4M Training: 14M Testing: 685K
Testing: 56k Testing: 226k Attributes 11 11 12 12 12 12 12 12 14 6 15 Classes
Total: 563 22 27 9 9 11 24 24 21 70 Training: 480 21 Testing: 84 Minimum
Frames 71 71 109 1717 164 48 41 1389 1000 51 96 Maximum Frames 3872 3872 3085
5527 1210 1507 1500 29700 11397 920 2368 Average Resolution $356\times 530$
$461\times 737$ $1231\times 699$ $1231\times 699$ $448\times 304$ $757\times
480$ $758\times 465$ $896\times 468$ $632\times 1089$ $929\times 1638$
$591\times 1013$ Average Duration 19.68s 15.6s 30.48s 97.8s 13.68s 357.6s
355.9s 139.85s 83.57s 15s 16.7s Attributes Name IV, SV, OCC, IV, SV, OCC, ARC,
BC, CM, ARC, BC, CM, IV, SV, OCC, OCO, SCO, AM, OCO, SCO, AM, IV, SV, OCC, IV,
SV, DEF, MB, IV, SV, IV, SV, DEF, MB, DEF, MB, FM, DEF, MB, FM, FM, FOC, IV,
FM, FOC, IV, DEF, MB, BC, ARC, CM, MOC, ARC, CM, MOC, DEF, MB, BC, FM, OV, BC,
LR, OCC, FM, FM, IPR, OPR, OV, IPR, OPR, OV, IPR, OPR, OV, LR, OV, POC, LR,
OV, POC, ARC, CM, MOC, DEF, MB, BC, DEF, MB, BC, ARC, CM, MOC, ARC, CM, FOC,
SLO, BC, LR, ARC, CM, BC, LR. BC, LR. SOB, SV, VC. SOB, SV, VC. OCO, SCO, AM.
IV, SV, OCC. IV, SV, OCC. OCO, SCO, AM. POC, VC, ROT. ARC. FOC, POC, SOB.
Frame Rate 30 FPS 30 FPS 30 FPS 30 FPS 30 FPS 30 FPS 30 FPS 30 FPS 30 FPS 10
FPS 30 FPS Overlapped Datasets OTB50 OTB50 OTB50 OTB100 OTB100, NUS-PRO [73]
OTB100, NUS-PRO [73] YouTube VOT VOT VOT VOT VOT TC128 TC128 ALOV++ [117]
ALOV++ [117] ImageNet WordNet [101] YouTube-BB TC128 OTB100 ALOV++ [117]
ALOV++ [117] TC128, UAV123 TC128, UAV123 ImageNet

###  5.1 Tracking Datasets

To provide a standard and fair performance evaluation of object trackers, a
number of benchmarks have been proposed with the passage of time. In addition
to short-term tracking, several recent datasets provide both short and long-
term tracking sequences. The publicly available benchmark datasets contain a
variety of tracking challenges, including Scale Variation (SV), Out-of-View
(OV), DEFormation (DEF), Low Resolution (LR), Illumination Variation (IV),
Out-of-Plane rotation (OPR), OCClusion (OCC), Background Clutter (BC), Fast
Motion (FM), In-Plane Rotation (IPR), Motion Blur (MB), Partial OCclusion
(POC), abrupt Camera Motion (CM), Aspect Ratio Change (ARC), Full OCclusion
(FOC), Viewpoint Change (VC), Similar OBject (SOB), Object Color Change (OCC),
Absolute Motion (AM), target ROTation (ROT), Scene COmplexity (SCO), Fast
Camera Motion (FCM), Low Resolution Objects (LRO), and MOtion Change (MOC).
Table I presents the description of each dataset employed in our experimental
comparison. Next, we briefly describe each tracking dataset.

####  5.1.1 OTB100 Dataset

Wu et al. proposed an object tracking benchmark known as OTB50 [143]. This
dataset consists of 51 video sequences with manually annotated bounding boxes
in each frame. The sequences are categorized into 11 different tracking
attributes (I). Later on, Wu et al. extended OTB50 to OTB100 dataset by adding
49 more videos [142]. OTB100 consists of 100 videos of 22 object categories
with the same 11 tracking attributes as in OTB50. The average resolution in
the OTB100 dataset is $356\times 530$, while the video length ranges between
71 and 3872 frames.

####  5.1.2 TC128 Dataset

TC128 was introduced to evaluate the impact of color on visual tracking [84].
It contains 128 fully annotated color video sequences of 27 object categories.
Out of 128 sequences, 78 sequences are different from OTB100, whereas the
remaining 50 sequences are common in both datasets. TC128 also comprises 11
tracking attributes, similar to OTB100 (Table I). The average resolution is
$461\times 737$, with 71 minimum and 3872 maximum number of frames.

####  5.1.3 UAV123 Dataset

UAV123 contains a realistic and synthetic HD video sequences captured by
professional-grade UAV [102]. Videos are captured from low-altitude UAVs which
are inherently different from sequences in other datasets. This dataset is
divided into two subsets namely, UAV123 and UAV20L. The UAV123 contains 123
short sequences of 9 diverse object categories, with 109 minimum number of
frames, and 3085 maximum number of frames. The UAV20L consists of 20 long
videos of 5 object classes generated from a flight simulator. These sequences
contain 1717 minimum and 5527 maximum number of frames. Both dataset subsets
contain an average resolution of $1231\times 699$ and labelled with 12
attributes (Table I).

####  5.1.4 VOT Dataset Series

The VOT dataset accompanies the annual VOT challenge competition to benchmark
tracking performance [71]. Each frame in the VOT datasets is annotated with a
rotated bounding box with a number of tracking challenges. Here, we select
VOT2016, VOT2018, and VOT2020 datasets as representatives of VOT series to
compare the performance of different trackers. A short description of VOT
datasets is presented in Table I.

VOT2016 Dataset [47]: This dataset contains 60 sequences, as in VOT2015 [67].
The only difference is that the ground-truth bounding boxes in VOT2016 are
more accurate than VOT2015, since a segmentation-based method is employed to
generate more precise and accurate bounding boxes. Each sequence is per-frame
annotated by different attributes, including OCC, IV, MOC, ARC, SCO, and FCM.
The average resolution of sequences is $757\times 480$, with 48 minimum and
and 1507 maximum number of frames.

VOT2018 Dataset: Different from VOT2016, this dataset consists of short and
long-term challenge splits [68]. The VOT2018 Short-Term (VOT2018-ST) challenge
consists of 60 sequences of 24 object categories, as in VOT2017 [69]. The
average resolution of sequences in short-term challenge is $758\times 465$,
with 41 minimum and 1500 maximum number of frames. The long-term split
consists of 35 long-term sequences. The average resolution of sequences in
long-term is $896\times 468$, with 1389 minimum and 29700 maximum number of
frames.

VOT2020 Dataset: VOT2020 consists of five sub-sets, including sequestered
short-term and long-term sequences [70]. We use VOT2020 Short-term
(VOT2020-ST) dataset to evaluate the performance of the trackers. The
VOT2020-ST is the same as VOT2018-ST in terms of number of videos, classes,
and attributes. The only difference is that the target position is encoded by
a segmentation mask with new performance evaluation measures and protocol. The
initial masks are obtained by a semi-automatic method and then all sequences
are frame-by-frame manually corrected. The main objective is to evaluate
trackers for segmentation tasks that compliments features from both tracking
and video object segmentation.

####  5.1.5 TackingNet Dataset

Muller et al. proposed the TackingNet dataset for long-term tracking in the
wild [104]. It consists of 60,643 sequences with more than 14 million dense
bounding box annotations. It covers 27 diverse object classes. The sequences
are also represented by 15 tracking attributes. The dataset is divided in to
training, validation, and test splits. The training split contains 30643
sequences, whereas test split consists of 511 videos. In test split, the
average resolution of a sequence is $591\times 1013$, with 96 minimum and 2368
maximum number of frames at 30fps.

####  5.1.6 LaSOT Dataset

Fan et al. proposed the LaSOT dataset that consists of 1120 training sequences
(2.8M frames) and 280 testing sequences (685K frames) [35]. All sequences are
annotated with bounding boxes in every frame. The object categories are
selected from the ImageNet. It contains 70 diverse object categories and each
category consists of 20 target sequences. The sequences are categorized
according to 14 attributes, including ARC, BC, FCM, DEF, POC, ROT, and VC. The
average resolution of a sequence is $632\times 1089$. Moreover, the dataset
contains very long sequences, ranging between 1000 and 11,397 number of
frames.

####  5.1.7 GOT-10K Dataset

GOT-10K [60] consists of 10,000 videos from semantic hierarchy of WordNet
[101]. The aim is to provide a unified training and evaluation platform for
the development of class-agnostic trackers with rich motion trajectories. The
sequences are classified to 563 classes of moving objects, six tracking
attributes, and 87 classes of motion to cover as many challenging patterns in
real-world scenarios as possible. GOT-10K is divided into training,
validation, and test splits. The training split contains 9,340 sequences with
480 object categories, whereas test split consists of 420 videos with 83
object categories and each sequence with an average length of 127 frames. In
test split, the average resolution of a sequence is $929\times 1638$, with 51
minimum and 920 maximum number of frames at 10fps.

###  5.2 Performance Evaluation Measures

To compare the performance of trackers, different evaluation metrics have been
proposed in the literature that evaluate the effectiveness in terms of
robustness, accuracy and speed.

Precision Plot: The precision plot is based on the central location error
which is defined as the average Euclidean distance between the predicted
centers of the target object and the ground truth centers in a frame [142].
However, this error does not compute the tracking performance accurately.
Therefore distance precision is employed, which is defined as the percentage
of frames where the target object is located within a center location error of
$T$ pixels [2]. The trackers are ranked using this metric with a threshold of
$T=20$ pixels. The precision plot is generated by plotting the distance
precision over a range of thresholds.

Success Plot: The precision metric only measures the localization performance
of the tracker which is not accurate to measure the target scale variations
[142]. Instead of the center location error, the IoU is employed to measure
the prediction error. Given the estimated bounding box $p$ and ground-truth
bounding box $g$, the IoU is defined as $\frac{p\cap g}{p\cup g}$. The success
rate is thus the percentage if frames where the IoU is smaller than a $T$. The
success plot is be generated by varying the overlap threshold from 0 to 1. The
trackers are ranked using area under the curve of the success plot.

Normalized Precision Plot: As distance precision is sensitive to the target
scale, Muller et al., employed normalized precision metric to evaluate
trackers based on the relative error [104]. It computes errors relative to the
target size instead of considering the absolute distance i.e.,
$||W(p_{x},p_{y})-(g_{x},g_{y})||$, where $W=\textrm{diag}(g_{x},p_{y})$. This
relative error is then plotted in the range of 0 to 0.5. The area under this
curve is called normalized precision which is used to rank trackers.

Average Overlap: This metric estimates the average of overlaps between the
ground-truth and estimated bounding boxes, as in the success plot [60].

SR0.50 and SR0.75: These metrics denote the success rate that measures the
percentage of successfully tracked frames, where the overlap precision exceeds
a threshold of 0.50 and 0.75

The One pass evaluation criteria is used as defined in [142] to measure the
tracking performance in terms of precision and success plots on OTB100, TC128,
UAV123, and LaSOT datasets. The trackers on these datasets are evaluated by
initializing bounding box on the first frame and letting it run until the end
of the sequence.

In the VOT series, a tracker is reset once it drifts off the target. Following
the VOT evaluation protocols [66, 47, 69], the trackers are compared in terms
of Accuracy (A), Robustness (R), and Expected Average Overlap (EAO) metrics. A
is the average overlap between the predicted and ground truth bounding boxes
during successful tracking periods. R measures how many times the tracker
loses the target (fails) during tracking. A reset mechanism starts after some
frames once tracker losses the target object. EAO is an estimator of the
average overlap a tracker is expected to attain on a large collection of
short-term sequences with the same visual properties as the given dataset.

(a) OTB100

(c) GOT-10K

(b) LaSOT

(d) TrackingNet

Figure 6: Tracking performance trends on popular benchmarks: (a) OTB100, (b)
LaSOT, (c) GOT-10k and (d) TrackingNet, during 2014 - 2020. The representative
trackers are: SAMF [81], DSST [26], KCF [53], HCF [96], SRDCF [29], DeepSRDCF
[28], STAPLE [3], SiamFC [4], GOTURN [52], SINT [127], CACF [103], ECO [24],
CCOT [22], BACF [41], CSR-DCF [95], CFNET [128], MemTrack [153], UPDT [7],
SiamRPN [75], DaSiamRPN [171], DRT [120], C-RPN [37], MCCT [136], DiMP [5],
SiamMask [140], D3S [94], PrDiMP [25], ATOM [23], SiamRPN++ [74], SiamR-CNN
[130], and SiamAttn [158].

###  5.3 Quantitative Comparison

Tables II, III, IV, and V present the performance comparison of representative
DCF-based trackers on nine tracking benchmarks.333For a more detailed
comparison, please see supplementary material. While earlier DCF-based
trackers employing deep features achieve promising performance on OTB100, they
provide inferior results on recent more challenging large-scale datasets such
as LaSOT. For instance, ECO achieves an impressive PR score of 91.0$\%$ on OTB
but obtains a PR score of only 30.1$\%$ on LaSOT. In contrast, recent end-to-
end DCF frameworks such as, DiMP and its successor PrDiMP achieve impressive
performance on OTB100 as well as on LaSOT. For instance, PrDiMP achieves PR
scores of 90.3$\%$, 87.8$\%$ and 60.9$\%$ on OTB100, UAV123, and LaSOT,
respectively. Among existing DCF-based trackers, DiMP and PrDiMP achieve
superior results on most benchmarks. PrDiMP achieves top performance on
UAV123, LaSOT, and GOT-10K, while also achieving competitive results (among
top-three) on OTB100, VOT2016, and VOT2018-ST. The success of these modern DCF
trackers (DiMP and [25, 6]) is due to their efficient end-to-end trainable
architectures that are capable of learning a discriminative target model
prediction by fully utilizing both target and background appearance
information. These trackers employ a dedicated optimization process to learn a
powerful model in few iterations. For instance, PrDiMP utilizes a more general
Newton approximation for addressing KL-divergence objective. Further, these
modern trackers comprise a dedicated target estimation component to perform
deep bounding box regression and also circumvent the problem of boundary
artefacts.

Tables II, III, IV, and V also present the performance comparison of
representative Siamese trackers on nine benchmarks 3. Among recent Siamese
methods, we observe trackers to focus on different fundamental issues such as,
online model update, re-detection components, improved region refinement,
effective box regression, and bridging the gap between object tracking and
object segmentation. For instance, SiamAttn introduces an attention mechanism
to adaptively update the target template and obtains the best performance on
OTB100, UAV123, VOT2016, while also achieving competitive results (among top-
three) on LaSOT, VOT2018-ST, and TrackingNet. SiamAttn obtains AUC scores of
71.2$\%$, 65.0$\%$, 56.0$\%$, and 75.2$\%$ on OTB100, UAV123, LaSOT, and
TrackingNet, respectively. Further, it obtains EAO scores of 53.7$\%$ and
47.0$\%$ on VOT2016 and VOT2018-ST, respectively. SiamR-CNN introduces a re-
detection architecture combined with a tracklet-based dynamic programming
scheme and obtains top performance on TC128 (64.9$\%$ AUC score), LaSOT
(64.8$\%$ AUC score), GOT-10K (64.9$\%$ mAO score) and TrackingNet (81.2$\%$
AUC score), while also achieving competitive results (among top-three) on
other datasets. Ocean introduces an approach to refine the imprecise bounding-
box predictions along with learning object-aware features and obtains the top
performance on VOT2018-ST (48.9$\%$ EAO score). D3S is a single-shot
segmentation tracker employing two target models with complementary properties
and obtains the best results on VOT2018-ST and VOT2020-ST (48.9$\%$ and
43.9$\%$ EAO score).

Figure 6 shows the tracking performance improvement trend on different
benchmarks (OTB100, LaSoT, GOT-10k, and TrackingNet) in recent years. We can
observe that the performance on OTB100 has saturated in recent years with
several visual trackers obtaining over 90$\%$ PR score (Table II), likely due
to numerous relatively easy videos. While, the recently introduced LaSOT,
GOT10K, and TrackingNet all show a similar trend with consistent improvements
obtained by recent trackers on these datasets. We also observed a similar
trend on the VOT dataset in Figure 1. For instance, the best reported AUC
score on LaSOT is still around 65$\%$. Similarly, there is still a significant
room to further improve the tracking performance in the VOT dataset, despite
witnessing an impressive leap in performance in recent years. This suggests
that these new challenging benchmarks are still very challenging for SOTA
trackers and their introduction is significantly contributing to pushing the
boundaries of visual tracking research.

The aforementioned datasets also have radically different properties and
characteristics. LaSOT and UAV123 contain long sequences and multiple
distractors. Trackers achieving high performance here demonstrate substantial
robustness and re-detection capabilities. We observe that the recent trackers
DiMP and PrDiMP achieve strong results, and that the distractor-aware track
generation in SiamR-CNN improves the robustness in such scenarios. In contrast
to LaSOT, TrackingNet, and GOT10k contain short sequences where robustness and
re-detection capability is of much lesser importance. Instead, these datasets
reward trackers with highly accurate bounding box prediction, such as SiamR-
CNN and PrDiMP. Among Siamese trackers, we obsserve that SiamR-CNN and
SiamAttn achieve the most consistently good results across several datasets.
The exception being LaSOT in case of SiamAttn, while SiamR-CNN struggles on
VOT. Among DCF-based methods, PrDiMP consistently achieves SOTA results across
all evaluated datasets.

TABLE II: Performance comparison of representative DCF and Siamese trackers,
in terms of Precision Rate (PR) at a threshold of 20 pixels, Normalized
Precision Rate (NPR), and Area Under the Curve (AUC), on OTB100, TC128,
UAV123, UAV20L, and LaSOT datasets. For both DCF and Siamese trackers, the
best two results are in red and blue font, respectively.

OTB100 TC128 UAV123 UAV20L LaSOT Speed (fps) DCF Trackers Publication PR AUC
PR AUC PR AUC PR AUC PR NPR AUC CPU GPU MOSSE [10] CVPR2010 0.414 0.311 - -
0.466 0.297 0.339 0.223 - - - 669 - SAMF [81] ECCV2014 0.743 0.548 - 0.467
0.592 0.396 0.457 0.317 0.203 0.239 0.233 7 - KCF [53] PAMI2015 0.695 0.477
0.588 0.418 0.523 0.331 0.311 0.198 0.166 0.190 0.178 172 - HCF [96] ICCV2015
0.837 0.562 0.703 0.488 - - 0.500 0.299 0.241 0.286 0.250 - 10.4 DeepSRDCF
[28] ICCV-W2015 0.851 0.649 0.740 0.536 - - - - - - - - 0.2 fDSST [27]
PAMI2016 0.726 0.551 0.575 0.435 - - 0.422 0.300 0.184 0.208 0.203 54.3 -
STAPLE [3] CVPR2016 0.784 0.581 0.665 0.498 - - - - 0.239 0.278 0.243 80 - ECO
[24] CVPR2017 0.910 0.691 0.800 0.605 0.741 0.537 0.604 0.435 0.301 0.338
0.324 60 8 PTAV [36] ICCV2017 0.849 0.635 0.741 0.544 - - 0.624 0.423 0.254
0.274 0.250 - 25 DCFNET [138] Arxiv2017 0.816 0.601 - - - - - - - - - - 65.94
STRCF [76] CVPR2018 0.864 0.683 0.753 0.601 - - - - 0.298 0.340 0.308 31.5 5.3
UPDT [7] ECCV2018 0.931 0.702 - 0.622 - 0.550 - - - - - - - DSLT [91] ECCV2018
0.909 0.660 0.807 0.586 0.746 0.530 - - 0.324 0.323 - 5.7 LSART [121] CVPR2018
0.923 0.672 - - - - - - - - - - 1 TRACA [16] CVPR2018 0.816 0.603 - - - - - -
0.227 0.278 0.257 - 101.3 DRT [120] CVPR2018 0.923 0.699 - - - - - - - - - - -
HCFTs [97] PAMI2019 0.870 0.598 - - - - - - - - - - 6.70 GFS-DCF [145]
ICCV2019 0.932 0.693 - - - - - - - - - 8 RPCF [123] CVPR2019 0.929 0.690 - - -
- - - - - - - 5 ASRCF [20] CVPR2019 0.922 0.692 0.825 0.603 - - - - 0.337
0.391 0.359 - 28 ATOM [23] CVPR2019 0.879 0.667 - - 0.856 0.650 - - 0.505
0.576 0.515 - 30 DiMP [5] ICCV2019 0.902 0.684 - - 0.858 0.654 - - 0.563 0.650
0.569 - 40 DSLT+ [92] PAMI2020 0.918 0.662 - - 0.768 0.535 - - 0.342 - 0.340 -
6.3 PrDiMP [25] CVPR2020 0.903 0.696 - - 0.878 0.680 - - 0.609 - 0.598 - 30
KYS [6] ECCV2020 - 0.695 - - - - - - - 0.633 0.554 - 20 Siamese Trackers
Publication PR AUC PR AUC PR AUC PR AUC PR NPR AUC CPU GPU SiamFC [4] ECCV2016
0.771 0.582 0.688 0.503 0.691 0.461 0.613 0.399 0.339 0.420 0.336 - 86 GOTURN
[52] ECCV2016 - 0.410 - - - - - - - - - - 100 SINT [127] CVPR016 0.788 0.592 -
- - - - - 0.295 0.354 0.314 - 4 CFNET [128] CVPR2017 0.748 0.568 0.607 0.441
0.651 0.436 0.570 0.349 0.259 0.312 0.275 - 75 DSiam [46] ICCV2017 - - 0.711
0.505 - - - - 0.322 0.405 0.333 - 45 DaSiamRPN [171] ECCV2018 0.880 0.659 - -
0.796 0.586 0.838 0.617 - 0.496 0.415 - 160 RTINET [155] ECCV2018 - 0.682 -
0.602 - - - - - - - - 9 SiamRPN [75] CVPR2018 0.851 0.637 - - 0.796 0.527
0.617 0.454 - - 0.433 - 160 GradNet [77] ICCV2019 0.861 0.639 0.764 0.556 - -
- - 0.351 - 0.365 - 80 SiamDW [167] CVPR2019 0.900 0.670 - - - - - - - - 0.384
- 35 TADT [80] CVPR2019 0.866 0.660 - 0.562 - - - - - - - - 33.7 SiamRPN++
[74] CVPR2019 0.914 0.696 - - 0.807 0.613 - - 0.491 0.569 0.496 - 35 SPLT
[151] ICCV2019 - - - - - - - - 0.396 0.494 0.426 - 25.7 SPM [131] CVPR2019
0.899 0.687 - - - - - - - - - - 120 CLNET [33] ECCV2020 - - - - 0.830 0.633 -
- 0.494 0.574 0.499 - 45.6 Ocean [168] ECCV2020 0.920 0.684 - - - - - - 0.566
- 0.560 - 26 ROAM [154] CVPR2020 0.904 0.680 - - - - - - 0.445 - 0.447 - 20
SiamBAN [15] CVPR2020 0.910 0.696 - - 0.833 0.631 - - 0.521 0.598 0.514 - 40
SiamFC++ [146] AAAI2020 - 0.683 - - - - - - - - 0.544 - 60 SiamR-CNN [130]
CVPR2020 0.891 0.701 - 0.649 0.834 0.649 - - 0.684 0.722 0.648 - 4.7 UDT++
[135] IJCV2020 0.843 0.639 0.725 0.552 - - - - - - 0.305 - 55 SiamAttn [158]
CVPR2020 0.926 0.712 - - 0.845 0.650 - - - 0.648 0.560 - 45 CSA [149] CVPR2020
0.914 0.696 - - - - - - - 0.569 0.496 - 109 D3S [94] CVPR2020 0.841 0.608 - -
- - - - 0.490 - 0.488 - 25 SiamCAR [45] CVPR2020 0.910 0.698 - - 0.760 0.614 -
- 0.510 0.600 0.507 - 52.27

TABLE III: Performance comparison of representative DCF and Siamese trackers,
in terms of EAO, A, and R, on VOT2016 and VOT2018-ST. In case of DCF and
Siamese trackers, the best two results are shown in red and blue fonts,
respectively.

VOT2016 VOT2018-ST DCF Trackers Publication EAO$\uparrow$ A$\uparrow$
R$\downarrow$ EAO$\uparrow$ A$\uparrow$ R$\downarrow$ Speed MOSSE [10]
CVPR2010 - - - 0.139 0.503 0.988 - SAMF [81] ECCV2014 0.186 0.507 0.587 0.093
0.484 1.302 4.099 KCF [53] PAMI2015 0.192 0.489 122 0.135 0.447 0.773 21.78
HCF [96] ICCV2015 0.220 0.450 0.396 - - - 1.057 DeepSRDCF [28] ICCV-W2015
0.276 0.528 0.326 0.154 0.492 0.707 0.38 STAPLE [3] CVPR2016 0.295 0.544 0.378
0.171 0.488 0.613 11.14 ECO [24] CVPR2017 0.374 0.546 11.67 0.280 0.484 0.276
- DCFNET [138] Arxiv2017 - - - 0.182 0.470 0.543 - STRCF [76] CVPR2018 0.313
0.550 0.920 0.345 0.523 0.215 - UPDT [7] ECCV2018 - - - 0.378 0.536 0.182 -
DSLT [91] ECCV2018 0.332 0.541 15.48 0.325 0.543 0.224 - LSART [121] CVPR2018
0.323 0.495 0.215 0.323 0.493 0.218 - TRACA [16] CVPR2018 - - - 0.137 0.424
0.857 - DRT [120] CVPR2018 0.442 0.569 0.140 0.356 0.507 0.155 - GFS-DCF [145]
ICCV2019 - - - 0.397 0.511 0.143 - RPCF [123] CVPR2019 - - - 0.316 0.500 0.234
- ASRCF [20] CVPR2019 0.391 0.563 0.187 0.328 0.494 0.234 - ATOM [23] CVPR2019
0.424 0.617 0.190 0.401 0.590 0.204 - DiMP [5] ICCV2019 0.479 0.617 0.190
0.440 0.597 0.153 - DSLT+ [92] PAMI2020 0.364 0.551 9.253 0.274 0.500 0.279 -
PrDiMP [25] CVPR2020 0.476 0.652 0.140 0.442 0.618 0.165 - KYS [6] ECCV2020 -
- - 0.462 0.609 0.143 - Siamese Trackers Publication EAO$\uparrow$ A$\uparrow$
R$\downarrow$ EAO$\uparrow$ A$\uparrow$ R$\downarrow$ Speed SiamFC [4]
ECCV2016 0.235 0.532 0.461 0.188 0.503 0.585 9.213 CFNET [128] CVPR2017 0.201
- - - - - - DSiam [46] ICCV2017 - - - 0.196 0.512 0.646 - DaSiamRPN [171]
ECCV2018 0.411 0.610 0.22 0.326 0.570 0.337 - RTINET [155] ECCV2018 0.298
0.570 1.070 - - - - SiamRPN [75] CVPR2018 0.344 0.560 1.12 0.244 0.490 0.464 -
GradNet [77] ICCV2019 - - - 0.247 0.507 0.375 - SiamDW [167] CVPR2019 0.370
0.580 0.240 0.300 0.520 0.410 - SiamRPN++ [74] CVPR2019 0.370 0.580 0.240
0.414 0.600 0.234 - TADT [80] CVPR2019 0.299 0.550 1.170 - - - - SPM [131]
CVPR2019 0.434 0.620 0.210 0.338 0.580 0.300 - Ocean [168] ECCV2020 - - -
0.489 0.592 0.117 - ROAM [154] CVPR2020 0.441 0.599 0.174 0.380 0.543 0.195 -
SiamBAN [15] CVPR2020 0.505 0.632 0.150 0.452 0.597 0.178 - SiamFC++ [146]
AAAI2020 - - - 0.426 0.587 0.183 - SiamR-CNN [130] CVPR2020 0.461 0.645 0.173
0.408 0.609 0.220 - UDT++ [135] IJCV2020 0.309 0.540 62 0.230 0.490 88 -
SiamAttn [158] CVPR2020 0.537 0.680 0.140 0.470 0.630 0.160 - CSA [149]
CVPR2020 - - - 0.414 0.600 0.234 - D3S [94] CVPR2020 0.493 0.660 0.131 0.489
0.640 0.150 -

TABLE IV: Performance comparison of representative DCF and Siamese trackers,
in terms of EAO, A, and R on VOT2020-ST. In case of DCF and Siamese trackers,
the best two results are in red and blue fonts, respectively.

VOT2020-ST DCF Trackers Publication EAO$\uparrow$ A$\uparrow$ R$\uparrow$ KCF
[53] PAMI2015 0.154 0.407 0.432 CSR-DCF [95] CVPR2017 0.193 0.406 0.582 UPDT
[7] ECCV2018 0.278 0.465 0.755 DiMP [5] ICCV2019 0.274 0.457 0.740 ATOM [23]
CVPR2019 0.271 0.462 0.734 Siamese Trackers Publication EAO$\uparrow$
A$\uparrow$ R$\uparrow$ SiamFC [4] ECCV2016 0.179 0.418 0.502 SiamMask [140]
CVPR2019 0.321 0.624 0.648 D3S [94] CVPR2020 0.439 0.699 0.769 Ocean [168]
ECCV2020 0.430 0.693 0.754

TABLE V: Performance comparison of representative DCF and Siamese trackers, in
terms of mean Average Overlape (mAO), mSR0.50, and mSR0.75, PR, NPR, and AUC,
on GOT-10K and TrackingNet. In case of DCF and Siamesetrackers, the best two
results are in red and blue fonts, respectively.

GOT-10K TrackingNet DCF Trackers Publication mAO mSR0.50 mSR0.75 PR NPR AUC
MOSSE [10] CVPR2010 - - - 0.326 0.442 0.388 SAMF [81] ECCV2014 0.330 0.331
0.130 0.477 0.598 0.504 KCF [53] PAMI2015 0.279 0.263 0.099 0.419 0.546 0.447
HCF [96] ICCV2015 0.379 0.380 0.134 - - - STAPLE [3] CVPR2016 0.332 0.333
0.135 0.470 0.603 0.528 fDSST [27] PAMI2016 0.289 0.278 0.121 - - - ECO [24]
CVPR2017 0.395 0.407 0.170 0.492 0.618 0.554 DCFNET [138] Arxiv2017 0.364
0.378 0.144 - - - STRCF [76] CVPR2018 0.449 0.481 0.169 - - - UPDT [7]
ECCV2018 - - - 0.557 0.703 0.611 GFS-DCF [145] ICCV2019 - - - 0.565 0.717
0.609 ATOM [23] CVPR2019 0.556 0.634 0.402 0.648 0.771 0.703 DiMP [5] ICCV2019
0.611 0.717 0.492 0.800 0.801 0.740 PrDiMP [25] CVPR2020 0.634 0.738 0.543
0.704 0.816 0.758 KYS [6] ECCV2020 - - - 0.633 0.554 Siamese Trackers
Publication mAO mSR0.50 mSR0.75 PR NPR AUC SiamFC [4] ECCV2016 0.392 0.426
0.135 0.533 0.654 0.571 GOTURN [52] ECCV2016 0.418 0.475 0.163 - - - CFNET
[128] CVPR2017 0.434 0.481 0.190 0.533 0.654 0.578 DSiam [46] ICCV2017 0.417
0.461 0.149 - 0.405 0.333 DaSiamRPN [171] ECCV2018 0.444 0.536 0.220 0.591
0.733 0.638 SiamRPN [75] CVPR2018 0.481 0.581 0.270 - - - SiamDW [167]
CVPR2019 0.411 0.456 0.154 - - - SiamRPN++ [74] CVPR2019 0.518 0.618 0.325
0.694 0.800 0.733 SPM [131] CVPR2019 0.513 - - 0.660 0.771 0.712 Ocean [168]
ECCV2020 0.611 0.721 - - - - ROAM [154] CVPR2020 0.465 0.532 0.236 0.623 0.754
0.670 SiamFC++ [146] AAAI2020 0.595 0.695 0.479 0.694 0.800 0.733 SiamR-CNN
[130] CVPR2020 0.649 0.728 0.597 0.800 0.854 0.812 UDT++ [135] IJCV2020 - - -
0.495 0.633 0.563 SiamAttn [158] CVPR2020 - - - 0.715 0.817 0.752 D3S [94]
CVPR2020 0.597 0.676 0.462 0.664 0.768 0.728 SiamCAR [45] CVPR2020 0.569 0.670
0.415 - - -

###  5.4 Speed Comparison

The tracking speed is another very important metric to evaluate the trackers
especially to meet the real-time requirements. However, evaluating the
tracking speed is not straightforward since a number of key-factors influence
including feature extraction, model update method, programming language, and
most importantly the hardware that the trackers are implemented on. To reduce
the influence of the hardware, the VOT2014 committee has introduced a new unit
known as Equivalent Filter Operations (EFO) that reports the tracking speed in
terms of a predefined filtering operations that the toolkit automatically
carries out prior to running the experiments [66]. Tables III presents the
tracking speed in terms of EFO measures of SOTA trackers. Trackers KCF and
STAPLE clearly show the best speed while deep trackers including DeepSRDCF and
HCF show competitive performance but worst speed.

##  6 Discussion and Conclusions

Here, we summarize the common lessons learned and a set of recommendations for
future work in generic object tracking.

Importance of end-to-end tracking framework: These frameworks have
demonstrated excellent performance recently. While end-to-end offline learning
is a prerequisite for Siamese tracking, recent DCF methods have also adopted
this paradigm with success. Thus, learning the underlying features, along with
prediction heads, have proven crucial for optimal performance. This has only
been possible in the last few years, with the introduction of large scale
training datasets.

Importance of robust target modeling: While Siamese-based approaches have
excelled in many areas, end-to-end DCF-based methods still demonstrate an
advantage in challenging long-term tracking scenarios, such as LaSOT. This
shows the importance of robust online target appearance modelling, achieved by
embedding discriminative learning modules into the network architecture. Such
methods effectively integrate background appearance cues and can easily be
updated during the tracking procedure using online learning.

Target state estimation: Siamese-based approaches have driven the advancement
of more accurate bounding box regression by leveraging progress in the
neighboring field of object detection. Recent one-stage (anchor free) based
approaches, such as Ocean, achieve simple, accurate, and efficient bounding
box regression. Furthermore, these strategies are generic and can easily be
integrated in any visual tracking architecture.

Role of segmentation: Although the task of bounding box regression have seen
substantial progress in tracking, such a target state model is inherently
limited. Instead, segmentation promises a pixel-precise estimation of the
target, which is highly desired in many applications. Moreover, segmentation
offers the potential of improving the tracking itself, by for example aiding
the target model update. Furthermore, as demonstrated in for instance [150],
segmentation further aids the regression of accurate bounding boxes and helps
estimating the scale at which the object is tracked. Future efforts should
therefore be aimed towards integrating accurate segmentation into robust
tracking frameworks.

Backbone architectures: The ResNet architecture has withstood the test of time
in several computer vision applications. In visual tracking, it remains the
most popular choice for feature extraction. The architecture is simple,
effective, and allows for the extraction of features at multiple resolutions.
While effective when pushing the boundaries of SOTA, it is still remains
computationally costly for real-time applications on platforms with harder
computational constraints, such as CPUs. A highly interesting future direction
is therefore to develop efficient backbone networks tailored for the tracking
task [148].

Estimating geometry: In some applications, e.g. in augmented reality, a
precise geometric transformation between frames is necessary for the added
graphics to appear as attached to objects. For planar objects, at least an
affine transformation, but preferably a homography, between a reference and
current view is required. For non-planar objects, the problems is linked to
online reconstruction of the 3D shape of the object. Neither DCF nor Siamese
methods have been equipped to provide precise geometric correspondence, which
remains an open research issue.

Role of Transformers: Transformers have recently shown success in a variety of
vision tasks [34]. Very recent tracking approaches employ transformers in
different ways. [14, 137] utilize transformers for feature enhancement, in
combination with either DCF of Siamese trackers. [100] employs a transformer
to associate the target object between frames in the presence of distractors.
In particular, STARK employs a transformer module for target detection and
bounding box regression [147]. In this work, the transformer thus takes on the
role of the DCF or Siamese correlation component. The transformer, with its
embedded attention module, bares interesting commonalities with DCF. Most
importantly, it allows for integration of background appearance information
through global operations. Moreover, the transformer employed in STARK
predicts a correlation filter. It therefore can be seen as a replacement of
the optimization based filter prediction in DCF. Much future effort is needed
to further analyze the effectiveness of transformers, as well as their
relation to the DCF and Siamese paradigms.

Future directions: In the recent past, with the introduction of precise
segmentation capabilities to visual trackers, the connection to video
segmentation has become apparent. Some top visual tracking methods perform
well on video segmentation benchmarks [94, 140], despite the restriction to
causal processing of the input.

In the near future, we expect convergence with areas as SLAM, which estimates
the relative position of the camera model of the scene that is build online,
assuming a rigid scene. As soon as the rigidity assumption is dropped, as in
multi-body SLAM, the output for the object not corresponding to the background
can be seen as tracking of the object, with an estimated 3D shape model added.

The steadily improving performance of single-object tracking, including
robustness to nearly identical distractors, demonstrated e.g., in sequences of
groups of insects, seem like to lead to convergence with the research in the
area of multi-target tracking. We are probably likely to see strong progress
in the Open-world Tracking Problem, as described in [89], i. e., of methods
capable of tracking, segmenting, and shape modelling of multiple objects from
a-priori unknown classes.

## References

  * [1] K. Alex, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional networks,” in _NIPS_ , 2012.
  * [2] B. Babenko, M.-H. Yang, and S. Belongie, “Robust object tracking with online multiple instance learning,” _in T-PAMI_ , vol. 33, no. 8, pp. 1619–1632, 2010.
  * [3] L. Bertinetto, J. Valmadre, S. Golodetz, O. Miksik, and P. H. Torr, “Staple: Complementary learners for real-time tracking,” in _CVPR_ , 2016.
  * [4] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, “Fully-convolutional siamese networks for object tracking,” in _ECCV_. Springer, 2016.
  * [5] G. Bhat, M. Danelljan, L. V. Gool, and R. Timofte, “Learning discriminative model prediction for tracking,” in _ICCV_ , 2019.
  * [6] G. Bhat, M. Danelljan, L. Van Gool, and R. Timofte, “Know your surroundings: Exploiting scene information for object tracking,” in _ECCV_ , 2020.
  * [7] G. Bhat, J. Johnander, M. Danelljan, F. Shahbaz Khan, and M. Felsberg, “Unveiling the power of deep tracking,” in _ECCV_ , 2018.
  * [8] G. Bhat, F. J. Lawin, M. Danelljan, A. Robinson, M. Felsberg, L. Van Gool, and R. Timofte, “Learning what to learn for video object segmentation,” in _ECCV_ , 2020.
  * [9] A. Bibi, M. Mueller, and B. Ghanem, “Target response adaptation for correlation filter tracking,” in _ECCV_ , 2016.
  * [10] D. S. Bolme, J. R. Beveridge, B. A. Draper, and Y. M. Lui, “Visual object tracking using adaptive correlation filters,” in _CVPR_ , 2010.
  * [11] S. Boyd, N. Parikh, and E. Chu, _Distributed optimization and statistical learning via the alternating direction method of multipliers_. Now Publishers Inc, 2011.
  * [12] J. Bromley, J. W. Bentz, L. Bottou, I. Guyon, Y. LeCun, C. Moore, E. Säckinger, and R. Shah, “Signature verification using a “siamese” time delay neural network,” _in IJPRAI_ , vol. 7, no. 04, pp. 669–688, 1993.
  * [13] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman, “Return of the devil in the details: Delving deep into convolutional nets,” _in BMVC_ , 2014.
  * [14] X. Chen, B. Yan, J. Zhu, D. Wang, X. Yang, and H. Lu, “Transformer tracking,” _in arXiv preprint arXiv:2103.15436_ , 2021.
  * [15] Z. Chen, B. Zhong, G. Li, S. Zhang, and R. Ji, “Siamese box adaptive network for visual tracking,” in _CVPR_ , 2020.
  * [16] J. Choi, H. Jin Chang, T. Fischer, S. Yun, K. Lee, J. Jeong, Y. Demiris, and J. Young Choi, “Context-aware deep feature compression for high-speed visual tracking,” in _CVPR_ , 2018.
  * [17] J. Choi, H. Jin Chang, J. Jeong, Y. Demiris, and J. Young Choi, “Visual tracking using attention-modulated disintegration and integration,” in _CVPR_ , 2016.
  * [18] J. Choi, H. Jin Chang, S. Yun, T. Fischer, Y. Demiris, and J. Young Choi, “Attentional correlation filter network for adaptive visual tracking,” in _CVPR_ , 2017.
  * [19] S. Chopra, R. Hadsell, and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification,” in _CVPR_ , 2005\.
  * [20] K. Dai, D. Wang, H. Lu, C. Sun, and J. Li, “Visual tracking via adaptive spatially-regularized correlation filters,” in _CVPR_ , 2019.
  * [21] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _CVPR_ , 2005.
  * [22] M. Danelljan, A. Robinson, F. K. Khan, and M. Felsberg, “Beyond correlation filters: Learning continuous convolution operators for visual tracking,” in _ECCV_ , 2016.
  * [23] M. Danelljan, G. Bhat, F. S. Khan, and M. Felsberg, “Atom: Accurate tracking by overlap maximization,” in _CVPR_ , 2019.
  * [24] M. Danelljan, G. Bhat, F. Shahbaz Khan, and M. Felsberg, “Eco: Efficient convolution operators for tracking,” in _CVPR_ , 2017.
  * [25] M. Danelljan, L. V. Gool, and R. Timofte, “Probabilistic regression for visual tracking,” in _CVPR_ , 2020.
  * [26] M. Danelljan, G. Häger, F. Khan, and M. Felsberg, “Accurate scale estimation for robust visual tracking,” in _BMVC_ , 2014.
  * [27] M. Danelljan, G. Häger, F. S. Khan, and M. Felsberg, “Discriminative scale space tracking,” _in T-PAMI_ , vol. 39, no. 8, pp. 1561–1575, 2016.
  * [28] M. Danelljan, G. Hager, F. Shahbaz Khan, and M. Felsberg, “Convolutional features for correlation filter based visual tracking,” in _ICCV-W_ , 2015\.
  * [29] M. Danelljan, G. Hager, F. Shahbaz Khan, and M. Felsberg, “Learning spatially regularized correlation filters for visual tracking,” in _ICCV_ , 2015\.
  * [30] M. Danelljan, F. Shahbaz Khan, M. Felsberg, and J. Van de Weijer, “Adaptive color attributes for real-time visual tracking,” in _CVPR_ , 2014.
  * [31] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _CVPR_ , 2009.
  * [32] X. Dong and J. Shen, “Triplet loss in siamese network for object tracking,” in _ECCV_ , 2018.
  * [33] X. Dong, J. Shen, L. Shao, and F. Porikli, “Clnet: A compact latent network for fast adjusting siamese trackers,” in _ECCV_ , 2020.
  * [34] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _in arXiv preprint arXiv:2010.11929_ , 2020.
  * [35] H. Fan, L. Lin, F. Yang, P. Chu, G. Deng, S. Yu, H. Bai, Y. Xu, C. Liao, and H. Ling, “Lasot: A high-quality benchmark for large-scale single object tracking,” in _CVPR_ , 2019.
  * [36] H. Fan and H. Ling, “Parallel tracking and verifying: A framework for real-time and high accuracy visual tracking,” in _ICCV_ , 2017.
  * [37] H. Fan and H. Ling, “Siamese cascaded region proposal networks for real-time visual tracking,” in _CVPR_ , 2019.
  * [38] M. Felsberg, in _Advanced Methods and Deep Learning in Computer Vision_ , ch. 9.
  * [39] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, “Object detection with discriminatively trained part-based models,” _in T-PAMI_ , vol. 32, no. 9, pp. 1627–1645, 2009.
  * [40] M. Fiaz, A. Mahmood, S. Javed, and S. K. Jung, “Handcrafted and deep trackers: Recent visual object tracking approaches and trends,” _in CSUR_ , vol. 52, no. 2, pp. 1–44, 2019.
  * [41] H. K. Galoogahi, A. Fagg, and S. Lucey, “Learning background-aware correlation filters for visual tracking,” in _CVPR_ , 2017.
  * [42] R. Girshick, “Fast r-cnn,” in _ICCV_ , 2015.
  * [43] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in _CVPR_ , 2014\.
  * [44] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, _Deep learning_. in MIT press Cambridge, 2016, vol. 1, no. 2.
  * [45] D. Guo, J. Wang, Y. Cui, Z. Wang, and S. Chen, “Siamcar: Siamese fully convolutional classification and regression for visual tracking,” in _CVPR_ , 2020.
  * [46] Q. Guo, W. Feng, C. Zhou, R. Huang, L. Wan, and S. Wang, “Learning dynamic siamese network for visual object tracking,” in _ICCV_ , 2017.
  * [47] S. Hadfield, R. Bowden, and K. Lebeda, “The visual object tracking vot2016 challenge results,” in _ECCV-W_ , 2016.
  * [48] A. Hadjidimos, “Successive overrelaxation (sor) and related methods,” _in JCAM_ , vol. 123, no. 1-2, pp. 177–199, 2000.
  * [49] X. Han, T. Leung, Y. Jia, R. Sukthankar, and A. C. Berg, “Matchnet: Unifying feature and metric learning for patch-based matching,” in _CVPR_ , 2015\.
  * [50] A. He, C. Luo, X. Tian, and W. Zeng, “A twofold siamese network for real-time object tracking,” in _CVPR_ , 2018.
  * [51] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [52] D. Held, S. Thrun, and S. Savarese, “Learning to track at 100 fps with deep regression networks,” in _ECCV_ , 2016.
  * [53] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speed tracking with kernelized correlation filters,” _in T-PAMI_ , vol. 37, no. 3, pp. 583–596, 2015.
  * [54] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “Exploiting the circulant structure of tracking-by-detection with kernels,” in _ECCV_ , 2012.
  * [55] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speed tracking with kernelized correlation filters,” _in T-PAMI_ , vol. 37, no. 3, pp. 583–596, 2014.
  * [56] Z. Hong, Z. Chen, C. Wang, X. Mei, D. Prokhorov, and D. Tao, “Multi-store tracker (muster): A cognitive psychology inspired approach to object tracking,” in _CVPR_ , 2015, pp. 749–758.
  * [57] R. A. Horn and C. R. Johnson, _Matrix analysis_. in CUP, 2012.
  * [58] C. Huang, S. Lucey, and D. Ramanan, “Learning policies for adaptive tracking with deep feature cascades,” in _ICCV_ , 2017.
  * [59] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama _et al._ , “Speed/accuracy trade-offs for modern convolutional object detectors,” in _CVPR_ , 2017.
  * [60] L. Huang, X. Zhao, and K. Huang, “Got-10k: A large high-diversity benchmark for generic object tracking in the wild,” _in T-PAMI_ , 2019.
  * [61] Z. Huang, C. Fu, Y. Li, F. Lin, and P. Lu, “Learning aberrance repressed correlation filters for real-time uav tracking,” in _ICCV_ , 2019.
  * [62] B. Jiang, R. Luo, J. Mao, T. Xiao, and Y. Jiang, “Acquisition of localization confidence for accurate object detection,” in _ECCV_ , 2018.
  * [63] U. Kart, A. Lukezic, M. Kristan, J.-K. Kamarainen, and J. Matas, “Object tracking by reconstruction with view-specific discriminative correlation filters,” in _CVPR_ , 2019.
  * [64] H. Kiani Galoogahi, T. Sim, and S. Lucey, “Multi-channel correlation filters,” in _ICCV_ , 2013.
  * [65] H. Kiani Galoogahi, T. Sim, and S. Lucey, “Correlation filters with limited boundaries,” in _CVPR_ , 2015.
  * [66] M. Kristan _et al._ , “The visual object tracking vot2014 challenge results,” in _ECCV-W_ , 2014, pp. 191–217.
  * [67] M. Kristan _et al._ , “The visual object tracking vot2015 challenge results,” in _ICCV-W_ , 2015.
  * [68] M. Kristan _et al._ , “The sixth visual object tracking vot2018 challenge results,” in _ECCV-W_ , 2018.
  * [69] M. Kristan, A. Leonardis, J. Matas, M. Felsberg, R. Pflugfelder, L. ˇCehovin Zajc, T. Vojir, G. Hager, A. Lukezic, A. Eldesokey _et al._ , “The visual object tracking vot2017 challenge results,” in _ICCV-W_ , 2017.
  * [70] M. Kristan, A. Leonardis, J. Matas, M. Felsberg, R. Pflugfelder, J.-K. Kämäräinen, M. Danelljan, L. Č. Zajc, A. Lukežič, O. Drbohlav _et al._ , “The eighth visual object tracking vot2020 challenge results,” in _ECCV-W_ , 2020.
  * [71] M. Kristan, J. Matas, A. Leonardis, T. Vojíř, R. Pflugfelder, G. Fernandez, G. Nebehay, F. Porikli, and L. Čehovin, “A novel performance evaluation methodology for single-target trackers,” _in T-PAMI_ , vol. 38, no. 11, pp. 2137–2155, 2016.
  * [72] H. Law and J. Deng, “Cornernet: Detecting objects as paired keypoints,” in _ECCV_ , 2018.
  * [73] A. Li, M. Lin, Y. Wu, M. Yang, and S. Yan, “NUS-PRO: A New Visual Tracking Challenge,” _in T-PAMI_ , vol. 38, no. 2, pp. 335–349, 2016.
  * [74] B. Li, W. Wu, Q. Wang, F. Zhang, J. Xing, and J. Yan, “Siamrpn++: Evolution of siamese visual tracking with very deep networks,” in _CVPR_ , 2019.
  * [75] B. Li, J. Yan, W. Wu, Z. Zhu, and X. Hu, “High performance visual tracking with siamese region proposal network,” in _CVPR_ , 2018.
  * [76] F. Li, C. Tian, W. Zuo, L. Zhang, and M.-H. Yang, “Learning spatial-temporal regularized correlation filters for visual tracking,” in _CVPR_ , 2018\.
  * [77] P. Li, B. Chen, W. Ouyang, D. Wang, X. Yang, and H. Lu, “Gradnet: Gradient-guided network for visual object tracking,” in _ICCV_ , 2019\.
  * [78] P. Li, D. Wang, L. Wang, and H. Lu, “Deep visual tracking: Review and experimental comparison,” _in PR_ , vol. 76, pp. 323–338, 2018.
  * [79] X. Li, W. Hu, C. Shen, Z. Zhang, A. Dick, and A. V. D. Hengel, “A survey of appearance models in visual object tracking,” _in T-IST_ , vol. 4, no. 4, pp. 1–48, 2013.
  * [80] X. Li, C. Ma, B. Wu, Z. He, and M.-H. Yang, “Target-aware deep tracking,” in _CVPR_ , 2019.
  * [81] Y. Li and J. Zhu, “A scale adaptive kernel correlation filter tracker with feature integration,” in _ECCV-W_ , 2014.
  * [82] Y. Li, J. Zhu, and S. C. Hoi, “Reliable patch trackers: Robust visual tracking by exploiting reliable patches,” in _CVPR_ , 2015.
  * [83] Y. Li, C. Fu, F. Ding, Z. Huang, and G. Lu, “Autotrack: Towards high-performance visual tracking for uav with automatic spatio-temporal regularization,” in _CVPR_ , 2020.
  * [84] P. Liang, E. Blasch, and H. Ling, “Encoding color information for visual tracking: Algorithms and benchmark,” _in T-IP_ , vol. 24, no. 12, pp. 5630–5644, 2015.
  * [85] T.-Y. Lin, Y. Cui, S. Belongie, and J. Hays, “Learning deep representations for ground-to-aerial geolocalization,” in _CVPR_ , 2015.
  * [86] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _ECCV_. Springer, 2014\.
  * [87] S. Liu, T. Zhang, X. Cao, and C. Xu, “Structural correlation filter for robust visual tracking,” in _CVPR_ , 2016.
  * [88] T. Liu, G. Wang, and Q. Yang, “Real-time part-based visual tracking via adaptive correlation filters,” in _CVPR_ , 2015.
  * [89] Y. Liu, I. E. Zulfikar, J. Luiten, A. Dave, A. Ošep, D. Ramanan, B. Leibe, and L. Leal-Taixé, “Opening up open-world tracking,” _arXiv preprint arXiv:2104.11221_ , 2021.
  * [90] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” _arXiv preprint arXiv:2103.14030_ , 2021.
  * [91] X. Lu, C. Ma, B. Ni, X. Yang, I. Reid, and M.-H. Yang, “Deep regression tracking with shrinkage loss,” in _ECCV_ , 2018.
  * [92] X. Lu, C. Ma, J. Shen, X. Yang, I. Reid, and M.-H. Yang, “Deep object tracking with shrinkage loss,” _in T-PAMI_ , 2020.
  * [93] X. Lu, W. Wang, C. Ma, J. Shen, L. Shao, and F. Porikli, “See more, know more: Unsupervised video object segmentation with co-attention siamese networks,” in _CVPR_ , 2019.
  * [94] A. Lukezic, J. Matas, and M. Kristan, “D3s-a discriminative single shot segmentation tracker,” in _CVPR_ , 2020.
  * [95] A. Lukezic, T. Vojir, L. ˇCehovin Zajc, J. Matas, and M. Kristan, “Discriminative correlation filter with channel and spatial reliability,” in _CVPR_ , 2017.
  * [96] C. Ma, J. B. Huang, X. Yang, and M. H. Yang, “Hierarchical convolutional features for visual tracking,” in _ICCV_ , 2015.
  * [97] C. Ma, J. Huang, X. Yang, and M. Yang, “Robust visual tracking via hierarchical convolutional features.” _in T-PAMI_ , vol. 41, no. 11, pp. 2709–2723, 2019.