# Exploring Fusion Strategies for Accurate RGBT Visual Object Tracking

Zhangyong Tang  Tianyang Xu  Hui Li  Xiao-Jun Wu  XueFeng Zhu  Josef Kittler
Jiangsu Provincial Engineering Laboratory of Pattern Recognition and
Computational Intelligence,
School of Artificial Intelligence and Computer Science, Jiangnan University,
214122, Wuxi, China 11footnotemark: 1 The Center for Vision, Speech and Signal
Processing, University of Surrey, Guildford, GU2 7XH, UK. 22footnotemark: 2

###### Abstract

We address the problem of multi-modal object tracking in video and explore
various options of fusing the complementary information conveyed by the
visible (RGB) and thermal infrared (TIR) modalities including pixel-level,
feature-level and decision-level fusion. Specifically, different from the
existing methods, paradigm of image fusion task is heeded for fusion at pixel
level. Feature-level fusion is fulfilled by attention mechanism with channels
excited optionally. Besides, at decision level, a novel fusion strategy is put
forward since an effortless averaging configuration has shown the superiority.
The effectiveness of the proposed decision-level fusion strategy owes to a
number of innovative contributions, including a dynamic weighting of the RGB
and TIR contributions and a linear template update operation. A variant of
which produced the winning tracker at the Visual Object Tracking Challenge
2020 (VOT-RGBT2020). The concurrent exploration of innovative pixel- and
feature-level fusion strategies highlights the advantages of the proposed
decision-level fusion method. Extensive experimental results on three
challenging datasets, i.e., GTOT, VOT-RGBT2019, and VOT-RGBT2020, demonstrate
the effectiveness and robustness of the proposed method, compared to the
state-of-the-art approaches. Code will be shared at
_https://github.com/Zhangyong-Tang/DFAT_.

###### keywords:

Visual object tracking, RGBT tracking, decision-level fusion, adaptive
weighting strategy

††journal: Information Fusion

##  1 Introduction

Visual object tracking, aiming at predicting the state of a target of interest
in video sequences, is a fundamental task in computer vision. It has received
considerable attention over the past decades owing to its wide range of
applications, such as robotics [1], video surveillance [2] and pedestrian
tracking [3], to name a few. Generally, visible spectrum (RGB) and thermal
infrared (TIR) tracking are two sub-tasks in visual object tracking. RGB data
contains more detailed perceptual information about the scenario, but the
tracking performance is sensitively affected by challenging appearance
variations, such as illumination, blur, and occlusion. On the contrary, the
TIR channel is more robust against lighting conditions, but with less
discriminative power owing to thermal crossover and insufficient texture
details. Hence, there is a major scope for taking advantage of the
complementary information provided by these two modalities to design a robust
tracker. This multi-modal approach is called RGBT tracking.

Figure 1: Qualitative comparison between our method (DFAT) and Baseline, with
ground truth on the VOT-RGBT2019 [4] dataset. Respectively, frames sampled
from video sequences _bus6_ , _face1_ , and _woman89_ are shown in the first,
second and third rows of the figure. Here ’Baseline’ means SiamRPN++ (RGBT),
which is the multi-modal extension of SiamRPN++ [5] in RGBT tracking.

With the development of RGB and thermal infrared all-in-one devices, RGBT
tracking has become an important research topic in the visual tracking
community.

In general, the existing approaches mainly conduct the cross-modal fusion of
the RGB and TIR content at pixel [6] or feature level [7, 8]. Pixel-level
fusion directly concatenates the information from one modality with the other
at the tracking input stage. This coarse fusion strategy endows the same
importance for the salient and low-frequency clues existed in both RGB and TIR
data. However, following the tracking framework, salient parts are supposed to
be more important than low-frequency parts.

In contrast, feature-level fusion performs the fusion processing in an
embedded way, enabling integrated feature extraction by deep neural networks.
Typically, the neural networks are trained on one dataset and tested on other
datasets, e.g., trained on RGBT234 [9] and tested on GTOT [10]. However, in
the VOT-RGBT challenge [4], GTOT is the largest RGBT dataset permitted for
model training. The number of video sequences in the other two RGBT datasets,
i.e., LITIV [11] and OSU-CT [12], is less than 20 and no more than 20000 image
pairs are contained in these three datasets. Consequently, since there is an
inadequate amount of training data, it is impossible to guarantee the network
would have a sufficient capacity to handle general scene content after
training on the available RGBT sequences. The deep learning trackers designed
by using a small number of training videos (limited to 65 sequences in total)
will have to be lightweight, with a high risk of over-fitting. The resulting
solutions are unable to deliver discriminative representations for both RGB
and TIR images. In conclusion, the lack of annotated RGBT tracking data limits
the representation capacity of the networks and compromises the efficiency of
the embedded-level feature fusion.

Different from the above strategies, fusion at decision level has not been
extensively studied in RGBT tracking. Traditional correlation filter (CF) [13]
based trackers, like [14], adopt only hand-crafted features to calculate the
responses for both modalities, which are then directly fused to predict the
target location. This approach is less effective when the target undergoes
severe appearance variation. In addition, advanced RGB trackers, such as [15]
and [16], are equipped with pre-trained feature extractors to get better
embeddings and improve the tracking performance. Although it has recently been
demonstrated that single modality trackers have excellent properties after
allocated superior feature extractors, extending their success to RGBT
tracking through decision-level fusion is still a challenge. Since there
exists a gap between the imaging mechanism of RGB and TIR modalities.

Considering the aforementioned concerns that fusion at all these three stages
have their shortcomings, we investigate three fusion strategies (fusion at
pixel-level, feature-level, and decision-level) and perform a comprehensive
comparison of their different characteristics in the RGBT tracking scenario.
Specifically, at pixel-level, inconsistent with the naive
addition/concatenation operator used in existing methods, we introduce a
general image fusion method to get the fused image with different focuses
lying on the salient and low-frequency parts. For online feature-level fusion,
adaptive fusion weights are employed to merge the information from the two
spectra. Before that, to eliminate the feature noise and redundancy, different
from the Weighted Random Selection strategy (WRS) [17] used in [8], an
adaptive selection method is conducted to suppress the channels with lower
discrimination by channel attention mechanism. For fusion at decision level,
originally, the results from both classification and regression branches are
simply averaged for the comparison with fusion at the other two stages.
However, under the constraint of limited amount of data, it is unexpected that
although conducted by a primitive fusion strategy, decision-level fusion has
already shown its superiority against fusion at pixel and feature levels.
Furthermore, decision-level fusion has already been verified effective in
other multi-modal tasks [18].

Drawing on this, we propose an adaptive decision-level fusion strategy for
accurate RGBT tracking (DFAT) in an end-to-end framework, whose tracking
results are intuitively presented in Fig. 1. To address the problem of
insufficient quantity of annotated RGBT data for model training, we use only
the RGB sequences to train the tracking network offline and perform cross-
modal fusion in the online tracking stage. More specifically, we use publicly
available large datasets of annotated RGB images, e.g., COCO [19] and YouTube-
VOS [20] for network training. For decision-level fusion, based on the
observation that the response scores of RGB images are higher than those of
TIR images in most sequences, which is caused by the optical difference
between RGB and TIR imaging principle, we design a novel strategy that focuses
on suppressing the bias between these two modalities. Fig. 6 also gives an
intuitive illustration of the bias. To this end, we assign an adaptive weight,
which is specified for debiasing, to each modality to harmonise the
complementary multi-modal appearance. As demonstrated by the experiments, the
proposed DFAT effectively exploits the synergies of the two modalities,
delivering superior tracking performance, as compared to other fusion
strategies.

Our contributions can be summarized as follows:

  * 1.

As potential alternatives, we concurrently investigate three innovative pixel-
level, feature-level and decision-level fusion methods for RGBT tracking, with
the results showing that the decision-level fusion is significantly superior.

  * 2.

In the context of the limited availability of annotated RGBT datasets for
training a multi-modal tracker, using the Siamese network trained on RGB
videos for embedding both modalities, RGB and TIR, we propose an online
adaptive decision-level fusion method to excavate the hidden complementaries.
A variant of the proposed tracker won the VOT-RGBT (2020) Challenge.

  * 3.

Based on the experimental phenomenon that the gap between RGB and TIR
modalities leads to classification bias, which is displayed in Fig. 6, a
dynamic weighting methodology of the RGB and TIR contributions is developed at
decision level.

  * 4.

The experimental results on the VOT-RGBT2019 and GTOT datasets demonstrate
that our method, DFAT, achieves promising performance. Moreover, on the VOT-
RGBT2020 dataset [21], DFAT defines the new state-of-the-art in RGBT tracking.

The rest of this paper is organised as follows. The related work and the
proposed method are discussed in Section II and Section III. Extensive
experiments and comprehensive analysis of the experimental results are
reported in Section IV. Finally, a conclusion on the research findings is
drawn in Section V.

##  2 Related work

As mentioned above, RGBT tracking, aiming at fusing the complementary
information between RGB and TIR modalities, is a sub-task of visual object
tracking. In this section, we briefly introduce the relevant tracking
approaches with single and multiple modalities.

###  2.1 Tracking with Single Modality

RGB tracking is the most fundamental sub-task in visual object tracking [15,
16, 22, 23, 24]. Among its numerous modelling techniques, trackers based on
Siamese networks are the widely studied in the recent deep learning paradigm.
SiamFC [25], which is the seminal work that introduced Siamese networks into
object tracking, uses an end-to-end network trained offline to learn a general
similarity metric without a fully connected layer. To obtain more accurate
predictions for the target bounding box, Region Proposal Network (RPN) [26] is
firstly utilized in SiamRPN [27]. The above-mentioned trackers only use
features from the output of one specific CNN layer, while features from
different CNN layers exhibit unequal spatial resolution and semantic extent.
To mitigate this problem, C-RPN [28] uses cascaded RPN blocks to combine high-
level semantics with low-level spatial information. Besides, to address the
issue that Siamese-based methods can not make full use of deeper and wider
architectures, SiamRPN++ [5] presents a spatially-aware sampling strategy
designed to exhibit adaptation between the deeper network and the basic
Siamese formulation. To this end, ResNet-50 [29] is employed as the backbone
of SiamRPN++, achieving improved performance compared with AlexNet backbones.

TIR tracking is also a hot topic in visual object tracking [30, 31]. Different
from Siamese-based RGB trackers, most TIR trackers model the target with only
handcrafted features rather than deep networks. Many TIR trackers, as in [32],
are simple extensions of RGB trackers. For example, SRDCFir extends SRDCF [33]
to TIR tracking by introducing motion features while using the same modelling
technique. EBT [34] uses a trained Support Vector Machine (SVM) classifier to
re-rank the edge boxes, thus obtaining instance-specific proposals. Similarly,
DSLT [35] combines motion features with edge and Histogram of Oriented
Gradients (HOG) [36] features, achieving real-time tracking in the Fourier
domain. In principle, TIR trackers seldomly use neural networks to extract
features, since there is insufficient data to train a network to learn a
satisfactory representation for the TIR modality. To this end, ECO-tir [37],
based on the RGB tracker ECO [38], generates a synthetic TIR dataset from
existing RGB datasets by a Generative Adversarial Network (GAN) [39]. The
modality-specific information learned from the generated TIR data contributes
to its ([6]) favourable results on VOT-RGBT2019 [4] dataset. To enhance the
robustness, MLSSNet [40] regards tracking as a matching problem and proposes a
multi-level similarity model, which incorporates global and local
similarities. Besides proposing a tracking algorithm, MLSSNet also provided
the first large scale TIR dataset, containing 430 video sequences. Similarly,
MMNet [31] uses a multi-task architecture to integrate the class-level and
fine-grained features. The authors also released a dataset consisting of 30
classes and over 1100 video sequences for TIR tracking.

Figure 2: Illustration of the proposed DFAT method. Here ’Cls’ and ’Reg’
represent classification and regression branches respectively. The left part
describes the architecture of the feature extraction network (’Net’). The
outputs from three convolutional layers are first projected into a common
space by neck modules and then input into the corresponding RPN blocks. The
features from different RPN blocks are modulated by adaptive weights within
’Net’. An overview of the proposed method (DFAT) is on the right. The outputs
of ’Net’ modules from both modalities are fused at the ’Decision-level Fusion’
module by Eq. (2). And the final result is obtained after the ’Postprocessor’.
We update the template every 10 frames. For convenience, the dimensionality of
feature representation is defined by the number of channels and the spatial
resolution. Thus, the shorthand notation (3,255) signifies that the size of
the input is 3x255x255.

###  2.2 Tracking with Multiple Modalities

Nowadays, tracking with multi-modal inputs draws increasing attention, such as
RGBT [41] and RGB-Depth tracking [42]. And the former, RGBT tracking, is
thoroughly discussed in this paper. Joint visual object tracking in RGB and
TIR modalities has been proposed to create more robust solutions in practical
scenarios . The core issue of multi-modal tracking is to merge the information
from both modalities efficiently. In this subsection, RGBT trackers will be
introduced according to the fusion strategies.

In general, fusion at pixel level is fulfilled in an add-up fashion. [43] uses
averaging technique and achieves satisfactory visualisation while [44] employs
Dual-Tree Complex Wavelet Transforms (DT-CWT) [45], obtaining higher
performance in its quantitative experiments. In recent years, seldom trackers
explore the fusion stage at pixel-level because of the existence of
misalignment between RGB and TIR image pixels. A simple concatenation operator
is utilized for pixel-level fusion in [6] to manifest the superiority of its
fusion strategy.

The most widely used fusion strategy is applied at the feature level. Adaptive
weights are obtained in [10] by calculating reconstruction residues in the
Bayesian filtering framework. Similarly, the weights in [46] are acquired by
the learned classifiers for each modality under the max-margin principle.
However, intuitively, feature-level fusion should be accomplished deeper in an
end-to-end framework. To this end, [47] proposes a lightweight fusion sub-
network, which consists of two convolutional layers, to accomplish feature
selection for accurate RGBT tracking. FANet [48], similarly, utilizes multi-
layer features, whose spatial resolutions are unified by max pooling. To
improve the representation capacity, an adaptive fusion sub-network is applied
to suppress channels with less reliability. DAPNet [8] simultaneously
integrates all the chosen layers and modalities by using an adaptive fusion
module recursively, achieving feature pruning by global average pooling to
reduce redundancy and noise. Further, DAFNet [49] integrates the adaptive
fusion sub-network from [48] and realizes quality-aware fusion in each layer
between the two modalities. To achieve fine-grained fusion, MANet [7] divides
features into three categories (modality-specific, modality-shared, and
instance-specific), exploiting a more exhaustive feature-level fusion approach
to get robust feature representation by multiple adaptors. On account of the
efficiency of Siamese structure in RGB tracking, SiamFT [50] primarily brings
in this structure to RGBT tracking and learns fusion weights from the medium-
term calculated response maps. Furthermore, based on dynamic Siamese network
[51], a multi-layer fusion strategy is adopted in DSiamMFT [52] to enrich the
semantic information of target in both modalities. To integrate the temporal
clues, CMPP [53] achieves pattern propagation spatially for inter-modal and
temporally for intra-modal by cross-modal attention. A multi-branch
architecture is employed in CAT [54] for target appearance modelling in
allusion to the modality-specific and modality-shared challenges, which
totally count to 5. Unlike DAPNet [8] and DAFNet [49] keep only the fused
features retained, TFNet [55] designs a trident architecture for better
excavating the modality-specific information. Constraint to the limited RGBT
data, the trackers aforementioned are all equipped with lightweight feature
extractors. To make better use of deep neural networks, one option is to
generate TIR data from existing RGB data [37]. It has been shown in [6] that,
with the dataset synthesized from GOT-10k, better performance is obtained.
However, we argue that synthesized data can not adequately represent the real
TIR data in terms of content and texture. Besides, there are some low-level
topics which regard the RGBT tracking as a downstream task to verify their
robustness and generalization like RFN-Nest [56]

Lastly, seldom trackers pay attention to the fusion at the decision level.
[57] and [14] use KL Divergence to estimate the reliability and achieve
adaptive fusion after the response maps of both modalities are available. [6]
sums up the scores from different modalities roughly and only obtains similar
results compared with pixel-level fusion. Once the response maps are
calculated from appearance trackers, [58] learns local and global weights from
the original image patch via a lightweight network. Then the decision-level
fusion is applied in a pixel-wise manner.

However, in this work, we explore fusion at all the three locations. For
pixel-level fusion, unlike [6] and [58], we adopt an advanced image fusion
method, MDLatLRR[59], to separate the detailed and salient information from
paired RGBT images. The salient information is averaged directly while the
detailed information is fused by nuclear-norm, and both of them are integrated
to get the final fused image. For fusion at feature level, a feature selection
method is designed based on channel attention mechanism while this in [8] and
[55] is carried out by WRS. Furthermore, spatial attention mechanism is also
considered for the cross-modal fusion. Decision-level fusion, experimentally,
is proved to be the best under the background of our method. Specifically, our
adaptive fusion weights are learned based on the hypothesis that there is
supposed to be a gap between the training RGB and testing TIR data, which is
illustrated in Fig. 6. Compared with JMMAC, VGGNet is used as the feature
extractor in JMMAC while ResNet-50 in our method, which is demonstrated to be
better and can extract features with more discrimination most of the time.
Besides, the weight matrix is learned from input image patches in JMMAC and
our weight scalar is acquired conveniently and rapidly from the classification
results. A more comprehensive comparison of multi-modal fusion is contained in
this work since fusion at the pixel level is not involved in JMMAC.

##  3 Proposed Method

In this section, we first briefly introduce our baseline method SiamRPN++ [5],
which is a single modality tracker with an end-to-end training framework.
Then, we will discuss our decision-level fusion strategy as well as fusion at
pixel and feature levels for a comprehensive comparison.

###  3.1 Baseline Tracker

The framework of SiamRPN++ architecture is displayed in the left of Fig. 2. In
the tracking stage, a typical Siamese tracker fixes the template features from
the first frame and keeps it unchanged for the rest of tracking frames. When a
new frame (instance) comes, SiamRPN++ first extracts multi-layer feature maps
(layer modules) and performs channel reduction (neck modules). Once the
feature embeddings are extracted, the results of classification (Cls) and
regression (Reg) branches from multiple RPN blocks (R_2, R_3, R_4) can be
calculated. In the baseline tracker, the multi-modal results are directly
averaged for fusion. After that, the general ’Postprocessor’ is followed to
acquire the tracking result from the fused classification and regression
results. Specifically, for the classification maps, a softmax operator is
employed to convert the original foreground/background response scores into
probabilities with values from the interval $[0,1]$, intuitively measuring
similarities. As for the regression branch, the result returned is transferred
to the bounding box since the offsets between anchors and ground truth are
learned from the given labels. Consequently, the index of the highest value
among the normalized classification scores is selected and its corresponding
bounding box in the regression branch is considered as the final tracking
result.

As for online tracking, SiamRPN++ uses multi-layer features to improve the
reliability of tracking. The computation process is defined in the following
formulation:

$\left\\{\begin{aligned}
\textbf{\emph{C}}=&\sum\limits_{i}\alpha_{i}\textrm{R}^{\textrm{C}}_{i}(f_{i}(\textbf{\emph{X}}),f_{i}(\textbf{\emph{Z}})),\\\
\textbf{\emph{B}}=&\sum\limits_{i}\beta_{i}\textrm{R}^{\textrm{B}}_{i}(f_{i}(\textbf{\emph{X}}),f_{i}(\textbf{\emph{Z}}))\end{aligned}\right..$
(1)

where X and Z are the image patches from an instance and the exemplar. $f_{i}$
represents the output of the $i$th neck module after the $i$th backbone layer.
$\textrm{R}_{i}^{\textrm{C}}$ and $\textrm{R}_{i}^{\textrm{B}}$ denote the
classification and bounding box regression branches respectively in the $i$th
RPN block. $\alpha_{i}$ and $\beta_{i}$ are the weights associated with the
corresponding branches for the $i$th layer, $\textrm{R}_{i}^{\textrm{C}}$ and
$\textrm{R}_{i}^{\textrm{B}}$, which are also learned offline. As shown in
Fig. 2, SiamRPN++ uses the features from layer 2-4 and fuses them by utilizing
multiple RPN blocks. In addition, depth-wise cross-correlation is adopted to
solve the parameter distribution imbalance caused by the increased dimensions.

As for offline training, SiamRPN++ employs a sampling strategy with
translations to mitigate the constraint that Siamese trackers are unable to
take advantage of deeper networks like ResNet-50 [29]. Specifically, the basic
formulation of the Siamese framework imposes a strict translation invariance,
and unfortunately, the padding operation is incompatible with this constraint.
Moreover, padding is an essential mechanism to match the input size
specification for deep networks. Based on an in-depth analysis, SiamRPN++
devises a simple sampling method that preserves the same spatial resolution
for layer 2, layer 3, and layer4, and works effectively. As a result, deep
networks, such as ResNet-50 [29] and VGGNet [60], achieve improved performance
compared to the original AlexNet [61], as shown in [62].

###  3.2 Fusion Mechanism

Figure 3: Introduction of pixel-level fusion achieved by MDLatLRR. The paired
RGB and TIR images are first fed into the MDLatLRR module with the chosen
vector manually determinated and the fused output is then sent to the tracking
network.

We deliver a brief introduction to our techniques of fusion at pixel, feature
and decision levels respectively. A novel decision-level fusion strategy,
since fusion at decision level performs considerable better results than
fusion at pixel and feature levels, is then described in this subsection and
our pipeline is given in Fig. 2.

Pixel-level fusion: For pixel-level fusion, we do not concatenate the image
matrices directly as [6] do. Instead, we choose MDLatLRR [59], which has been
proved to be efficient, to fuse the RGB and TIR images. MDLatLRR uses multi-
level LatLRR-based [63] decomposition to detach the detailed and salient
information from both RGB and TIR images. The low-frequency parts are fused by
weighted averaging while adaptive weights are learned for the fusion of detail
parts based on nuclear-norm. Fig. 3 shows the minutiae of pixel-level fusion.
The chosen vector is set in advance to acquire the fused image at a particular
level and only one element is supposed to be activated. Particularly, the
result from level2 is selected as input in Fig. 3.

Feature-level fusion: Fusion at feature level is carried out using attention
mechanism. Generally, feature selection is executed through channel mechanism
within each modality. After that, spatial and channel attention, combined with
normalization, is further employed for cross-modal fusion. In particular,
considering the noise and the level of redundancy in deep features, which are
discussed in detail in [16], we apply feature selection, which involves two
parts, i.e., channel saliency measurement and channel selection, to achieve
adaptive fusion. For assessing the channels, we first employ adaptive pooling
to indicate the significance of the individual channels. In other words, the
discriminatory information conveyed by each channel is represented by an
attention value. For channel selection, as shown in Fig. 4, we first sort the
channels by the learned representative scalar in the descending order of its
values. We retain the top 80% of channels based on the significance score and
set the others to zero. Subsequently, spatial or channel attention is
conducted by adaptive pooling and results in representative vectors, from
which the final fusion scalars are computed by mean operator. The operations
outlined above are applied after the neck module, as the projection to a low-
dimensional space can decrease noise and suppress redundancy.

Figure 4: Illustration of the process of feature selection for the fusion of
the two modalities at the feature level, which is embedded in after the neck
module. Here the channels highlighted with dark green represent those with
less discrimination and are further suppressed.

Decision-level fusion: To address the issue raised by the lack of RGBT data,
especially for the VOT-RGBT challenge, we investigate the merits of fusing at
the decision level. Ordinarily, RGB and TIR images are simultaneously fed into
the tracking network, with two pairs of outputs received from the RPN blocks.
The averaged results of classification and regression branches are then fed
into the ’Postprocessor’. Specifically, the scores of the classification
branch are normalized to probabilities by a softmax operator. Finally, the
bounding box is determined based on the maximum value of the normalized
scores. The experiments suggest that fusion before normalization works better
as shown in TABLE 3. For this reason, in this paper, we focus on fusion before
normalization. Although fusion at this stage yields considerable improvements,
we find that the scores produced by RGB images are always greater than the TIR
image scores as shown in Fig. 6(a) and 6(c), which means that weighting the
scores directly may produce inherently biased results.

Fig. 5 shows the details of our decision-level fusion method. Since we select
the bounding box according to the maximum of the normalized response scores
(classification branch), we first use ReLU (Rectified Linear Unit) activation
[64] to suppress the locations with a response below zero, which is less
likely to be the selected index paired with the highest similarity. Besides,
we amplify the scores of RGB and TIR images to the same magnitude by
multiplying them by the average value, which is calculated by GAP (Global
Average Pooling) [65], to alleviate the bias. The final result produced by the
classification branch is weighted by a factor consisting of two parts, i.e.,
shrink and scale. Specifically, to maintain the data consistency, we shrink
the zoomed-in data to a magnitude similar to the original scores by dividing
by the sum of average values. To further adjust the scores after shrinkage, we
introduce a hyper-parameter called the scaling factor, which balances out the
relative contributions of the classification and regression branches. More
details of the above operators are presented in the experimental section. As
for the regression branch, we obtain the fused result in a similar way.

The decision-level fusion process is formulated as follows:

$\begin{split}\textbf{\emph{B}}_{F}&=(\lambda_{11}*\textbf{\emph{B}}_{\textrm{RGB}}+\lambda_{21}*\textbf{\emph{B}}_{\textrm{TIR}})/(\lambda_{11}+\lambda_{21})\\\
\textbf{\emph{C}}_{F}&=s*(\lambda_{12}*\textbf{\emph{C}}_{\textrm{RGB}}+\lambda_{22}*\textbf{\emph{C}}_{\textrm{TIR}})/(\lambda_{12}+\lambda_{22})\\\
\lambda_{12}&=M_{\textbf{\emph{C}}_{\textrm{TIR}}>0}(\textbf{\emph{C}}_{\textrm{TIR}})\\\
\lambda_{22}&=M_{\textbf{\emph{C}}_{\textrm{RGB}}>0}(\textbf{\emph{C}}_{\textrm{RGB}})\end{split}$
(2)

where $\textbf{\emph{C}}_{F}$ and $\textbf{\emph{B}}_{F}$ are respectively the
fused results of the classification and regression branches.
$\textbf{\emph{C}}_{\textrm{RGB}}$ and $\textbf{\emph{B}}_{\textrm{RGB}}$ are
the outputs of the two branches from the RGB images. Similarly,
$C_{\textrm{TIR}}$ and $R_{\textrm{TIR}}$ are the outputs from the TIR images.
$M$ denotes the ’mean’ operation. $\lambda_{12}$ and $\lambda_{22}$ are the
learned weights, while $\lambda_{11}$ and $\lambda_{21}$ are fixed throughout
the tracking process. $s$ is the scaling factor. The merits of our DFAT are
confirmed by self-comparison and comparison with state-of-the-art trackers on
the VOT-RGBT2019 dataset presented in the experimental section.

Figure 5: Details of our decision-level fusion module. ’Net’ and
’Postprocessor’ have already been introduced in Fig. 2 and Section III as
parts of the pipeline. The results from the regression branches are averaged
while the results from the classification branches are adaptively weighted.
Here ’Modulation’ represents two numerical operations, i.e., shrink and scale,
which are detailed in the description of our decision-level fusion method. Eq.
(2) is the mathematical description of our decision-level fusion.

###  3.3 Template Update

As confirmed in [66], updating the template is beneficial to tracking
performance, especially in long-term scenarios. Traditionally, the target of
each video sequence is annotated at the first frame and the appearance model
is also learned from the initial frame. In principle, the target is often
subject to appearance variation throughout the video sequence, which means a
tracker with a constant template would encounter a model degradation.
Consequently, with an attempt to make the template exploit temporal clues, in
our approach, it is updated by linear interpolation after the object is
located in the current frame. Specifically, for the embeddings of the three
neck modules, we use a pre-defined learning rate to combine them with their
corresponding historical template embeddings. To prevent the template from
over-fitting, we update the template every 10 frames.

##  4 Experiments

In this section, we present the implementation details. The performance of the
fusion strategies developed at the three levels is measured on the VOT-
RGBT2019 dataset. In addition, we also verify the merits of the proposed DFAT
on the published VOT-RGBT2020 and GTOT dataset.

###  4.1 Implementation Details

Experimental platform: Our method DFAT is implemented with Pytorch 0.4.1 on a
platform with Intel Core i9-9980XE CPU and NVIDIA GeForce RTX 2080Ti GPU. The
speed of DFAT reaches 20 Frame Per Second (FPS).

Parameters setup: In our experiments, the hyperparameters $\lambda_{11}$ and
$\lambda_{21}$ are set to 0.5 for fusing the results from the regression
branches. To guarantee the scale continuity in the successive frames, we set
the learning rate for updating the width and height of the bounding box to
0.32. The learning rate for the linear interpolation update is chosen to be
0.1. The single-channel TIR images are triplicated to emulate RGB channels and
thus our network can process TIR inputs without any modification.

Figure 6: Visualization of the original classification scores and scores after
debaising.

###  4.2 Experimental Setup

Network: Our backbone network (ResNet-50) [29] is pretrained on ImageNet [67].
The RPN blocks and neck modules are trained by SGD optimizer on COCO [19] and
YouTube-VOS [20] with the batch size set to 96. A weight decay of 0.0001 and
momentum of 0.9 are employed. The learning rate is warmed up from 0.001 to
0.005 in the first 3 epochs and then it decays to 0.0025 in the remaining 17
epochs. It is worth noting that the target mask in these two datasets is
enabled, which contributes to the high accuracy of DFAT.

Datasets: Our experiments are evaluated on the VOT-RGBT2019 [4], VOT-RGBT2020
[21] and GTOT datasets.

On VOT-RGBT2019 and VOT-RGBT2020, which are consisted of 60 video pairs, we
use three evaluation measures named accuracy (A), robustness (R), and Expected
Average Overlap (EAO) to estimate the results. Accuracy is measured in terms
of the overlap of the estimated bounding boxes with the ground truth, while
robustness is inversely proportional to the failure rate. EAO is the key
criterion for evaluating a tracker in the VOT protocol. Note that a re-start
mechanism is employed when the overlap between the calculated bounding box and
ground truth falls to zero. As the original protocol is prone to favour the
trackers using a larger bounding box, a different evaluation protocol is
adopted in the VOT2020 challenge
333https://data.votchallenge.net/vot2020/vot-2020-protocol.pdf. The reset
mechanism is replaced by initialization points referred to as anchors.
Trackers run forward if the start anchor is before the middle of the video and
vice versa. All the anchors are set manually with an average gap of 50 frames.

GTOT includes 100 video sequences and 15700 frames in total. Success and
precision rate are the two evaluation metrics on this dataset. Success rate is
the major measurement which counts the percentage of successfully predicted
frames, whose IoU between their corresponding given bounding boxes are above a
certain threshold, 0.6 in the benchmark. Likewise, a prediction is considered
precise if the distance between its centre point and the groundtruth is below
a threshold, 5 in the benchmark. After the number of precise frames are
obtained, the precision rate can be computed by dividing 15700.

###  4.3 Results of Self-Comparison

In this section, we validate our method by self-comparison. Three fusion
mechanisms are discussed in detail. The results are displayed in TABLE 1, 2
and 3, respectively.

Table 1: The results of pixel-level fusion (Fig. 3) on VOT-RGBT2019 dataset
with MDLatLRR employed as the fusion strategy. (The best result is highlighted
in bold)

Level | Input Images | Results
---|---|---
| A ($\uparrow$) |  | R ($\downarrow$) |  | EAO ($\uparrow$) |
– | TIR + TIR |  | 0.6248 |  | 0.4531 |  | 0.2826 |
– | RGB + RGB |  | 0.6228 |  | 0.4481 |  | 0.3189 |
– | RGB + TIR |  | 0.6482 |  | 0.3784 |  | 0.3433 |
2 | Fused + Fused |  | 0.6180 |  | 0.4332 |  | 0.3120 |
3 | Fused + Fused |  | 0.6199 |  | 0.4132 |  | 0.3216 |
2 | Fused + TIR |  | 0.6138 |  | 0.4182 |  | 0.3286 |
3 | Fused + TIR |  | 0.6262 |  | 0.3933 |  | 0.3481 |

Pixel-level fusion: For pixel-level fusion, as described in Fig. 3, we choose
the fused images from the second and third decomposition level of MDLatLRR.
The experimental results are reported in TABLE 1. Here ’fused’ means the
output image of MDLatLRR. As we can see, the results of images from level 3
are better than those of level 2. For these two levels, better performance is
obtained by adding TIR images as the second input. The results of the baseline
achieved with only TIR or RGB images reach 0.2826 and 0.3189 on the EAO
evaluation metric, respectively. Thanks to the complementary information
conveyed by the RGB and TIR modalities, the results of the pixel-level fusion
by MDLatLRR show an improvement of 3.9% and 0.27%. The outputs from deeper
levels of MDLatLRR are expected to capture the salient information from each
modality, leading to better performance. Since fused images are visually close
to the RGB modality, we also pair them with the original TIR image as the
input of the tracking network and an improvement of 2.65% on EAO is obtained.
However, as pixel-level fusion is not suitable for the end-to-end framework,
the most corresponding results are lower than the result of using the original
RGB and TIR images as inputs (Baseline), which is also shown in Fig. 7.

Table 2: The results of fusion at feature-level on the VOT-RGBT2019 dataset.
’Selection’ is the flag of whether using feature selection or not.
Respectively, ’C’ and ’S’ mean channel attention and spatial attention.
’Vector’ denotes the pooling method to obtain the attention vector. ’Scalar’
represents the manner to acquire the fusion scalar from its corresponding
attention vector. For example, without feature selection, the attention vector
for the first line is calculated by average pooling spatially and then the
fusion scalar is obtained by applying a maximum operator to this vector. (The
best result is highlighted in bold)

Selection | Type | Vector | Scalar | Results
---|---|---|---|---
|  A ($\uparrow$) |  |  R ($\downarrow$) |  |  EAO ($\uparrow$) |
0 | S | mean | max |  | 0.6570 |  | 0.3883 |  | 0.3470 |
0 | C | mean | max |  | 0.6398 |  | 0.3136 |  | 0.3593 |
0 | S | max | mean |  | 0.6433 |  | 0.3037 |  | 0.3615 |
0 | C | max | mean |  | 0.6350 |  | 0.3136 |  | 0.3618 |
1 | S | max | mean |  | 0.6433 |  | 0.3037 |  | 0.3615 |
1 | C | max | mean |  | 0.6416 |  | 0.2987 |  | 0.3788 |

Feature-level fusion:  It is expected that fusion at the feature level will
deliver better performance than pixel-level fusion. This is confirmed in our
study. Generally, we compute the fusion weights from the attention vectors,
which are previously obtained by the attention mechanism. Specifically, for
fusion with feature selection, we first employ global max pooling to omit the
channels with lower discriminative power and keep the sequential order
immutably (’Selection’). Considering the feature noise and redundancy, we set
the selection rate to 20% so that 80% of the channels are retained. After the
type of attention to be used is selected (’Type’), we obtain the corresponding
attention vector with average or max-pooling (’Vector’). And the
representative scalar is supposed to be calculated from its corresponding
attention vector (’Scalar’), which is used for feature-level fusion after
normalization. It can be seen that by executing feature-level fusion after
removing less discriminative channels, we get the best result in robustness
(0.2987) as well as EAO (0.3788) at this fusion level.

Table 3: A comparison of the decision-level fusion before and after
normalization on the VOT-RGBT2019 dataset. Here ’DF’ means the decision-level
fusion strategy proposed in this paper while ’KLD’ represents KL Divergence.
’After’ and ’Before’ separately denote fusion after and before normalization.
(The best result is highlighted in bold)

Operating bias | Location | Fusion Strategy | Results
---|---|---|---
| Cls |  | Reg |  |  A ($\uparrow$) |  |  R ($\downarrow$) |  |  EAO ($\uparrow$) |
F | After |  | KLD |  | KLD |  | 0.6523 |  | 0.3385 |  | 0.3680 |
F | After |  | KLD |  | 0.5 |  | 0.6667 |  | 0.3435 |  | 0.3656 |
F | After |  | 0.5 |  | 0.5 |  | 0.6648 |  | 0.2788 |  | 0.3863 |
F | Before |  | 0.5 |  | 0.5 |  | 0.6619 |  | 0.2788 |  | 0.3877 |
T | Before |  | DF |  | DF |  | 0.5978 |  | 0.2639 |  | 0.3663 |
T | Before |  | DF |  | 0.5 |  | 0.6652 |  | 0.2539 |  | 0.3986 |

Decision-level fusion: For the decision-level fusion, we consider two
variants, depending on the positioning of the fusion step, i.e., before
normalization (’Before’) and after normalization (’After’). Since the
classification scores are restrainted between 0 and 1 after normalization, the
data bias between RGB and TIR modalities vanishes. We compare the fusion
results at these two variants to confirm our hypothesis. As mentioned before,
another tracker uses decision-level fusion to obtain a reliable response map
with KL Divergence [14]. This criterion is well suited for a probabilistic
model matching, exemplified by the case of fusion after normalization. On VOT-
RGBT2019, the given ground truth comes from the TIR modality. As the paired
RGB and TIR images are not accurately aligned, the misalignment increases the
difficulty of handling the problem of fusion in the regression branch. So we
just consider exploiting both KL Divergence and our method on the
classification branch. As can be seen, the variant with the KL Divergence used
after normalization does not achieve a promising result, which is even worse
than adding the two sources of information together with the weights set to
0.5. This is caused by the labels used in network training not being of a
Gaussian-shape matrix, whereas [14] is a CF-based tracker, with the target
response satisfying this requirement. Thus, KL Divergence performs
unsatisfactorily in our framework.

On the other hand, the fusion before normalization is more effective which
also indicates the necessity of debiasing. The result with weights manually
set to 0.5 is 0.3877 on EAO, achieving a gain of 0.14% compared to the highest
result after normalization. By implanting our method on the classification
branch, the result of our tracker, which ranks the best in the self-
comparison, reaches 0.3986. We obtain a performance improvement of 1.09% on
EAO and 2.49% in failure rate.

Illustrating the details of the decision-level fusion: In order to gain a
clear understanding of our decision-level fusion, we first clarify the three
numerical operations, i.e.,  _adaptive weighting_ , _normalization_ and
_scaling_. As illustrated in the second part of Section III, adaptive
weighting method is applied to reduce the bias, which is visualized in Fig.
6(a) and 6(c), as well as make better use of the complementary information
from different modalities. We use softmax as the normalization method as most
Siamese-based trackers do. Actually, the fused result does not represent the
final measure of ’similarity’, which is of great importance in the Siamese
framework [25]. From the fused result, we do not know whether the candidate
anchor is considered positive or not. So the normalization operation is
introduced to convert the raw results to probabilities, which is also
consistent with the subsequent operations in the framework. The scaling factor
is instrumental in achieving better results in the classification branch,
which is discussed in the fifth part of Section IV.

The experiments demonstrate that fusion at the decision level is the best
mechanism in an end-to-end framework trained without TIR images. Ignoring the
method proposed in this paper, a simple fusion at the decision level achieves
0.3877, which is merely worse than DFAT. Besides, fusion before normalization
is more effective than fusion after normalization. Moreover, from Fig. 6(b)
and 6(d), it can be seen that the bias is significantly reduced after equipped
with our debiasing module.

###  4.4 Comparing to State-of-the-art Trackers

In this section, we verify the merits of our method by comparing it to the
state-of-the-art trackers. Here the comparative trackers include JMMAC [58],
SiamDW_T [68], CISRDCF, FSRPN, GESBTT, MPAT, mfDimp [6], MANet [7], FANet [48]
and TFNet [55]. All the trackers can be found in [4] except the last two
trackers. TABLE 4 provides the results of trackers on VOT-RGBT2019 dataset.
Our DFAT ranks marginally the best in accuracy and the second in terms of EAO.
RPN is utilized for precise estimation of the bounding box, which proves that
FSRPN and DFAT are both among the top three trackers in accuracy. As can be
seen, a compact mask is a more exact requirement than the bounding box. So
training the network based on the masks also contributes to the high accuracy
of DFAT. DFAT only falls behind the first place of the published VOT-RGBT2019
dataset on EAO evaluation and exceeds the runner-up of about 0.61%. As the
robustness displayed in [4] is different from the value obtained from the vot-
toolkit, for completeness, the exhibited scores for all contrastive trackers
are received by one minus the published scores in [4] and [55].

Figure 7: Expected overlap as a function of sequence length and the average
expected overlap (EAO) of all variants. Figure 8: The success and precision
rates of all variants on GTOT. Figure 9: A subjective comparison of the
respective trackers on the VOT-RGBT2020 dataset. From top to bottom, we show
the results on three video sequences, i.e.,  _car41_ , _caraftertree_ and
_carnotfar_.

###  4.5 Self-Analysis

Ablation study: In this subsection, we perform an analysis of the impact of
each component of our DFAT tracker, i.e., template update and adaptive fusion
at the decision level. The visualization is presented in Fig. 7 and Fig. 8, in
which ’W/O bias’ is the tracker without bias correction and ’W/O update’
represents without the update. ’Baseline’ means the extension of our baseline
tracker SiamRPN++ to RGBT tracking while RGB and TIR mean only RGB and TIR
images available. Respectively, ’RGB’ and ’TIR’ means only the visible and
thermal infrared data is used as input.

According to the ablation study on the VOT-RGBT2019 dataset, the trackers with
single modality information perform undesirable, with the result on RGB being
0.3189 in terms of EAO. Since our network is trained on RGB datasets, using it
for tracking on the TIR input achieves an even worse result, the EAO of which
is 0.2826. By exploiting the complementary information between the two
modalities, the baseline RGBT tracker improves from 0.3189 to 0.3433. In the
baseline tracker, a fixed template is employed while the appearance of the
target will change continuously throughout the whole video sequence.
Introducing a simple yet effective update strategy, which is realized by
linear interpolation, we generate a more robust template and the results gain
a 4.44% enhancement compared to our baseline. By dealing with the
preconception, significantly caused by the method of training, the ’W/O
update’ tracker outperforms our baseline by 2.23%. An intuitive payoff is
displayed in Fig. 6. As can be seen, the scores of the RGB modality are not
consistently higher than that of the TIR modality. The specific value will be
discussed in the next subsection. Finally, by taking advantage of these two
components jointly, the best result is achieved, with gains of 1.13% and 3.30%
compared to ’W/O bias’ and ’W/O update’ respectively. The experimental results
demonstrate the merits of the template update and reducing data bias.

Similarly, on GTOT, the success rate of RGB only tracker is also higher than
the TIR only variant. With multi-modal clues integrated, the baseline method
obtains a success rate of 0.687. Furthermore, the update and debiasing
operators bring an increment of 2.5% and 1% respectively. Jointly, our DFAT
reaches 0.723 on the measurement of success rate and gains 3.6% in total
compared with baseline method.

Figure 10: Analysis of the scaling factor on the VOT-RGBT2019 dataset.

Analysis of the scaling factor: Generally, as mentioned above, the results
from the classification branch is transferred to $[0,1]$ after normalization
(softmax in this paper). Under this situation, an incorrect index would be
received from the classification branch if there exists more than one large
value (1) in the normalized scores, caused by rounding errors. This would
result in a rough estimate of the bounding box. To address this problem, we
introduce a scaling factor to modulate the scores before normalization and
amplify the disparity of the normalized scores. This is the reason why the
scores handled by our method are lower than the original scores, as shown in
Fig. 6. To illuminate further the impact of the scaling factor, a sensitivity
analysis is performed on the VOT-RGBT2019 dataset. Fig. 10 reports the
experimental results. With the fusion weights at the regression branch set to
0.5, we find that the worst result (0.3884) is obtained with a scaling factor
of 0.44, while the tracker with a scaling factor of 0.47 ranks the best
(0.4080). From the results shown, we can conclude that our DFAT is quite
insensitive to the value of the scaling factor.

Table 4: Tracking results on VOT-RGBT2019 dataset. (The best result is
highlighted in bold)

Tracker | FANet[48] | TFNet[55] | GESBTT[4] | CISRDCF[4] | MPAT[4] | MANet[7] | FSRPN[4] | mfDimp[6] | SiamDW_T[68] | JMMAC[58] | DFAT
---|---|---|---|---|---|---|---|---|---|---|---
Published | TIV2021 | TCSVT2021 | - | - | - | ICCVW2019 | - | ICCVW2019 | CVPR2019 | TIP2021 | -
A | 0.4724 | 0.4617 | 0.6163 | 0.5215 | 0.5723 | 0.5823 | 0.6362 | 0.6019 | 0.6158 | 0.6649 | 0.6652
R | 0.4922 | 0.4064 | 0.3650 | 0.3096 | 0.2758 | 0.2990 | 0.2931 | 0.1964 | 0.2161 | 0.1789 | 0.2539
EAO | 0.2465 | 0.2878 | 0.2896 | 0.2923 | 0.3180 | 0.3463 | 0.3553 | 0.3879 | 0.3925 | 0.4826 | 0.3986

Table 5: Tracking results on VOT-RGBT2020 dataset. (The best result is
highlighted in bold)

Variation | Baseline | W/O update | W/O bias | DFAT
---|---|---|---|---
A | 0.6751 | 0.6741 | 0.6754 | 0.6796
R | 0.7423 | 0.7415 | 0.7728 | 0.7869
EAO | 0.3902 | 0.3937 | 0.4073 | 0.4178

###  4.6 Evaluation on the VOT-RGBT2020 Dataset

To validate the merits of our method, we also examine DFAT on the VOT-RGBT2020
dataset [21] with vot-toolkit 0.2.0. Each proposed component is also
reconfirmed to be effective. TABLE 5 provides an intuitive comparison for each
component. Interestingly, the variant ’W/O bias’ submitted to the VOT-RGBT2020
challenge won the championship. Correcting the data bias, we gain an
improvement of 1.05% which raises the best result to 0.4178 and sets a new
state-of-the-art in RGBT tracking. We also conducted qualitative experiments
to assess the impact of each component and visualize the results in Fig. 9. We
show the results on TIR images for clarity. As is evident, our DFAT achieves
the top performance. The ’W/O bias’ variant is only marginally worse than DFAT
and ranks second, which demonstrates the necessity of the template update. As
for the experiments in TABLE 5, the scaling factor is still set to 0.5. A more
suitable scaling factor can be found to obtain better performance. For
example, setting the scaling factor to 0.47, which is found to be the best on
the VOT-RGBT2019 dataset, gains a slight increase of 0.04%.

##  5 Conclusion

In this paper, we first explore different fusion strategies at three levels,
i.e., pixel-level, feature-level and decision-level, and the experimental
results show that fusion at the decision level performs the best with only
visible data employed for training. Therefore, we proposed a novel fusion
strategy at the decision level for accurate multi-modal (RGB and TIR) object
tracking (DFAT) in a video sequence. The tracker is built using the SiamRPN++
architecture trained with RGB data only, but deployed for embedding both, the
RGB and TIR modalities. As our network is trained using only RGB datasets, TIR
images are copied three times to emulate the network input format of the RGB
channels. The tracking is facilitated by a template updating mechanism based
on a linear target interpolation every 10 frames.

Based on a deep analysis of the outputs of the RPN blocks of SiamRPN++, we
design a novel fusion strategy to merge the complementary information conveyed
by the RGB and TIR modalities. It involves a dynamic de-biasing of the RGB and
TIR contributions to the classification branch of the network. The strategy
also performs a classification branch scaling to balance the relative
contributions of the classification and regression results. A variant of the
proposed decision-level fusion was the winning entry of the VOT-RGBT(2020)
Challenge. The experimental results on the VOT-RGBT2019, VOT-RGBT2020 and GTOT
datasets confirm the effectiveness and robustness of our method. Moreover, the
outcome of the VOT-RGBT2020 challenge also demonstrates that our DFAT
represents a new state-of-the-art in RGBT tracking.

## Acknowledgement

This work was supported in part by the National Key Research and Development
Program of China under Grant 2017YFC1601800, in part by the National Natural
Science Foundation of China (Grant NO.62020106012, U1836218, 61672265), the
111 Project of Ministry of Education of China (Grant No.B12018), and the UK
EPSRC (EP/N007743/1, MURI/EPSRC/DSTL, EP/R018456/1).

## References

  * [1] T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, and N. Díaz-Rodríguez, “Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges,” _Information Fusion_ , vol. 58, pp. 52–68, 2020.
  * [2] M. D. la Torre, E. Granger, P. V. Radtke, R. Sabourin, and D. O. Gorodnichy, “Partially-supervised learning from facial trajectories for face recognition in video surveillance,” _Information Fusion_ , vol. 24, pp. 31–53, 2015\.
  * [3] X. Song, X. Shao, Q. Zhang, R. Shibasaki, H. Zhao, and H. Zha, “A novel dynamic model for multiple pedestrians tracking in extremely crowded scenarios,” _Information Fusion_ , vol. 14, no. 3, pp. 301–310, 2013.
  * [4] M. Kristan, J. Matas, A. Leonardis _et al._ , “The seventh visual object tracking vot2019 challenge results,” in _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , 2019, pp. 0–0.
  * [5] B. Li, W. Wu, Q. Wang, F. Zhang, J. Xing, and J. Yan, “Siamrpn++: Evolution of siamese visual tracking with very deep networks,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 4277–4286.
  * [6] L. Zhang, M. Danelljan, A. Gonzalez-Garcia, J. van de Weijer, and F. Shahbaz Khan, “Multi-modal fusion for end-to-end rgb-t tracking,” in _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , 2019, pp. 2252–2261.
  * [7] C. L. Li, A. Lu, A. H. Zheng, Z. Tu, and J. Tang, “Multi-adapter rgbt tracking,” in _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , 2019, pp. 2262–2270.
  * [8] Y. Zhu, C. Li, B. Luo, J. Tang, and X. Wang, “Dense feature aggregation and pruning for rgbt tracking,” in _Proceedings of the 27th ACM International Conference on Multimedia_ , 2019, pp. 465–472.
  * [9] C. Li, X. Liang, Y. Lu, N. Zhao, and J. Tang, “Rgb-t object tracking: Benchmark and baseline,” _Pattern Recognition_ , vol. 96, p. 106977, 2019\.
  * [10] C. Li, H. Cheng, S. Hu, X. Liu, J. Tang, and L. Lin, “Learning collaborative sparse representation for grayscale-thermal tracking,” _IEEE Transactions on Image Processing_ , vol. 25, no. 12, pp. 5743–5756, 2016.
  * [11] A. Torabi, G. Masse, and G. Bilodeau, “An iterative integrated framework for thermal-visible image registration, sensor fusion, and people tracking for video surveillance applications,” _Computer Vision and Image Understanding_ , vol. 116, no. 2, pp. 210–221, 2012.
  * [12] J. W. Davis and V. Sharma, “Background-subtraction using contour-based fusion of thermal and visible imagery,” _Computer Vision and Image Understanding_ , vol. 106, no. 2, pp. 162–182, 2007.
  * [13] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speed tracking with kernelized correlation filters,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 37, no. 3, pp. 583–596, 2015\.
  * [14] M. Feng, K. Song, Y. Wang, J. Liu, and Y. Yan, “Learning discriminative update adaptive spatial-temporal regularized correlation filter for rgb-t tracking,” _Journal of Visual Communication and Image Representation_ , vol. 72, p. 102881, 2020.
  * [15] T. Xu, Z. Feng, X. Wu, and J. Kittler, “Learning adaptive discriminative correlation filters via temporal consistency preserving spatial feature selection for robust visual object tracking,” _IEEE Transactions on Image Processing_ , vol. 28, no. 11, pp. 5596–5609, 2019.
  * [16] T. Xu, Z. Feng, X. Wu, and J. Kittler, “Joint group feature selection and discriminative filter learning for robust visual object tracking,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 7949–7959.
  * [17] P. S. Efraimidis and P. G. Spirakis, “Weighted random sampling with a reservoir,” _Information Processing Letters_ , vol. 97, no. 5, pp. 181–185, 2006.
  * [18] Y. Xie, J. Zhang, Y. Xia, M. Fulham, and Y. Zhang, “Fusing texture, shape and deep model-learned information at decision level for automated classification of lung nodules on chest ct,” _Information Fusion_ , vol. 42, pp. 102–110, 2018.
  * [19] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European Conference on Computer Vision (ECCV)_ , 2014, pp. 740–755.
  * [20] N. Xu, L. Yang, Y. Fan, D. Yue, Y. Liang, J. Yang, and T. S. Huang, “Youtube-vos: Sequence-to-sequence video object segmentation,” in _European Conference on Computer Vision_ , 2018, pp. 603–619.
  * [21] M. Kristan, J. Matas, A. Leonardis _et al._ , “The eighth visual object tracking vot2020 challenge results,” in _European Conference on Computer Vision Workshop (ECCVW)_ , 2020, pp. 0–0.
  * [22] T. Xu, Z.-H. Feng, X.-J. Wu, and J. Kittler, “Learning low-rank and sparse discriminative correlation filters for coarse-to-fine visual object tracking,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 10, pp. 3727–3739, 2020.
  * [23] X. Zhu, X.-J. Wu, T. Xu, Z.-H. Feng, and J. Kittler, “Complementary discriminative correlation filters based on collaborative representation for visual object tracking,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 31, no. 2, pp. 557–568, 2021.
  * [24] Q. Wang, L. Zhang, L. Bertinetto, W. Hu, and P. H. S. Torr, “Fast online object tracking and segmentation: A unifying approach,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 1328–1338.
  * [25] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. S. Torr, “Fully-convolutional siamese networks for object tracking,” in _European Conference on Computer Vision workshops (ECCVW)_ , 2016, pp. 850–865.
  * [26] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 6, pp. 1137–1149, 2017.
  * [27] B. Li, J. Yan, W. Wu, Z. Zhu, and X. Hu, “High performance visual tracking with siamese region proposal network,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 8971–8980.
  * [28] H. Fan and H. Ling, “Siamese cascaded region proposal networks for real-time visual tracking,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 7944–7953.
  * [29] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _2016 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778.
  * [30] X. Li, Q. Liu, N. Fan, Z. He, and H. Wang, “Hierarchical spatial-aware siamese network for thermal infrared object tracking,” _Knowledge-Based Systems_ , vol. 166, pp. 71–81, 2019.
  * [31] Q. Liu, X. Li, Z. He, N. Fan, D. Yuan, W. Liu, and Y. Liang, “Multi-task driven feature models for thermal infrared tracking,” in _Thirty-Fourth AAAI Conference on Artificial Intelligence_ , 2020, pp. 11 604–11 611.
  * [32] M. Felsberg, A. Berg, G. Hager _et al._ , “The thermal infrared visual object tracking vot-tir2015 challenge results,” in _2015 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , 2015, pp. 639–651.
  * [33] M. Danelljan, G. Häger, F. S. Khan, and M. Felsberg, “Learning spatially regularized correlation filters for visual tracking,” in _2015 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2015, pp. 4310–4318.
  * [34] G. Zhu, F. Porikli, and H. Li, “Beyond local search: Tracking objects everywhere with instance-specific proposals,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition CVPR_ , 2016, pp. 943–951.
  * [35] X. Yu, Q. Yu, Y. Shang, and H. Zhang, “Dense structural learning for infrared object tracking at 200+ frames per second,” _Pattern Recognition Letters_ , vol. 100, pp. 152–159, 2017.
  * [36] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _2005 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2005, pp. 886–893.
  * [37] L. Zhang, A. Gonzalez-Garcia, J. van de Weijer, M. Danelljan, and F. S. Khan, “Synthetic data generation for end-to-end thermal infrared tracking,” _IEEE Transactions on Image Processing_ , vol. 28, no. 4, pp. 1837–1850, 2019.
  * [38] M. Danelljan, G. Bhat, F. S. Khan, and M. Felsberg, “Eco: Efficient convolution operators for tracking,” in _2017 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 6931–6939.
  * [39] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” _Advances in Neural Information Processing Systems_ , vol. 3, pp. 2672–2680, 2014.
  * [40] Q. Liu, X. Li, Z. He, N. Fan, D. Yuan, and H. Wang, “Learning deep multi-level similarity for thermal infrared object tracking,” _IEEE Transactions on Multimedia_ , pp. 1–13, 2020.
  * [41] X. Zhang, P. Ye, H. Leung, K. Gong, and G. Xiao, “Object fusion tracking based on visible and infrared images: A comprehensive review,” _Information Fusion_ , vol. 63, pp. 166–187, 2020.
  * [42] M. xin Jiang, C. Deng, J. song Shan, Y. yuan Wang, Y. jie Jia, and X. Sun, “Hierarchical multi-modal fusion fcn with attention model for rgb-d tracking,” _Information Fusion_ , vol. 50, pp. 1–8, 2019.
  * [43] L. Mihaylova, A. Loza, S. G. Nikolov, J. J. Lewis, and D. R. Bull, “The influence of multi-sensor video fusion on object tracking using a particle filter,” in _Informatik 2006 - Informatik für Menschen, Band 1, Beiträge der 36. Jahrestagung der Gesellschaft für Informatik e.V. (GI), 2.-6. Oktober 2006 in Dresden_ , 2006.
  * [44] N. Cvejic, S. G. Nikolov, H. D. Knowles, A. Loza, A. Achim, D. R. Bull, and C. N. Canagarajah, “The effect of pixel-level fusion on object tracking in multi-sensor surveillance video,” in _2007 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2007, pp. 1–7.
  * [45] T. Dixon, J. Li, J. Noyes, T. Troscianko, S. Nikolov, J. Lewis, E. Canga, D. Bull, and C. Canagarajah, “Scanpath analysis of fused multi-sensor images with luminance change: A pilot study,” in _2006 9th International Conference on Information Fusion_ , 2006, pp. 1–8.
  * [46] X. Lan, M. Ye, S. Zhang, and P. C. Yuen, “Robust collaborative discriminative learning for rgb-infrared tracking,” in _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_. AAAI Press, 2018, pp. 7008–7015.
  * [47] C. Li, X. Wu, N. Zhao, X. Cao, and J. Tang, “Fusing two-stream convolutional neural networks for rgb-t object tracking,” _Neurocomputing_ , vol. 281, pp. 78–85, 2018.
  * [48] Y. Zhu, C. Li, J. Tang, and B. Luo, “Quality-aware feature aggregation network for robust rgbt tracking,” _IEEE Transactions on Intelligent Vehicles_ , vol. 6, no. 1, pp. 121–130, 2021.
  * [49] Y. Gao, C. Li, Y. Zhu, J. Tang, T. He, and F. Wang, “Deep adaptive fusion network for high performance rgbt tracking,” in _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , 2019, pp. 91–99.
  * [50] X. Zhang, P. Ye, S. Peng, J. Liu, K. Gong, and G. Xiao, “Siamft: An rgb-infrared fusion tracking method via fully convolutional siamese networks,” _IEEE Access_ , vol. 7, pp. 122 122–122 133, 2019.
  * [51] Q. Guo, F. Wei, C. Zhou, H. Rui, and W. Song, “Learning dynamic siamese network for visual object tracking,” in _2017 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2017.
  * [52] X. Zhang, P. Ye, S. Peng, J. Liu, and G. Xiao, “Dsiammft: An rgb-t fusion tracking method via dynamic siamese networks using multi-layer feature fusion,” _Signal Processing: Image Communication_ , vol. 84, p. 115756, 2020\.
  * [53] C. Wang, C. Xu, Z. Cui, L. Zhou, and J. Yang, “Cross-modal pattern-propagation for rgb-t tracking,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [54] C. Li, L. Liu, A. Lu, Q. Ji, and J. Tang, “Challenge-aware rgbt tracking,” in _European Conference on Computer Vision_. Springer International Publishing, 2020, pp. 222–237.
  * [55] Y. Zhu, C. Li, J. Tang, B. Luo, and L. Wang, “Rgbt tracking by trident fusion network,” _IEEE Transactions on Circuits and Systems for Video Technology_ , pp. 1–1, 2021.
  * [56] H. Li, X.-J. Wu, and J. Kittler, “Rfn-nest: An end-to-end residual fusion network for infrared and visible images,” _Information Fusion_ , vol. 73, pp. 72–86, 2021.
  * [57] C. Luo, B. Sun, K. Yang, T. Lu, and W. C. Yeh, “Thermal infrared and visible sequences fusion tracking based on a hybrid tracking framework with adaptive weighting scheme,” _Infrared Physics & Technology_, vol. 99, pp. 265–276, 2019.
  * [58] P. Zhang, J. Zhao, C. Bo, D. Wang, H. Lu, and X. Yang, “Jointly modeling motion and appearance cues for robust rgb-t tracking,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 3335–3347, 2021.
  * [59] H. Li, X. Wu, and J. Kittler, “Mdlatlrr: A novel decomposition method for infrared and visible image fusion,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 4733–4746, 2020.
  * [60] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations_ , 2015, pp. 1–14.
  * [61] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _neural information processing systems_ , 2012, pp. 1097–1105.
  * [62] Z. Zhang and H. Peng, “Deeper and wider siamese networks for real-time visual tracking,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 4586–4595.
  * [63] G. Liu and S. Yan, “Latent low-rank representation for subspace segmentation and feature extraction,” in _2011 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2011, pp. 1615–1622.
  * [64] X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier neural networks,” in _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011_ , 2011, pp. 315–323.
  * [65] M. Lin, Q. Chen, and S. Yan, “Network in network,” in _IEEE International Conference on Learning Representations_ , 2014, pp. 1–10.
  * [66] L. Zhang, A. Gonzalez-Garcia, J. v. d. Weijer, M. Danelljan, and F. S. Khan, “Learning the model update for siamese trackers,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 4010–4019.
  * [67] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International Journal of Computer Vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [68] Z. Zhang and H. Peng, “Deeper and wider siamese networks for real-time visual tracking,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 4586–4595.