# PuTR: A Pure Transformer for Decoupled and Online Multi-Object Tracking

Chongwei Liu1 Haojie Li2∗ Zhihui Wang1 Rui Xu1
1Dalian University of Technology
2Shandong University of Science and Technology
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Recent advances in Multi-Object Tracking (MOT) have achieved remarkable
success in short-term association within the decoupled tracking-by-detection
online paradigm. However, long-term tracking still remains a challenging task.
Although graph-based approaches can address this issue by modeling
trajectories as a graph in the decoupled manner, their non-online nature poses
obstacles for real-time applications. In this paper, we demonstrate that the
trajectory graph is a directed acyclic graph, which can be represented by an
object sequence arranged by frame and a binary adjacency matrix. It is a
coincidence that the binary matrix matches the attention mask in the
Transformer, and the object sequence serves exactly as a natural input
sequence. Intuitively, we propose that a pure Transformer can naturally unify
short- and long-term associations in a decoupled and online manner. Our
experiments show that a classic Transformer architecture naturally suits the
association problem and achieves a strong baseline compared to existing
foundational methods across four datasets: DanceTrack, SportsMOT, MOT17, and
MOT20, as well as superior generalizability in domain shift. Moreover, the
decoupled property also enables efficient training and inference. This work
pioneers a promising Transformer-based approach for the MOT task, and provides
code to facilitate further research. https://github.com/chongweiliu/PuTR

##  1 Introduction

Multi-Object Tracking (MOT) is a fundamental yet challenging task, playing a
pivotal role in modern perception systems for demanding applications like
autonomous driving [17], video surveillance [16], and behavior analysis [21].
Over the decades, various methodologies, such as motion estimation [4; 57] or
graph-based approaches [22; 5], have been exploited to tackle this problem,
garnering significant attention.

Among them, the decoupled tracking-by-detection online paradigm has emerged as
the dominant approach in recent years due to its simplicity and effectiveness.
In this setting, detection results are obtained from a pre-trained detector
(e.g., YOLOX [20]) and treated as immutable. Researchers then focus primarily
on the association problem. Methods like SORT [4] or ByteTrack [57] exemplify
this approach, exploiting motion cues via the Kalman filter [23] and
associating objects based on the Hungarian algorithm [24], demonstrating
impressive generalization ability in short-term tracking.

To simultaneously handle short- and long-term tracking challenges, methods
such as MPNTrack [5] or SUSHI [10] model trajectories as a graph in the
decoupled manner and leverage Graph Neural Networks (GNNs) [40] for
association. These methods excel at maintaining tracklet completeness after
long-range occlusions, with the only drawback being their non-online nature.

Consequently, an intriguing question arises: is it possible to unify short-
and long-term tracking in a decoupled and online manner? We attempt to address
this question in Fig.1. By observing the trajectory graph in Fig.1a, we notice
that each trajectory propagates independently along the temporal dimension,
indicating that each object (node) has at most one in-degree and one out-
degree. This implies that the trajectory graph is a strict directed acyclic
graph, and its corresponding adjacency matrix is binary. Therefore, we can
arrange objects by frame as a sequence and form a binary adjacency matrix to
describe the trajectory graph, as shown in Fig. 1b. The binary matrix can be
divided into white blocks representing possible connections (edges) and gray
blocks representing impossible connections, where established trajectories
will always lie within the white blocks. Due to the causality of trajectories
in the frame sequence and forbidden connections among objects within the same
frame, the white blocks exhibit an approximate lower triangular structure,
formally aligned with the attention mask in the Transformer [46], while the
object sequence serves exactly as a natural input sequence, as shown in Fig.
1c. This coincidence suggests an intuitive idea: directly applying a pure
Transformer for association in a decoupled and online fashion.

Figure 1: Illustration of our chain of thought. Trajectories are inherently a
directed acyclic graph in the temporal order (a). We can thus transform it
equivalently into a binary adjacency matrix (b), exactly aligning with the
Transformer’s attention mask. Consequently, arranging objects by frame forms a
natural input sequence for the Transformer (c), enabling it to model the
association problem.

While this idea seems straightforward, has there been any research already
introducing the Transformer structure into MOT? The answer is yes, and no.
Yes, because the pioneering works TrackFormer [31] and MOTR [54] have
introduced the Transformer-based object detector DETR [9] into MOT in an end-
to-end manner. They modify Deformable-DETR [61] to treat a tracklet as a
hidden state in a query token and propagate it across frames to perform
detection and association simultaneously. This inherent propagation gene is
derived from earlier works like Tracktor [2] or CenterTrack [59], where CNN
detectors (Faster RCNN [36] or CenterNet [60]) were employed in that era. We
therefore also say the answer is no, because track propagation is more akin to
a Recurrent Neural Network (RNN) [39] instead of a Transformer - the true
power of the self-attention mechanism to model long-range dependencies is not
exploited in the association. Our straightforward idea aims to take a small
step forward in exploring the potential of a pure Transformer for the
association problem of MOT, paving the way for another promising new research
direction in this field.

In this paper, we exploit a Pure Transformer, named PuTR, to validate our
idea. PuTR leverages the basic Transformer architecture to address the
association challenge uniformly, as Fig. 1c shows. We modify the attention
mask and incorporate temporal and spatial positional encodings to adapt it to
the unique characteristics of the object sequence compared to a normal text
sequence. Specifically, unlike a sentence, the object sequence is ordered in
the outer frame temporal dimension but unordered in the inner image spatial
dimension. Therefore, we modify the attention mask to ensure permutation
invariance within the same frame, and also introduce temporal and spatial
positional encodings to encode the frame order and the object coordinate
information into the object sequence. We conduct extensive experiments on four
MOT datasets: DanceTrack [43], SportsMOT [11], MOT17 [32], and MOT20 [12], to
demonstrate the effectiveness of our approach. The results show that PuTR
achieves a strong baseline compared to existing foundational methods,
indicating the potential of this new direction. Notably, PuTR exhibits
superior generalizability in domain shift, i.e., without fine-tuning, the
maximum cross-dataset gap is only $2.7\%$ in IDF1 [37] and $2.3\%$ in HOTA
[29]. Furthermore, PuTR is efficient both in training and inference. It can be
trained from scratch on a single GPU in less than $1$ hour without additional
data or complex training processes, and achieves up to $90$ FPS in inference
when given detection results. We provide our code to facilitate other
researchers in following and promoting this promising direction into a new
phase.

##  2 Related Work

Over the years, researchers have exploited various methodologies to address
the challenging association problem in MOT, leading to diverse approaches in
this field. Here, we provide a concise overview of the most representative
methodologies in MOT. Additionally, the MOT task can also be modeled using
techniques such as unsupervised learning [26], trajectory forecasting [13],
and deep reinforcement learning [47]; however, these are not covered here due
to space limitations.

Heuristic Methods. These have been widely studied and remain the most
prevalent paradigm in this field. In the decoupled approach, these methods
first detect objects using pre-trained object detectors [36; 20; 60] in each
frame, and then associate them across frames online. Motion and spatial
proximity cues are central to these methods. The foundational method SORT [4]
utilizes the Kalman filter to estimate object motion and the Hungarian
algorithm for association, and introduces a run-time manager to dynamically
schedule the tracklet states (tracked, lost, or new). Subsequent methods
primarily focus on increasing the robustness of frame-by-frame object
association by incorporating appearance cues via an independent ReID model
[49; 15; 51], introducing a more sophisticated motion model [57; 8; 25], or
leveraging camera motion compensation to handle camera movements [1; 30].
These methods achieve impressive performance in short-term tracking scenarios,
while more ingenious hand-crafted rules are urgently needed to maintain
tracklet completeness in long-term tracking scenarios.

Graph-based Methods. Another perspective is to model the MOT as a graph
optimization problem. Also under the decoupled approach, these methods
represent nodes as detected objects and edges as trajectory hypotheses. In
contrast to the frame-by-frame trackers mentioned above, graph-based methods
search for global solutions to the data association problem over several
frames or even the entire video sequence. This allows them to be more robust
in semi-online or offline settings. Initially, numerous optimization
frameworks were studied, including minimizing fixed costs [22; 34; 55], multi-
cuts [44], and minimum cliques [38]. Recently, GNN-based methods [5; 10] have
shown promise as an alternative to traditional optimization methods. MPNTrack
[5] introduces a message passing network to dissect information and associate
detections through edge classification. Building upon it, SUSHI [10] uses
multiple hierarchical levels recursively to further optimize the graph space
overhead and extend the ability to process long videos. Under the global
perspective, they can handle short- and long-term tracking challenges
uniformly, while the results are obtained offline, limiting wider real-time
applications. Based on the heuristic and graph-based methods, PuTR is
constantly exploring a new direction to unify association in a decoupled and
online manner.

Detector-based Methods. Since a ready-made object detector is a prerequisite
for the tracking-by-detection paradigm, it is natural to consider integrating
the detector and tracker into a unified framework. As a transitional form
before unification, JDE [48] and FairMOT [58] propose modifying CNN-based
detectors [35; 60] to generate object bounding boxes and appearance features
simultaneously for heuristic association, termed Joint-Detection-and-Embedding
(JDE). For the unified framework in the CNN era, Tracktor [2] and CenterTrack
[59] modify CNN-based detectors [36; 60] to use the previous tracklet results
to predict the current tracklet results, thereby merging them into
trajectories. In the Transformer era, the DETR family [9; 61; 27] has become
mainstream. TrackFormer [31] and MOTR [54] therefore first extended the object
queries in Deformable-DETR to the MOT task, using queries to represent tracked
objects and propagating them into the next frame. Subsequent methods primarily
focus on introducing a memory bank to store historical object embeddings [7;
18] or resolving conflicts between newborn and tracked object queries [50;
53]. Although the base detector has evolved from CNN to Transformer, the core
mechanism for handling tracklets remains the same, i.e., updating the hidden
state of the tracklet by the current frame and then propagating it to the next
in an RNN-style manner. Therefore, we suggest that PuTR explores another way
of employing the Transformer to MOT, i.e., association - not detection.

More recently, MOTIP [19] has also realized the potential of the Transformer
for association, formulating it as an end-to-end in-context ID prediction
problem. It constructs a learnable ID dictionary and attaches a Transformer
decoder behind Deformable-DETR to predict tracking IDs of current detections,
using historical trajectories as $K$ and $V$, and current detections as $Q$ in
the attention mechanism. It seems we have discovered a similar idea, albeit
PuTR operates in a decoupled manner akin to a pure decoder-only Transformer,
allowing efficient single-GPU training without pre-training or complex
stabilization techniques. In contrast, MOTIP’s end-to-end nature necessitates
a more complicated training process involving substantial data and multiple
GPUs (usually $8$) to mitigate overfitting. Additionally, MOTIP’s ID
dictionary requires empirical tuning of its size for different datasets and
risks exhaustion when tracked objects exceed its capacity, whereas PuTR
proposes to associate objects via relative affinity, circumventing such issues
and achieving superior generalizability under domain shift.

Notably, we do not aim to replace the end-to-end methods. Each methodology
does not exist in opposition; instead, there are connections among them. For
instance, the run-time manager is also utilized in DETR-based methods, and our
idea draws inspiration from the trajectory graph used in GNN-based approaches.
On the spectrum of methodologies, MOTIP may be viewed as an intermediate form
bridging MOTR and PuTR. Each method explores MOT in their respective
ecological niches, introducing new diversity to the field. Indeed, we
emphasize that with sufficient video-level training data (e.g., DanceTrack),
these end-to-end frameworks outperform decoupled methods. Our decoupled
approach serves as a baseline when video data is scarce (e.g., MOT17). We
believe that different paradigms can coexist and complement each other,
providing a more comprehensive solution to the MOT community.

##  3 Method

###  3.1 Preliminary

Problem Formulation. We first introduce the fundamental concepts and notations
used in PuTR. Our approach follows the tracking-by-detection paradigm. Given a
video, we assume that object detections are computed for every frame, and our
task is to perform data association by linking object detections into online
trajectories.

We denote the set of objects in the $t$-th frame as
$\mathcal{O}_{t}=\\{o_{\textit{tid}}^{t}\\}$, where $o_{\textit{tid}}^{t}$
represents the detected object in the $t$-th frame with the tracking identity
tid. Each $o_{\textit{tid}}^{t}$ is represented by its bounding box
coordinates, confidence score, and category label. The tracking identity tid
is assigned to each object belonging to the same trajectory, with
$\textit{tid}=-1$ indicating that the object does not belong to any
trajectory. Within an $\mathcal{O}_{t}$ set, each tid is unique, except for
$-1$. The goal of data association is to obtain the set of trajectories
$\mathcal{T}^{*}$ that group detections corresponding to the same identity.
Each trajectory $\mathcal{T}_{k}\in\mathcal{T}^{*}$ is given by its set of
detections $\mathcal{T}_{k}=\\{o_{k}^{t}\ |\ t_{s,k}\leq t\leq t_{e,k}\\}$,
where $t_{s,k}$ and $t_{e,k}$ are the start and end frames of the trajectory
$\mathcal{T}_{k}$, respectively.

Input Object Sequence. Similar to Transformer [46], the input of PuTR is also
a sequence, but consisting of object detections in recent $T+1$ consecutive
frames. We denote the input object sequence as
$\mathcal{S}=({\textsf{<bos>}},\mathcal{O}_{t-T},...,\mathcal{O}_{t-1},\mathcal{O}_{t})$.
Following language models [6; 45], the sequence $\mathcal{S}$ starts with a
special token <bos> to indicate the beginning of the sequence, which is an
all-zero token. The input sequence is then fed into PuTR after tokenization.

Tokenization. To feed the object detections into PuTR, we first convert each
object detection $o_{\textit{tid}}^{t}$ into a token representation. We denote
the token representation of object $o_{\textit{tid}}^{t}$ as
$x_{\textit{tid}}^{t}$ and the corresponding token set as
$\mathcal{X}_{t}=\\{x_{\textit{tid}}^{t}\\}$ in the $t$-th frame. It is
feasible to map objects into high-dimensional vectors by employing an
independent Re-ID model [49] or DETR object embeddings [54]. However, in this
work, we adopt a variant from ViT [14] to encode the objects, avoiding the
introduction of additional complexity. Specifically, for each object image
patch $p_{\textit{tid}}^{t}\in\mathbb{R}^{h\times w\times 3}$, generated by
cropping the corresponding object detection bounding box111$h$ and $w$ are the
height and width of the bounding box, respectively. from the $t$-th frame, we
divide it into a $64\times 64$ grid and sample the center point of each grid
via bilinear interpolation. We then flatten these points to obtain the one-
dimensional representation $r_{\textit{tid}}^{t}\in\mathbb{R}^{(64\times
64\times 3)}$, and pass it through a linear layer to obtain the tokenized
representation $x_{\textit{tid}}^{t}$ with the same dimension as the PuTR
dimension $d_{\textit{model}}=512$. The tokenized sequence thus becomes
$\mathcal{S}_{\textit{in}}=({\textsf{<bos>}},\mathcal{X}_{t-T},...,\mathcal{X}_{t-1},\mathcal{X}_{t})\in\mathbb{R}^{n_{\mathcal{S}}\times
d_{\textit{model}}}$, where $n_{\mathcal{S}}$ is the token number of the
sequence.

###  3.2 Architecture & Modifications

Architecture. As illustrated in Fig. 1c, PuTR essentially follows the standard
architecture of the Transformer [46], composed of a stack of $N=6$ identical
layers. Each layer has two sub-layers: a masked multi-head self-attention
mechanism with $8$ heads, and a fully connected feed-forward network. A
residual connection is employed around each sub-layer. To improve training
stability, we perform layer normalization before each sub-layer input
following [6; 45].

The $\mathcal{S}_{\textit{in}}$ is fed into PuTR after adding temporal
positional encodings, and spatial positional encodings are added before being
linearly projected into $K$ and $V$ in each attention mechanism. After the $N$
layers, the output object embeddings are linearly projected into high-
dimensional representations, denoted as
$\mathcal{S}_{\textit{out}}=({\textsf{<bos>}},\mathcal{Z}_{t-T},...,\mathcal{Z}_{t-1},\mathcal{Z}_{t})\in\mathbb{R}^{n_{\mathcal{S}}\times
d_{\textit{model}}}$, which is then used to calculate the relative affinity
matrix between trajectories and current detections to solve the association
problem.

Modifications. Unlike sentences, object sequences are ordered temporally
across frames but unordered spatially within each frame. To account for this,
we modify the attention mask and introduce temporal and spatial positional
encodings in PuTR.

Since objects in $\mathcal{O}_{t}$ are unordered and cannot be interlinked, we
modify the conventional causal mask (a lower triangular matrix) to ensure
permutation invariance within the same frame. Specifically, based on the
causal mask, we additionally set the positions to zero for tokens in the same
frame, i.e., $\textit{mask}_{ij}=0$ if $t_{i}==t_{j}\ \&\ i\neq j$, where
$t_{i}$ and $t_{j}$ are the frame indices of the $i$-th and $j$-th tokens,
respectively. We refer to this as the frame causal mask, which precisely
aligns with the distribution of the white blocks in Fig. 1b, and thus
consistent with the trajectory graph structure.

For the positional encodings, we adopt the basic absolute positional encoding
from the original Transformer [46] to encode the temporal order and the
spatial positional encoding from DETR [9] to encode spatial location.
Specifically, in $\mathcal{S}_{\textit{in}}$, tokens within $\mathcal{X}_{t}$
are added with the same temporal positional encoding before the first layer,
indicating their frame order within the current clip (e.g.,
$\mathcal{X}_{t-T}$ has position $0$, $\mathcal{X}_{t}$ has $T$). Following
DETR, the spatial positional encoding concatenates two absolute positional
encodings with length $d_{\textit{model}}/2$, representing the object’s center
coordinate in the image, and is added to the object embedding before linear
projection to $K$ and $V$ in each attention mechanism to emphasize the spatial
location information, as shown in Fig. 1c.

###  3.3 Training & Inference

Training. Unlike language model pre-training, where each output token is
required to predict the next word in a fixed vocabulary table via
classification, in MOT, the same individual object generates a unique
trajectory, making it impractical to establish a fixed individual object
vocabulary. While MOTIP [19] proposes using a shared tracking ID dictionary to
predict the ID labels of current detected objects, determining the appropriate
dictionary size requires empirical tuning for different datasets, and there is
a risk of ID exhaustion when the number of tracked objects exceeds the
dictionary size.

To ensure generalizability, PuTR circumvents the ID dictionary and instead
proposes using the relative affinity matrix to classify the current detected
objects, thus accomplishing the association task. Specifically, we calculate
the inner product between $\mathcal{Z}_{t-1}$ and $\mathcal{Z}_{t}$, i.e.,
$\mathcal{Z}_{t-1}\mathcal{Z}_{t}^{\top}$ $(t\geq 1)$, to obtain the relative
affinity matrix, and apply the softmax along rows to obtain the probability
distribution of the previous objects corresponding to the next objects.
Through the affinity matrix, we can determine the association via the
Hungarian algorithm, similar to heuristic methods [4; 57], without the need
for a fixed vocabulary table. The classification loss is simply calculated as
the cross-entropy loss between the predicted probability and the ground truth
label. Thus, without bells and whistles, the training loss of PuTR is
straightforwardly calculated as:

$\mathcal{L}=\frac{\sum\limits^{\textit{bs}}\sum\limits^{T}\textit{CE}(\textit{softmax}(\mathcal{Z}_{t-1}\mathcal{Z}_{t}^{\top}),Y_{t})}{\textit{bs}\times
T},$ (1)

where $\textit{CE}(\cdot)$ denotes the cross-entropy loss, $Y_{t}$ is the
ground truth label in the $t$-th frame, and bs is the batch size.
$\textit{CE}(\cdot)$ calculates the cross-entropy loss between the predicted
probability and the ground truth label, averaging over the number of objects
in $\mathcal{Z}_{t-1}$. For a tracking identity tid in $\mathcal{Z}_{t}$ but
not in $\mathcal{Z}_{t-1}$, we assign the nearest output object embedding with
the same tid before $\mathcal{Z}_{t-1}$ to calculate the loss, ensuring that
each object is supervised during training.

Inference. Different methodologies are not opposed to each other. Although
DETR-based methods [54; 56; 18; 19] are considered end-to-end, some of them
still require a run-time manager to schedule tracklet states and occasionally
resolve assignment conflicts between tracklets and detections via heuristic
rules or the Hungarian algorithm during inference. From this perspective, they
can also be regarded as “Heuristic Methods”, with the distinction that the
similarity matrix is generated by neural networks instead of hand-crafted
models. Consequently, it is of little surprise that PuTR, as a decoupled
method, also requires a run-time manager and the Hungarian algorithm to deal
with similar issues during inference.

In the first frame of a video sequence during inference, detected objects with
a confidence score greater than $\tau_{\textit{new}}$ are recorded as newborn
objects and assigned unique tracking identities. At each subsequent time step
$t$, we first filter the detection results with a confidence threshold
$\tau_{\textit{det}}$. These remaining detections are then concatenated with
the object sequence $\mathcal{S}$ from the recent $T$ frames and fed into PuTR
to generate the relative affinity matrix. We then employ the Hungarian
algorithm to obtain assignment results and a run-time manager to update the
tracklet states. The unmatched object with a detection confidence greater than
$\tau_{\textit{new}}$ will be considered a newborn target and given a new
identity. For more details, please refer to Appendix A.

##  4 Experiments

###  4.1 Settings

Datasets. We evaluate PuTR on four MOT datasets: the widely-used DanceTrack
[43], MOT17 [32], and MOT20 [12], as well as the recently proposed SportsMOT
[11]. DanceTrack and SportsMOT provide large-scale training data, while MOT17
and MOT20 have limited training sequences, comprising seven and four
sequences, respectively. Notably, DanceTrack and SportsMOT include official
validation sets, enabling exploratory experiments.

Metrics. We employ three widely-used metrics: HOTA [29], IDF1 [37], and MOTA
[3]. MOTA measures object coverage, while IDF1 focuses on identity
preservation. HOTA strikes a balance between these two aspects, as illustrated
in Figure 1 of [29]. In the decoupled setting, where all methods are evaluated
on the same detection results, MOTA values are nearly identical across
different methods. However, HOTA and especially IDF1 can better reflect the
performance of the association component. Therefore, we primarily focus on
IDF1 and HOTA in our experiments when evaluated on the same detections.

Implementation Details. PuTR is implemented using PyTorch, and both training
and inference are conducted on a single NVIDIA RTX 4090 GPU. During training,
consistent settings are maintained across all datasets for simplicity,
although further fine-tuning of hyperparameters could potentially improve
performance. PuTR is optimized using the AdamW optimizer and a cosine learning
rate schedule with a batch size of $4$ for $7$ epochs on the corresponding
training set. The length of training clips is progressively increased from $4$
to $8$, $16$, $32$, $64$, and $128$ at the $2$-nd, $3$-rd, $4$-th, $5$-th, and
$6$-th epoch, respectively, except on MOT20, where the clip length stays at
$64$ after the $6$-th epoch due to GPU memory limitations. The training time
for any dataset is less than $1$ hour. During inference, we adopt publicly
available YOLOX detections for a fair comparison, as previous works [57; 43;
11]. The number of consecutive frames $T$ is set to $30$ for DanceTrack,
SportsMOT, and MOT20, while for MOT17, it is $25$ empirically. Hyperparameters
are consistent within the same dataset and not specifically tuned for each
test sequence. Please refer to the Appendix B for more details.

Table 1: Comparison with existing foundational online methods. Higher scores
indicate better performance for the IDF1 (I$\uparrow$), HOTA (H$\uparrow$),
and MOTA (M$\uparrow$) metrics. Superscripts of other methods denote the
publisher, while subscripts specify the base detector employed. Blue
backgrounds highlight decoupled methods utilizing the same detection results,
with best scores in bold. The domain shift test assesses PuTR’s
generalizability across datasets, where the subscript indicates the training
set, and $\Delta_{\textit{max}}$ represents the maximum performance gap across
the four test sets.

| Dancetrack test | SportsMOT test | MOT17 test | MOT20 test
---|---|---|---|---
Method | I$\uparrow$ | H$\uparrow$ | M$\uparrow$ | I$\uparrow$ | H$\uparrow$ | M$\uparrow$ | I$\uparrow$ | H$\uparrow$ | M$\uparrow$ | I$\uparrow$ | H$\uparrow$ | M$\uparrow$
Detector-based: |  |  |  |  |  |  |  |  |  |  |  |
Tracktor++ [2]${}_{\textbf{Faster R-CNN}}^{\textit{ICCV'19}}$ | - | - | - | - | - | - | 52.3 | 42.1 | 53.5 | - | - | -
CenterTrack [59]${}_{\textbf{CenterNet}}^{\textit{ECCV'20}}$ | 35.7 | 41.8 | 86.8 | 60.0 | 62.7 | 90.8 | 64.7 | 52.2 | 67.8 | - | - | -
FairMOT [58]${}_{\textbf{CenterNet}}^{\textit{IJCV'21}}$ | 40.8 | 39.7 | 82.2 | 53.5 | 49.3 | 86.4 | 72.3 | 59.3 | 73.7 | 67.3 | 54.6 | 61.8
QDTrack [33]${}_{\textbf{Faster R-CNN}}^{\textit{CVPR'21}}$ | 44.8 | 45.7 | 83.0 | 62.3 | 60.4 | 90.1 | 66.3 | 53.9 | 68.7 | - | - | -
TrackFormer [31]${}_{\textbf{Deformable-DETR}}^{\textit{CVPR'22}}$ | - | - | - | - | - | - | 68.0 | 57.3 | 74.1 | 65.7 | 54.7 | 68.6
MOTR [54]${}_{\textbf{Deformable-DETR}}^{\textit{ECCV'22}}$ | 51.5 | 54.2 | 79.7 | - | - | - | 68.4 | 57.2 | 71.9 | - | - | -
MOTIP [19]${}_{\textbf{Deformable-DETR}}^{\textit{arXiv'24}}$ | 72.2 | 67.5 | 90.3 | 75.0 | 71.9 | 92.9 | 71.2 | 59.2 | 75.5 | - | - | -
Heuristic: |  |  |  |  |  |  |  |  |  |  |  |
SORT [4]${}_{\textbf{YOLOX}}^{\textit{ICIP'16}}$ | 50.8 | 47.9 | 91.8 | - | - | - | 78.2 | 63.0 | 80.1 | - | - | -
DeepSORT [49]${}_{\textbf{YOLOX}}^{\textit{ICIP'17}}$ | 47.9 | 45.6 | 87.8 | - | - | - | 74.5 | 61.2 | 78.0 | 69.6 | 57.1 | 71.8
ByteTrack [57]${}_{\textbf{YOLOX}}^{\textit{ECCV'22}}$ | 53.9 | 47.7 | 89.6 | 71.4 | 64.1 | 95.9 | 77.3 | 63.1 | 80.3 | 75.2 | 61.3 | 77.8
OC-SORT [8]${}_{\textbf{YOLOX}}^{\textit{CVPR'23}}$ | 54.6 | 55.1 | 92.0 | 74.0 | 73.7 | 96.5 | 77.5 | 63.2 | 78.0 | 75.9 | 62.1 | 75.5
PuTR (Ours) | 58.2 | 55.8 | 91.9 | 75.9 | 74.9 | 97.0 | 74.9 | 61.7 | 78.5 | 74.1 | 61.1 | 74.8
Domain shift test: |  |  |  |  |  |  |  |  |  |  |  |
PuTR${}_{\textit{MOT17}}$ | 57.3 | 55.9 | 91.7 | 73.2 | 73.1 | 96.8 | 74.9 | 61.7 | 78.5 | 74.0 | 60.9 | 75.2
PuTR${}_{\textit{MOT20}}$ | 57.4 | 55.8 | 91.6 | 73.7 | 73.3 | 96.9 | 73.9 | 61.1 | 78.5 | 74.1 | 61.1 | 74.8
PuTR${}_{\textit{DanceTrack}}$ | 58.2 | 55.8 | 91.9 | 73.3 | 72.6 | 96.9 | 74.3 | 61.3 | 78.7 | 73.8 | 60.9 | 75.1
PuTR${}_{\textit{SportsMOT}}$ | 58.8 | 56.5 | 91.8 | 75.9 | 74.9 | 97.0 | 74.1 | 61.3 | 78.6 | 73.9 | 60.9 | 75.0
$\Delta_{\textit{max}}$ | 1.5 | 0.7 | 0.3 | 2.7 | 2.3 | 0.2 | 0.8 | 0.6 | 0.2 | 0.3 | 0.2 | 0.4

###  4.2 Comparison with Existing Foundational Online Methods

Tab. 1 compares PuTR with previous foundational online methods on four
datasets. As a decoupled method, we primarily focus on the association
performance (IDF1 and HOTA) under the same detection results and the
generalizability across datasets (domain shift test).

Comparison with Heuristic Methods. Regarding association performance across
different datasets, the pioneering MOT17 and MOT20 benchmarks have established
a solid foundation for the MOT task. However, their limited test sequences
($7$ and $4$, respectively) often lead to sequence-specific hyperparameter
tuning by many methods [57; 8] to achieve better performance, potentially
distorting the assessment values. In contrast, more recent large-scale
datasets like DanceTrack and SportsMOT provide massive test sequences ($35$
and $150$, respectively), making such sequence-specific tuning impractical.
Therefore, drawing conclusions about a method’s general association
performance should rely on a synthesis of assessments across datasets,
especially large-scale ones.

In PuTR, we maintain consistent hyperparameters within each test set without
sequence-specific tuning. Under this setting, PuTR achieves the highest IDF1
and HOTA scores on the large-scale DanceTrack and SportsMOT222For SportsMOT,
we use the same detection results as ByteTrack and OC-SORT following the
Train+Val setup, referring to Table 3 in [11]. In the Train setup, PuTR’s
I$\uparrow$, H$\uparrow$, M$\uparrow$ are 74.2%, 73.0%, 95.1%, respectively.
datasets and exhibits competitive performance on the smaller MOT17 and MOT20
benchmarks. Unlike other methods [57; 8] that perform better on small datasets
than large ones, we can reasonably conclude that PuTR, as a new approach,
demonstrates solid association ability across diverse scenarios compared to
foundational heuristic methods.

Furthermore, by leveraging a pure Transformer to handle various associations
uniformly, PuTR is liberated from hand-crafted rules and does not require
explicit consideration of challenging factors such as complex non-linear
motion [25], low frame rates [28], or camera motion [52], which are essential
for heuristic methods. This is particularly evident on DanceTrack, where
dancers exhibit complex interactions and non-linear motions, and PuTR
maintains a better IDF1 score than the second-best OC-SORT ($58.2\%$ vs.
$54.6\%$) due to its long-range dependency modeling capability (see
visualizations in Appendix C). These results indicate that PuTR explores a
promising new direction for the association problem in the decoupled and
online manner.

Comparison with Detector-based Methods. PuTR outperforms the representative
JDE [58; 33], CNN-based [2; 59], and DETR-based [54; 18] methods on all
benchmarks, except the latest DETR-based approach MOTIP [19], warranting a
detailed analysis. MOTIP leverages its methodology’s advantages but requires
additional training data and substantial computational resources (typically
$8$ GPUs for days) to achieve satisfactory performance, hence excelling on
DanceTrack and SportsMOT. For MOT17, the CrowdHuman [42] dataset, containing
massive human images, is incorporated following prior work [54; 18] to
mitigate overfitting, improving performance while increasing training
complexity. For MOT20, where the average pedestrian count per frame exceeds
DETR’s detection capacity, only the early TrackFormer [31] is evaluated by
modifying Deformable-DETR, while later incremental works focus on datasets
with more moderate object query numbers. In contrast, PuTR performs
streamlined training from scratch on a single GPU in under $1$
hour333Specifically, training time on DanceTrack, SportsMOT, MOT17, and MOT20
is $55$, $35$, $20$, and $20$ minutes. without any additional data or
pretraining, regardless of dataset scale or object count.

On large-scale datasets, PuTR surpasses MOTIP on SportsMOT, while leaving room
for improvement on DanceTrack. This is attributed to the complexity of dancer
interactions, where YOLOX generates more False Positives (FPs) on DanceTrack
than SportsMOT. Through end-to-end learning, MOTIP can detect interacting
dancers more precisely under tracking guidance, thus addressing DanceTrack’s
most challenging aspect. This indicates that no methodology can perfectly
handle all MOT challenges, necessitating a balanced approach. Consequently,
PuTR excels on SportsMOT with simple training and precise detection, while
even MOTIP with additional image data cannot outperform PuTR on MOT17.
Furthermore, PuTR can be easily extended to the dense MOT20 dataset.

Through the aforementioned comparisons, PuTR demonstrates a strong baseline
for this emerging direction. Although a performance gap exists between PuTR
and the latest incremental methods, the analysis indicates the promising
potential of leveraging Transformers for MOT in an alternative way.

Inference Efficiency. In addition to training efficiency, PuTR demonstrates
remarkable inference speed. Unlike MOTIP’s 22 FPS444We test the official
model’s speed on the same machine as PuTR and exceed the original $16$ FPS
report. on DanceTrack, SportsMOT, and MOT17, PuTR can process video sequences
at a much higher frame rate due to its relatively simple architecture.
Specifically, PuTR’s average FPS on the sparse DanceTrack, SportsMOT, and
MOT17 datasets are $89$, $92$, and $73$, respectively. Even on the dense MOT20
dataset, it can still achieve an impressive $27$ FPS. While the massive number
of tracked objects per frame in MOT20 can lengthen the input object sequence
and consequently slow down the speed, it still meets real-time requirements.
Moreover, the decoupled approach offers the advantage of independent execution
for detection and association, enabling parallel processing pipelines for
further efficiency enhancement.

Generalizability. Robust association ability across various scenarios is
crucial for MOT methods. Heuristic methods can tune hyperparameters to adapt
to most common cases, while other learning-based methods rarely report
generalizability across datasets despite the urge to address this issue. Only
the work [41] addresses the domain adaptation problem in MOT, reporting that
QDTrack [33] exhibits a striking degradation, decreasing by $49\%$ in IDF1 and
$35\%$ in HOTA when evaluated on DanceTrack with weights trained on MOT17. We
also evaluate the official MOTIP weights trained on MOT17 on the DanceTrack
test set, only yielding $44.9\%$ IDF1 and $43.0\%$ HOTA.

In contrast, from the domain shift test, we find that PuTR has excellent
generalizability across domains. Without any other operation, just applying
the weights trained on one dataset to another, the maximum cross-dataset gap
is only $2.7\%$ in IDF1 and $2.3\%$ in HOTA, which has never been reported in
previous learning-based works. This property of PuTR can be attributed to the
fact that it takes advantage of different methodologies, which is appropriate
in terms of methodological cross-pollination and integration. It is the
decoupled manner that PuTR can obtain precise detection results like heuristic
methods, and the self-attention mechanism of the Transformer that can
discriminate individuals in the feature space, even in scenes that have never
been trained. Furthermore, the relative affinity matrix also plays a crucial
role, as it circumvents concerns regarding the dictionary capacity or the
exhaustion issue in MOTIP. This property makes the new direction we proposed
ever more fascinating, and we believe it will attract more attention in the
future and promote a new development in the MOT field. We also report the
combined training performance in Appendix D.

###  4.3 Ablation Studies

We conduct ablation studies on the DanceTrack and SportsMOT datasets due to
their large-scale training data and availability of official validation sets.
More studies are presented in Appendix E.

Attention Mask. The attention mask plays a crucial role in enabling the
Transformer to capture dependencies between tokens. We evaluate the impact of
our proposed frame causal mask compared to the conventional causal mask, and
the results are presented in Tab. 3. The models are trained and inferred using
the same mask type, and during inference, the shuffle operation randomly
shuffles the object order within one frame. Without shuffling, we find that
the IDF1 scores of the frame causal mask are $0.2\%$ and $0.5\%$ higher on the
DanceTrack and SportsMOT validation sets, respectively, compared to the
conventional causal mask (the first and third rows). Although the increase is
not significant, the two types of masks are fundamentally different.

Our frame causal mask ensures permutation invariance of objects within the
same frame, thereby maintaining consistency between the trajectory graph and
the Transformer, i.e., all objects in the same frame are concurrent and cannot
associate with each other. Consequently, even when the object order in the
frame is randomly shuffled (the second row), the performance remains
consistent. In contrast, the causal mask does not possess this property, and
the model’s performance is coupled with the training object order, leading to
unpredictable results when shuffling (the fourth row).

Table 2: Ablations on the attention mask.

PuTR | Dancetrack val | SportsMOT val
---|---|---
Mask Type | Shuffle | I$\uparrow$ | H$\uparrow$ | M$\uparrow$ | I$\uparrow$ | H$\uparrow$ | M$\uparrow$
frame causal |  | 55.7 | 54.5 | 88.3 | 78.6 | 76.4 | 95.7
frame causal | $\checkmark$ | 55.7 | 54.5 | 88.3 | 78.6 | 76.4 | 95.7
causal |  | 55.5 | 54.4 | 88.3 | 78.1 | 76.1 | 95.6
causal | $\checkmark$ | 54.9 | 53.6 | 88.2 | 77.9 | 76.0 | 95.6

Table 3:  Ablations on the positional encoding.

Positional Encoding | Dancetrack val | SportsMOT val
---|---|---
Temporal | Spatial | I$\uparrow$ | H$\uparrow$ | M$\uparrow$ | I$\uparrow$ | H$\uparrow$ | M$\uparrow$
|  | 55.0 | 53.9 | 88.1 | 78.0 | 76.3 | 95.6
$\checkmark$ |  | 55.2 | 54.3 | 88.1 | 78.2 | 76.4 | 95.6
$\checkmark$ | $\checkmark$ | 55.7 | 54.5 | 88.2 | 78.6 | 76.4 | 95.7

Positional Encoding. Our model incorporates spatial and temporal positional
encodings to adjust to the unique characteristics of the object sequence. We
experiment with several combinations of these encodings, and the results are
presented in Tab. 3. Without these two positional encodings (the first row),
the model still achieves reasonably high IDF1 scores of $55.0\%$ and $78.0\%$
on the respective validation sets, losing only $0.7\%$ and $0.6\%$ compared to
the complete model (the third row). Introducing the temporal positional
encoding leads to a $0.2\%$ increase in IDF1 on both sets (the second row).
Since the frame causal mask ensures the correct frame order, temporal
positional encodings are not strictly necessary for identifying the frame
order, thus leading to a minor improvement. Incorporating the spatial
positional encodings into PuTR additionally resulted in an increase of $0.5\%$
and $0.4\%$ in IDF1, respectively. This suggests that even without spatial
positional encodings, PuTR could recognize the same individual object based
solely on appearance cues, and the spatial positional encodings further
enhance this ability by encoding the spatial location in the image.

Overall, even without these modifications, the model still achieves a strong
baseline, indicating our proposed approach naturally matches the association
problem of MOT.

##  5 Conclusion & Impacts

Conclusion. Starting from a trajectory graph, we propose a straightforward
approach to model the association problem with a pure Transformer in a
decoupled and online fashion. We introduce PuTR to identify challenges in this
new direction, establishing a strong baseline compared to existing
foundational methods based on other methodologies. We find that the
Transformer naturally aligns with this problem, and the decoupled manner
enables efficient training and inference while achieving strong generalization
across diverse domains. Additionally, we present new perspectives on the
relationships among different methodologies, stimulating beneficial cross-
pollination and integration while considering the requirements and needs of
each other.

Impacts. Our work primarily focuses on the conceptual contribution of
developing a solution via a pure Transformer architecture, demonstrating the
feasibility of this direction for association. Through releasing our code, we
aim to inspire and facilitate further research in this promising new paradigm,
paving the way for more powerful and efficient MOT solutions. Furthermore, we
outline potential future directions:

Context length. The context length of the Transformer is a critical factor
influencing model performance. In our work, we set the context length to $30$
frames in most cases. However, as shown in Appendix E, simply increasing the
context length does not consistently yield benefits. Future work could
leverage advancements in long context modeling techniques from the language
model domain to specifically address this challenge.

Motion cues. Although PuTR primarily focuses on appearance cues for re-
identification, incorporating motion cues into the Transformer may further
enhance performance, particularly in challenging scenarios involving tiny
objects, as observed in MOT17 (Appendix F).

Overall, this work pioneers another innovative Transformer-based direction for
MOT. The self-attention-driven modeling of trajectories showcased in PuTR
opens up exciting possibilities for further advancements in this crucial
computer vision task.

## References

  * [1] Nir Aharon, Roy Orfaig, and Ben-Zion Bobrovsky.  Bot-sort: Robust associations multi-pedestrian tracking.  arXiv preprint arXiv:2206.14651, 2022.
  * [2] Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe.  Tracking without bells and whistles.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 941–951, 2019.
  * [3] Keni Bernardin and Rainer Stiefelhagen.  Evaluating multiple object tracking performance: the clear mot metrics.  EURASIP Journal on Image and Video Processing, 2008:1–10, 2008\.
  * [4] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft.  Simple online and realtime tracking.  In 2016 IEEE international conference on image processing (ICIP), pages 3464–3468. IEEE, 2016.
  * [5] Guillem Brasó and Laura Leal-Taixé.  Learning a neural solver for multiple object tracking.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6247–6257, 2020.
  * [6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [7] Jiarui Cai, Mingze Xu, Wei Li, Yuanjun Xiong, Wei Xia, Zhuowen Tu, and Stefano Soatto.  Memot: Multi-object tracking with memory.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8090–8100, 2022.
  * [8] Jinkun Cao, Jiangmiao Pang, Xinshuo Weng, Rawal Khirodkar, and Kris Kitani.  Observation-centric sort: Rethinking sort for robust multi-object tracking.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9686–9696, 2023.
  * [9] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [10] Orcun Cetintas, Guillem Brasó, and Laura Leal-Taixé.  Unifying short and long-term tracking with graph hierarchies.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22877–22887, 2023.
  * [11] Yutao Cui, Chenkai Zeng, Xiaoyu Zhao, Yichun Yang, Gangshan Wu, and Limin Wang.  Sportsmot: A large multi-object tracking dataset in multiple sports scenes.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9921–9931, 2023.
  * [12] Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixé.  Mot20: A benchmark for multi object tracking in crowded scenes.  arXiv preprint arXiv:2003.09003, 2020.
  * [13] Patrick Dendorfer, Vladimir Yugay, Aljosa Osep, and Laura Leal-Taixé.  Quo vadis: Is trajectory forecasting the key towards long-term multi-object tracking?  Advances in Neural Information Processing Systems, 35:15657–15671, 2022.
  * [14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [15] Yunhao Du, Zhicheng Zhao, Yang Song, Yanyun Zhao, Fei Su, Tao Gong, and Hongying Meng.  Strongsort: Make deepsort great again.  IEEE Transactions on Multimedia, 2023.
  * [16] Mohamed Elhoseny.  Multi-object detection and tracking (modt) machine learning model for real-time video surveillance systems.  Circuits, Systems, and Signal Processing, 39(2):611–630, 2020.
  * [17] Andreas Ess, Konrad Schindler, Bastian Leibe, and Luc Van Gool.  Object detection and tracking for autonomous navigation in dynamic environments.  The International Journal of Robotics Research, 29(14):1707–1725, 2010.
  * [18] Ruopeng Gao and Limin Wang.  Memotr: Long-term memory-augmented transformer for multi-object tracking.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9901–9910, 2023.
  * [19] Ruopeng Gao, Yijun Zhang, and Limin Wang.  Multiple object tracking as id prediction.  arXiv preprint arXiv:2403.16848, 2024.
  * [20] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun.  Yolox: Exceeding yolo series in 2021.  arXiv preprint arXiv:2107.08430, 2021.
  * [21] Weiming Hu, Tieniu Tan, Liang Wang, and Steve Maybank.  A survey on visual surveillance of object motion and behaviors.  IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 34(3):334–352, 2004.
  * [22] Hao Jiang, Sidney Fels, and James J Little.  A linear programming approach for multiple object tracking.  In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2007.
  * [23] R. E. Kalman.  A New Approach to Linear Filtering and Prediction Problems.  Journal of Basic Engineering, 82(1):35–45, 03 1960.
  * [24] Harold W Kuhn.  The hungarian method for the assignment problem.  Naval research logistics quarterly, 2(1-2):83–97, 1955.
  * [25] Chongwei Liu, Haojie Li, and Zhihui Wang.  Fasttrack: A highly efficient and generic gpu-based multi-object tracking method with parallel kalman filter.  International Journal of Computer Vision, pages 1–21, 2023.
  * [26] Kai Liu, Sheng Jin, Zhihang Fu, Ze Chen, Rongxin Jiang, and Jieping Ye.  Uncertainty-aware unsupervised multi-object tracking.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9996–10005, 2023.
  * [27] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang.  DAB-DETR: Dynamic anchor boxes are better queries for DETR.  In International Conference on Learning Representations, 2022.
  * [28] Y. Liu, J. Wu, and Y. Fu.  Collaborative tracking learning for frame-rate-insensitive multi-object tracking.  In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9930–9939, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society.
  * [29] Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe.  Hota: A higher order metric for evaluating multi-object tracking.  International journal of computer vision, 129:548–578, 2021.
  * [30] Gerard Maggiolino, Adnan Ahmad, Jinkun Cao, and Kris Kitani.  Deep oc-sort: Multi-pedestrian tracking by adaptive re-identification.  In 2023 IEEE International Conference on Image Processing (ICIP), pages 3025–3029. IEEE, 2023.
  * [31] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer.  Trackformer: Multi-object tracking with transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8844–8854, 2022.
  * [32] Anton Milan, Laura Leal-Taixé, Ian Reid, Stefan Roth, and Konrad Schindler.  Mot16: A benchmark for multi-object tracking.  arXiv preprint arXiv:1603.00831, 2016.
  * [33] Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu.  Quasi-dense similarity learning for multiple object tracking.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 164–173, 2021.
  * [34] Hamed Pirsiavash, Deva Ramanan, and Charless C. Fowlkes.  Globally-optimal greedy algorithms for tracking a variable number of objects.  In CVPR 2011, pages 1201–1208, 2011.
  * [35] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [36] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [37] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi.  Performance measures and a data set for multi-target, multi-camera tracking.  In European conference on computer vision, pages 17–35. Springer, 2016.
  * [38] Amir Roshan Zamir, Afshin Dehghan, and Mubarak Shah.  Gmcp-tracker: Global multi-object tracking using generalized minimum clique graphs.  In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part II 12, pages 343–356. Springer, 2012.
  * [39] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning representations by back-propagating errors.  nature, 323(6088):533–536, 1986.
  * [40] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.  The graph neural network model.  IEEE transactions on neural networks, 20(1):61–80, 2008.
  * [41] Mattia Segu, Bernt Schiele, and Fisher Yu.  Darth: Holistic test-time adaptation for multiple object tracking.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9717–9727, 2023.
  * [42] Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun.  Crowdhuman: A benchmark for detecting human in a crowd.  arXiv preprint arXiv:1805.00123, 2018.
  * [43] Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, and Ping Luo.  Dancetrack: Multi-object tracking in uniform appearance and diverse motion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20993–21002, 2022.
  * [44] Siyu Tang, Mykhaylo Andriluka, Bjoern Andres, and Bernt Schiele.  Multiple people tracking by lifted multicut and person re-identification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3539–3548, 2017.
  * [45] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [47] Shuai Wang, Da Yang, Yubin Wu, Yang Liu, and Hao Sheng.  Tracking game: Self-adaptative agent based multi-object tracking.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 1964–1972, 2022.
  * [48] Zhongdao Wang, Liang Zheng, Yixuan Liu, Yali Li, and Shengjin Wang.  Towards real-time multi-object tracking.  In European conference on computer vision, pages 107–122. Springer, 2020.
  * [49] Nicolai Wojke, Alex Bewley, and Dietrich Paulus.  Simple online and realtime tracking with a deep association metric.  In 2017 IEEE international conference on image processing (ICIP), pages 3645–3649. IEEE, 2017.
  * [50] Feng Yan, Weixin Luo, Yujie Zhong, Yiyang Gan, and Lin Ma.  Bridging the gap between end-to-end and non-end-to-end multi-object tracking.  arXiv preprint arXiv:2305.12724, 2023.
  * [51] Mingzhan Yang, Guangxin Han, Bin Yan, Wenhua Zhang, Jinqing Qi, Huchuan Lu, and Dong Wang.  Hybrid-sort: Weak cues matter for online multi-object tracking.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6504–6512, 2024.
  * [52] Kefu Yi, Kai Luo, Xiaolei Luo, Jiangui Huang, Hao Wu, Rongdong Hu, and Wei Hao.  Ucmctrack: Multi-object tracking with uniform camera motion compensation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6702–6710, 2024.
  * [53] En Yu, Tiancai Wang, Zhuoling Li, Yuang Zhang, Xiangyu Zhang, and Wenbing Tao.  Motrv3: Release-fetch supervision for end-to-end multi-object tracking.  arXiv preprint arXiv:2305.14298, 2023.
  * [54] Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei.  Motr: End-to-end multiple-object tracking with transformer.  In European Conference on Computer Vision, pages 659–675. Springer, 2022.
  * [55] Li Zhang, Yuan Li, and Ramakant Nevatia.  Global data association for multi-object tracking using network flows.  In 2008 IEEE conference on computer vision and pattern recognition, pages 1–8. IEEE, 2008.
  * [56] Y. Zhang, T. Wang, and X. Zhang.  Motrv2: Bootstrapping end-to-end multi-object tracking by pretrained object detectors.  In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22056–22065, Los Alamitos, CA, USA, jun 2023. IEEE Computer Society.
  * [57] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang.  Bytetrack: Multi-object tracking by associating every detection box.  In European conference on computer vision, pages 1–21. Springer, 2022.
  * [58] Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, and Wenyu Liu.  Fairmot: On the fairness of detection and re-identification in multiple object tracking.  International Journal of Computer Vision, 129:3069–3087, 2021.
  * [59] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Tracking objects as points.  In European conference on computer vision, pages 474–490. Springer, 2020.
  * [60] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  arXiv preprint arXiv:1904.07850, 2019.
  * [61] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable DETR: Deformable transformers for end-to-end object detection.  In International Conference on Learning Representations, 2021.

Appendix

The appendix complements our work with additional information on inference
pipeline in Appendix A. Furthermore, we provide more experiment details in
Appendix B, and visual results in Appendix C. We also conduct additional
experiments in Appendix D and E. Finally, we analyze the failure case in
Appendix F.

##  Appendix A Inference Pipeline.

Algorithm 1 Inference pipeline.

1:A video sequence $\mathcal{V}$; a detector $\textit{Det}(\cdot)$; a lost
tolerance $T$.

2:The set of trajectories $\mathcal{T^{*}}\Leftarrow\emptyset$; The set of
active trajectories $\mathcal{T}_{\textit{active}}^{*}\Leftarrow\emptyset$.

3:for $\textit{frame}_{t}$ in $\mathcal{V}$ do

4: /* generate detected objects */

5: $\mathcal{O}_{t}\Leftarrow\textit{Det}(\textit{frame}_{t})$;

6: /* generate an affinity matrix */

7:
$\mathbf{A}\Leftarrow\textit{GenAffinity}(\mathcal{T_{\textit{active}}^{*}},\mathcal{O}_{t})$;

8: /* associate */

9:
$\mathcal{O}_{m},\mathcal{T}_{m}^{*},\mathcal{O}_{u},\mathcal{T}_{u}^{*}\Leftarrow\textit{Hungarian}(\mathbf{A})$

10: $\mathcal{O}_{m}\\!:\text{matched detections from }\mathcal{O}_{t}$;

11: $\mathcal{T}_{m}^{*}:\text{matched tracklets from
}\mathcal{T_{\textit{active}}^{*}}$;

12: $\mathcal{O}_{u}:\text{unmatched detections from }\mathcal{O}_{t}$;

13: $\mathcal{T}_{u}^{*}:\text{unmatched tracklets from
}\mathcal{T_{\textit{active}}^{*}}$;

14: /* update tracklets */

15: for $\mathcal{T}_{\textit{tid}},o_{\textit{tid}}^{t}\text{ \rm\bf in
}\mathcal{T}_{m}^{*},\mathcal{O}_{m}$ do

16:
$\mathcal{T}_{\textit{tid}}\Leftarrow\textit{Update}(\mathcal{T}_{\textit{tid}},o_{\textit{tid}}^{t})$;

17: end for

18: /* remove tracklets */

19: for $\mathcal{T}_{\textit{tid}}\text{ \rm\bf in }\mathcal{T}_{u}^{*}$ do

20: if $\textit{LostGap}(\mathcal{T}_{\textit{tid}})>T\text{ \rm\bf or
}\mathcal{T}_{\textit{tid}}.\textsf{State}==\textit{New}$ then

21:
$\mathcal{T}_{u}^{*}\Leftarrow\mathcal{T}_{u}^{*}\setminus\\{\mathcal{T}_{\textit{tid}}$
};

22:
$\mathcal{T^{*}}\Leftarrow\mathcal{T}^{*}\cup\\{\mathcal{T}_{\textit{tid}}\\}$;

23: else

24: $\mathcal{T}_{\textit{tid}}.\textsf{State}\Leftarrow\textit{Lost}$;

25: end if

26: end for

27:
$\mathcal{T}_{\textit{active}}^{*}\Leftarrow\mathcal{T}_{m}^{*}\cup\mathcal{T}_{u}^{*}$;

28: /* initialize new tracklets */

29:
$\mathcal{T}_{\textit{active}}^{*}\Leftarrow\mathcal{T}_{\textit{active}}^{*}\cup$
Init( $\mathcal{O}_{u}$);

30:end for

31:$\mathcal{T}^{*}\Leftarrow\mathcal{T}^{*}\cup\mathcal{T}_{\textit{active}}^{*}$;

32:$\mathcal{T}^{*}$.

Algo. 1 shows the inference pipeline of our method. The pipeline follows a
similar style to heuristic methods, such as ByteTrack [57], while maintaining
a concise variant adopted from FastTrack [25], requiring only one association
during one frame to avoid increasing complexity. The inputs of the pipeline
are a video sequence $\mathcal{V}$, a detector $\textit{Det}(\cdot)$, and a
lost tolerance $T$ (same as the input frame length of PuTR). The output is the
set of trajectories $\mathcal{T}^{*}$, and we also maintain a set of active
trajectories $\mathcal{T}_{\textit{active}}^{*}$ to record the tracklets that
are not terminated yet. Each tracklet in $\mathcal{T}_{\textit{active}}^{*}$
has a state attribute, called State (one of Track, Lost, or New), to indicate
its status at the current frame. For each frame $\textit{frame}_{t}$ in
$\mathcal{V}$, we obtain the set of detected objects $\mathcal{O}_{t}$ via
$\textit{Det}(\cdot)$ (line 3), filtering by confidence threshold
$\tau_{\textit{det}}$ and initializing tracking identities tid to $-1$. We
then generate a relative affinity matrix using
$\mathcal{T}_{\textit{active}}^{*}$ and $\mathcal{O}_{t}$ through PuTR as
described in the main paper. To enhance tracking performance, we calculate the
affinity matrix on the entire output sequence $\mathcal{S}_{\textit{out}}$
instead of just the last two frames. We first compute a matrix via
$\textit{softmax}([\mathcal{Z}_{t-T},...,\mathcal{Z}_{t-1}]\mathcal{Z}_{t}^{\top})$,
and average the rows with the same tid as well as incorporate bounding box
information as in FastTrack to obtain the robust trajectory-detection relative
affinity matrix $\mathbf{A}$. The Hungarian algorithm then processes
$\mathbf{A}$ to obtain the assignment results, i.e., matched objects
$\mathcal{O}_{m}$, matched tracklets $\mathcal{T}_{m}^{*}$, unmatched
detections $\mathcal{O}_{u}$, and unmatched tracklets $\mathcal{T}_{u}^{*}$
(lines 4 to 9). The objects in $\mathcal{O}_{m}$ will be updated into the
corresponding tracklets in $\mathcal{T}_{m}^{*}$ (lines 10 to 12), where the
State of tracklets will be set to Track. Then, the tracklets in
$\mathcal{T}_{u}^{*}$ will be moved from $\mathcal{T}_{\textit{active}}^{*}$
into $\mathcal{T}^{*}$ if the lost gap is larger than $T$ or its State is New,
following ByteTrack. Otherwise, they will remain in
$\mathcal{T}_{\textit{active}}^{*}$, and their State will be set to Lost
(lines 13 to 20). Next, objects in $\mathcal{O}_{u}$ will be initialized as
new tracklets and added into $\mathcal{T}_{\textit{active}}^{*}$ after
filtering with the confidence threshold $\tau_{\textit{new}}$ (line 22).
Finally, after processing all frames in $\mathcal{V}$,
$\mathcal{T}_{\textit{active}}^{*}$ will be merged into $\mathcal{T}^{*}$, and
$\mathcal{T}^{*}$ is the final tracking result of the video sequence
$\mathcal{V}$ (line 24). Due to space constraints, more specific details about
$\textit{GenAffinity}(\cdot)$, $\textit{Update}(\cdot)$,
$\textit{LostGap}(\cdot)$, and $\textit{Init}(\cdot)$ functions cannot be
elaborated here. Please refer to our code repository for further information.

Table 4:  Comparison between the combined and single training on PuTR.

| Dancetrack test | SportsMOT test | MOT17 test | MOT20 test
---|---|---|---|---
PuTR |  I$\uparrow$ |  H$\uparrow$ |  M$\uparrow$ |  I$\uparrow$ |  H$\uparrow$ |  M$\uparrow$ |  I$\uparrow$ |  H$\uparrow$ |  M$\uparrow$ |  I$\uparrow$ |  H$\uparrow$ |  M$\uparrow$
Combined training | 57.0 | 54.5 | 91.8 | 75.0 | 74.0 | 97.0 | 74.7 | 61.4 | 78.3 | 74.3 | 61.2 | 75.2
Single training | 58.2 | 55.8 | 91.9 | 75.9 | 74.9 | 97.0 | 74.9 | 61.7 | 78.5 | 74.1 | 61.1 | 74.8

##  Appendix B Experiment Details

This section provides a comprehensive description of the experimental details
regarding the training and inference of our proposed PuTR. For more specific
details, we kindly refer the readers to our code repository.

Computational Efficiency. We leverage the grid_sample
function555torch.nn.functional.grid_sample to simultaneously sample the grid
center points of all objects within a frame. Moreover, the
scaled_dot_product_attention
function666torch.nn.functional.scaled_dot_product_attention is employed to
accelerate the calculation of the attention mechanism during both training and
inference. Additionally, we implement a run-time manager following FastTrack
[25], maintaining active tracklets in a tensor array instead of a list of
Python entities, thereby facilitating parallel processing and further
improving efficiency.

Training Details. During the training phase, the initial learning rate is set
to $0.0002$. We apply weight decay of $0.0005$, gradient clipping of $1.0$,
and accumulate gradients for $32$ steps. Following the practices of previous
works [56, 18], training clips are sampled from the training sequences with a
random interval ranging from $1$ to $10$, and augmented with random cropping,
random resizing, HSV color jittering, and random horizontal flipping to
enhance diversity.

Inference Details. In the inference stage, we set the confidence threshold
$\tau_{\textit{det}}$ to $0.1$ across all four datasets. The newborn threshold
$\tau_{\textit{new}}$ is set to $0.6$ for DanceTrack, SportsMOT, and MOT17,
and $0.4$ for MOT20, generally following the practices of previous works [57,
8].

Detection Results. The detection results for DanceTrack are obtained from
https://github.com/DanceTrack/DanceTrack, for MOT17 and MOT20 from
https://github.com/ifzhang/ByteTrack, and for SportsMOT from
https://github.com/MCG-NJU/MixSort.

Figure 2: Visual results of PuTR on the DanceTrack (top row) and the SportsMOT
(bottom row), showcasing the model’s capability to handle challenging
scenarios such as extended occlusions (the blue arrow), and the individual
moving out of and re-entering the camera view (the dark arrow).

##  Appendix C Visual Results

Fig.2 showcases visual results of PuTR on the DanceTrack and SportsMOT
datasets. The top row displays the tracking performance on the #dancetrack0011
video sequence from DanceTrack, while the bottom row illustrates the tracking
results on the #v_1UDUODIBSsc_c004 video sequence from SportsMOT. Leveraging
the Transformer’s capability to model long-range dependencies, PuTR can
identify individuals even after extended occlusions, without relying on motion
cues, as exemplified in the top row of Fig.2. The blue dancer is successfully
re-identified after a long-term occlusion and a large position change.
Furthermore, the long-range dependency modeling enables PuTR to track
individuals that move out of and re-enter the camera view. As depicted in the
bottom row of Fig. 2, the dark athlete moves out of the camera view in the
third frame due to camera motion and reappears in the fourth frame. In such
scenarios, conventional motion estimation methods struggle, whereas PuTR can
seamlessly track the individual within a simple Transformer architecture. The
above results constantly demonstrate PuTR’s ability to uniformly handle
challenging cases under a pure Transformer architecture, indicating the
promising potential of this new direction.

##  Appendix D Combined Training

In this section, we compare the performance of the combined training approach
with the single dataset training. The combined training refers to training
PuTR on the combined training sets of DanceTrack, SportsMOT, MOT17, and MOT20,
while the model is evaluated on the respective test sets. Conversely, the
single dataset training is the default training process described in the main
paper, where the model is trained on a single training set and evaluated on
the corresponding test set. The training process for the combined training is
consistent with the single dataset training, and the hyperparameters are kept
the same. However, due to GPU memory limitations, the training clip length is
fixed at $64$ frames after the $6$-th epoch, as MOT20 is included in the
combination.

As shown in Tab. 4, the combined training approach only yields a slight
improvement on the MOT20 dataset, while the performance decreases on the other
datasets. This observation suggests that the model trained on the combined
datasets does not exhibit a significant advantage over the model trained on a
single dataset.

##  Appendix E More Ablation Studies

In this section, we provide further ablation studies on the DanceTrack and
SportsMOT validation datasets, where I$\uparrow$, H$\uparrow$, M$\uparrow$,
and F$\uparrow$ denote IDF1, HOTA, MOTA, and FPS, respectively.

Structure Parameters. Tab. 6 presents the ablation study on the structure
parameters of PuTR. We experiment with different numbers of layers and hidden
dimensions $d_{\textit{model}}$. For the DanceTrack dataset, which features
similar appearance and complex interactions, PuTR requires more hidden
dimensions to distinguish individuals, and the performance improves with $512$
dimensions compared to $256$ dimensions. For SportsMOT, where athletes are
sparsely distributed and have straightforward trajectories, the performance is
not significantly affected by the hidden dimensions. Unlike end-to-end
methods, PuTR solely focuses on the relatively simple association task. Deep
layers can lead to overfitting, consequently decreasing performance, as
observed with the $9$-layer model compared to the $6$-layer model under $512$
dimensions. Therefore, we adopt the $6$-layer model with $512$ hidden
dimensions as the default setting to ensure robust performance across diverse
datasets.

Input Frame Length. Tab. 6 presents the ablation study on the input frame
length of PuTR during inference. A consistent observation is that the
processing speed decreases with increasing input frame length. From $5$ to
$120$ frames, the speed reduction is approximately $12$ FPS on both datasets,
while still maintaining a high level. Regarding performance, there is no
consistent frame length that achieves the best performance on both datasets.
For DanceTrack, $5$ frames achieve the best HOTA performance, while for
SportsMOT, $90$ frames achieve the best IDF1 performance. This indicates that
the distribution of lost gaps differs between datasets, and stronger long
context modeling techniques are required to specifically address this
challenge in future works. In our case, we set the default frame length to
$30$ to strike a balance between performance and speed across various
scenarios.

Table 5:  Ablations on the structure parameters.

PuTR | Dancetrack val | SportsMOT val
---|---|---
$d_{\textit{model}}$ | layers |  I$\uparrow$ |  H$\uparrow$ |  M$\uparrow$ |  I$\uparrow$ |  H$\uparrow$ |  M$\uparrow$
256 | 3 | 54.3 | 52.7 | 88.3 | 78.7 | 76.8 | 95.7
256 | 6 | 54.2 | 52.5 | 88.3 | 78.3 | 76.4 | 95.7
256 | 9 | 54.4 | 52.5 | 88.2 | 78.0 | 76.0 | 95.7
512 | 3 | 56.1 | 54.3 | 88.3 | 78.4 | 76.4 | 95.7
512 | 6 | 55.7 | 54.5 | 88.2 | 78.6 | 76.4 | 95.7
512 | 9 | 54.5 | 53.8 | 87.9 | 76.4 | 75.0 | 95.6

Table 6:  Ablations on the input frame length.

PuTR | Dancetrack val | SportsMOT val
---|---|---
$T$ |  I$\uparrow$ |  H$\uparrow$ |  M$\uparrow$ |  F$\uparrow$ |  I$\uparrow$ |  H$\uparrow$ |  M$\uparrow$ |  F$\uparrow$
5 | 54.3 | 55.5 | 89.0 | 91 | 76.8 | 75.9 | 95.6 | 91
15 | 55.3 | 54.0 | 88.7 | 89 | 78.7 | 77.1 | 95.8 | 89
30 | 55.7 | 54.5 | 88.2 | 89 | 78.6 | 76.4 | 95.7 | 88
60 | 53.6 | 51.7 | 88.0 | 86 | 79.0 | 76.5 | 95.6 | 84
90 | 53.7 | 51.8 | 87.7 | 83 | 79.3 | 76.1 | 95.6 | 83
120 | 53.9 | 51.9 | 87.8 | 78 | 77.8 | 74.8 | 95.5 | 77

##  Appendix F Failure Case

PuTR can re-identify the same object based on appearance cues, even after an
extended occlusion period, irrespective of its position change. However, this
characteristic also leads to a well-known limitation: the inability to
discriminate objects with weak appearance information, a common issue in
methods relying on the similarity of appearance features. As illustrated in
Fig. 3, we observe that PuTR produces false matches between small objects in
the MOT17 dataset, where these objects lack sufficient appearance cues for re-
identification. In contrast, from a motion perspective, they are easily
distinguishable. To alleviate this issue, future work could encode motion cues
into the Transformer to enhance the association performance.

Figure 3: An example of the failure case in video sequence #MOT17-14. The tiny
pink bounding box changes individuals twice in the second and fourth frame,
due to the weak appearance cues.