First, a model localizes all instances of a pre-defined object class in every video frame, and then, it links these together to form object tracks of arbitrary length.

While object class detection has improved considerably in the last few years [@girshick2015fast; @ren2015faster; @dai2016r; @he2017mask; @lin2017feature; @dai2017deformable], association remains a challenging task. Many approaches modelled this as a graph-based optimization problem, where each node is a detection and each edge a possible link [@zhang2008global; @berclaz2011multiple; @zamir2012gmcp; @kim2015multiple; @tang2017multiple; @henschel2017improvements; @ristani2018features; @keuper2018motion; @sheng2018heterogeneous; @xu2019spatial; @xu2019train]. These approaches depend on a complex combination of multiple cues, like appearance, re-id, motion estimation and object interactions to solve this association. These have helped MOT models gradually improve their performance [@milan2016mot16], but at the cost of inefficiency and high computation, as each of the employed cues is often its own deep network trained on its own specific dataset.

Bergmann et al. [@bergmann2019tracking] identified this shortcoming and as a step towards accurate and fast MOT proposed Tracktor, an efficient tracking framework that only relies on a detection module. While Tracktor achieves state-of-the-art performance on the MOTChallenge [@milan2016mot16], it suffers from two limitations: (i) it does not capture the appearance of the target object during tracking, which causes 