a novel approach for automatically generating high-quality training data (see Fig. \[fig:sequence\] for an example) from generic street-level videos for the task of joint multi-object tracking and segmentation. Our methods are conceptually simple and leverage state-of-the-art instance [@He2017] (or Panoptic [@Por+19_cvpr; @cheng2019panopticdeeplab]) segmentation models trained on *existing* image datasets like Mapillary Vistas [@MVD2017] for the task of object instance segmentation. We further describe how to automatically mine tracking labels for all detected object instances, building upon state-of-the-art optical flow models [@Yin_2019_CVPR], that in turn have been trained on automatically harvested flow supervision obtained from image-based 3d modeling (structure-from-motion).

Another important contribution of our paper is [MOTSNet]{}, a deep learning, tracking-by-detection approach for MOTS. [MOTSNet]{}takes advantage of a novel mask-pooling layer, which allows it to exploit instance segmentation masks to compute a representative embedding vector for each detected object. The tracking process can then be formulated as a series of Linear Assignment Problems, optimizing a payoff function which compares detections in the learned embedding space.

We demonstrate the superior quality for each step of our proposed MOTS training data extraction pipeline, as well as the efficacy of our novel mask-pooling layer. The key contributions and findings of our approach are:

-   