Introduction and Motivation {#sec:intro}
===========================

We focus on the challenging task of multi-object tracking and segmentation (MOTS) [@Voigtlaender_2019_CVPR], which was recently introduced as an extension of bounding-box based multi-object tracking. Joining instance segmentation with tracking was shown to noticeably improve tracking performance, as unlike bounding boxes, instance segmentation masks do not suffer from overlapping issues but provide fine-grained, pixel-level information about the objects to be tracked.

While this finding is encouraging, it comes with the downside of requiring pixel-level (typically polygon-based) annotations, which are known to be time-consuming in generation and thus expensive to obtain. The works in [@Cordts2016; @MVD2017] report annotation times of $\approx$90 minutes per image, to manually produce high-quality panoptic segmentation masks. Analogously, it is highly demanding to produce datasets for the MOTS task where instance segmentation masks need to also contain tracking information across frames.

To avoid generating MOTS labels completely from scratch and in a purely manual way, [@Voigtlaender_2019_CVPR] has followed the semi-automatic annotation procedure from [@Caelles_2017_CVPR], and extended the existing multi-object tracking datasets *KITTI tracking* [@Geiger2012CVPR] (21 sequences from the raw KITTI dataset) and MOTChallenge [@MilanL0RS16] (4/7 sequences). These datasets already provide bounding box-based tracklets for cars and pedestrians. Instance segmentation masks were generated 