constraint, which allows segments to remain unassigned. In other terms, we solve the following optimization problem: $$\begin{aligned}
  \label{eq:optim}
    \text{maximize}&\int\int \pi(\hat s, s) \alpha(\hat s, s)\, d\hat s\,ds\\
  \label{eq:constr1}
    \text{s.t.}& \int\alpha(\hat s,s)ds\leq 1\qquad \forall \hat s\in{\ensuremath{\mathcal{S}}}_{t-1}\\
  \label{eq:constr2}
    & \int\alpha(\hat s,s)d\hat s\leq 1\qquad \forall s\in{\ensuremath{\mathcal{S}}}_{t}\\
  \label{eq:constr3}
    &\alpha(\hat s, s)\in\{0,1\}\quad\forall (\hat s, s)\in{\ensuremath{\mathcal{S}}}_{t-1}\times{\ensuremath{\mathcal{S}}}_t\,.\end{aligned}$$ Finally, the solution $\alpha$ of the assignment problem can be used to update the set of edges ${\ensuremath{\mathcal{T}}}_t$, ${\ensuremath{\mathcal{T}}}_t={\ensuremath{\mathcal{T}}}_{t-1}\cup \alpha^{-1}(1)$, where $\alpha^{-1}(1)$ is the set of matching segments, the pre-image of $1$ under $\alpha$.

Note that this algorithm can’t track objects across full occlusions, since optical flow can’t be used to infer the trajectories of invisible objects. When training [MOTSNet]{}however, we easily overcome this limitation with a simple heuristic, described in Sec. \[sec:training\]. As will be shown in Sec. \[sec:experiments-ks\], [MOTSNet]{}predictions actually *outperform* the ground truth data it was trained on.

[MOTSNet]{} {#sec:main}
===========

Here we describe our multi-object tracking and segmentation approach. Its main component is [MOTSNet]{}(see Sec. \[sec:architecture\]), a deep net based on the Mask R-CNN [@He2017] framework. Given an RGB image, [MOTSNet]{}outputs a set of instance segments, each augmented with a learned embedding vector which represents the object’s identity in the sequence (see Sec. \[sec:training\]). The object 