We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as  offensive basketball gameplay.   When modeling such settings, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables.   Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulable way. We present a hierarchical framework that can effectively learn such sequential generative models.   Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime. In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods. We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts. The ongoing explosion of recorded tracking data is enabling the study of fine-grained behavior in many domains: sports BID25 BID44 BID46 BID20 , video games BID34 , video & motion capture BID36 BID38 BID43 , navigation & driving BID48 BID45 BID22 , laboratory animal behaviors BID16 BID7 , and tele-operated robotics BID0 BID23 . However, it is an open challenge to develop sequential generative models leveraging such data, for instance, to capture the complex behavior of multiple cooperating agents. FIG1 shows an example of offensive players in basketball moving unpredictably and with multimodal distributions over possible trajectories. FIG1 depicts a simplified Boids model from BID31 for modeling animal schooling behavior in which the agents can be friendly or unfriendly. In both cases, agent behavior is highly coordinated and non-deterministic, and the space of all multi-agent trajectories is naively exponentially large.When modeling such sequential data, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables or representations BID21 BID46 ). An attractive use-case for these intermediate variables is to capture interesting highlevel behavioral semantics in an interpretable and manipulable way. For instance, in the basketball setting, intermediate variables can encode long-term strategies and team formations. Conventional approaches to learning interpretable intermediate variables typically focus on learning disentangled latent representations in an unsupervised way (e.g., BID22 BID42 ), but it is challenging for such approaches to handle complex sequential settings BID4 . To address this challenge, we present a hierarchical framework that can effectively learn such sequential generative models, while using programmatic weak supervision. Our approach uses a labeling function to programmatically produce useful weak labels for supervised learning of interpretable intermediate representations. This approach is inspired by recent work on data programming BID29 , which uses cheap and noisy labeling functions to significantly speed up learning. In this work, we extend this approach to the spatiotemporal regime.Our contributions can be summarized as follows:• We propose a hierarchical framework for sequential generative modeling. Our approach is compatible with many existing deep generative models.• We show how to programmatically produce weak labels of macro-intents to train the intermediate representation in a supervised fashion. Our approach is easy to implement and results in highly interpretable intermediate variables, which allows for conditional inference by grounding macro-intents to manipulate behaviors.• Focusing on multi-agent tracking data, we show that our approach can generate highquality trajectories and effectively encode long-term coordination between multiple agents.In addition to synthetic settings, we showcase our approach in an application on modeling team offense in basketball. We validate our approach both quantitatively and qualitatively, including a user study comparison with professional sports analysts, and show significant improvements over standard baselines. The macro-intents labeling functions used in our experiments are relatively simple. For instance, rather than simply using location-based macro-intents, we can also incorporate complex interactions such as "pick and roll". Another future direction is to explore how to adapt our method to different domains, e.g., defining a macro-intent representing "argument" for a dialogue between two agents, or a macro-intent representing "refrain" for music generation for "coordinating instruments" BID40 . We have shown that weak macro-intent labels extracted using simple domain-specific heuristics can be effectively used to generate high-quality coordinated multi-agent trajectories. An interesting direction is to incorporate multiple labeling functions, each viewed as noisy realizations of true macro-intents, similar to BID29 BID2 .
We blend deep generative models with programmatic weak supervision to generate coordinated multi-agent trajectories of significantly higher quality than previous baselines. Proposes multi-agent sequential generative models. The paper proposes training generative models that produce multi-agent trajectories using heuristic functions that label variables that would otherwise be latent in training data