Intelligent agents can learn to represent the action spaces of other agents simply by observing them act. Such representations help agents quickly learn to predict the effects of their own actions on the environment and to plan complex action sequences. In this work, we address the problem of learning an agent’s action space purely from visual observation. We use stochastic video prediction to learn a latent variable that captures the scene's dynamics while being minimally sensitive to the scene's static content. We introduce a loss term that encourages the network to capture the composability of visual sequences and show that it leads to representations that disentangle the structure of actions. We call the full model with composable action representations Composable Learned Action Space Predictor (CLASP). We show the applicability of our method to synthetic settings and its potential to capture action spaces in complex, realistic visual settings. When used in a semi-supervised setting, our learned representations perform comparably to existing fully supervised methods on tasks such as action-conditioned video prediction and planning in the learned action space, while requiring orders of magnitude fewer action labels. Project website: https://daniilidis-group.github.io/learned_action_spaces
We learn a representation of an agent's action space from pure visual observations. We use a recurrent latent variable approach with a novel composability loss. Proposes a compositional latent-variable model to learn models that predict what will happen next in scenarios where action-labels are not available in abundance. A variational IB based approach to learn action representations directly from videos of actions being taken, achieving better efficiency of subsequent learning methods while requiring lesser amount of action label videos. This paper proposes an approach to video prediction which autonomously finds an action space encoding differences between subsequent frames