We consider the problem of unsupervised learning of a low dimensional, interpretable, latent state of a video containing a moving object. The problem of distilling dynamics from pixels has been extensively considered through the lens of graphical/state space models that exploit Markov structure for cheap computation and structured graphical model priors for enforcing interpretability on latent representations. We take a step towards extending these approaches by discarding the Markov structure; instead, repurposing the recently proposed Gaussian Process Prior Variational Autoencoder for learning sophisticated latent trajectories. We describe the model and perform experiments on a synthetic dataset and see that the model reliably reconstructs smooth dynamics exhibiting U-turns and loops. We also observe that this model may be trained without any beta-annealing or freeze-thaw of training parameters. Training is performed purely end-to-end on the unmodified evidence lower bound objective. This is in contrast to previous works, albeit for slightly different use cases, where application specific training tricks are often required. We consider the problem of unsupervised learning of a low dimensional, interpretable, latent state of a video containing a moving object. The problem of distilling interpretable dynamics from pixels has been extensively considered through the lens of graphical/state space models (Fraccaro et al., 2017; Lin et al., 2018; Pearce et al., 2018; Chiappa and Paquet, 2019 ) that exploit Markov structure for cheap computation and structured priors for enforcing interpretability on latent representations. We take a step towards extending these approaches by discarding the Markov structure; inspired by Gaussian process dynamical models (Wang et al., 2006) , we instead repurpose the recently proposed Gaussian Process Prior Variational Autoencoder (Casale et al., 2018) for learning interpretable latent dynamics. We describe the model and perform experiments on a synthetic dataset and see that the model reliably reconstructs smooth dynamics exhibiting U-turns and loops. We also observe that this model may be trained without any β annealing or freeze-thaw of training parameters in contrast to previous works, albeit for slightly different use cases, where application specific training tricks are often required. We present a simple model and show proof-of-concept results that a Gaussian Process Prior within a VAE may be used for learning complex but smooth latent dynamics without any Input VAE Latent GPP-VAE Latent Figure 3 : Left: top: 19 images, bottom: 25 images generated with the ball in a regular pattern. Centre: the patterns output from the recognition network q * (x, y|v) from the trained VAE. Ground truth in blue and recognition network means in orange (rotated onto ground truth). Lines are for visual aid only. Right: the output of q * (x, y|v) from the trained GPP-VAE (rotated onto ground truth). There is no time correlation in the images hence we do not plot approximate posterior/apply smoothing. The VAE latent space is a highly distorted and discontinuous transformation of the pixel space while the GPP-VAE latent space is much more coherent. For training, see video https://www.youtube.com/watch?v=riVhb6K_iMo . special training. In this work we consider a toy dataset and the dynamics model generating the data was also used to fit the model removing miss-specification issues. Hence future work is to apply the model to a wider variety of less controlled settings, and comparison with more sophisticated baselines. By comparison, using similar data, the KalmanVariational Autoencoder learnt dynamics (also including sharp turns, hence non-smooth) using an LSTM and training required freeze-thaw of model parameters and re-weighting of objective terms. Likewise extensions to this model (Chiappa and Paquet, 2019; Pearce et al., 2018) consider multiple objects constrained to parabolic motion and either require β annealing or other training tricks.
We learn sohpisticated trajectories of an object purely from pixels with a toy video dataset by using a VAE structure with a Gaussian process prior.