In this work we introduce a new framework for performing temporal predictions
 in the presence of uncertainty. It is based on a simple idea of disentangling com-
 ponents of the future state which are predictable from those which are inherently
 unpredictable, and encoding the unpredictable components into a low-dimensional
 latent variable which is fed into the forward model. Our method uses a simple su-
 pervised training objective which is fast and easy to train. We evaluate it in the
 context of video prediction on multiple datasets and show that it is able to consi-
 tently generate diverse predictions without the need for alternating minimization
 over a latent space or adversarial training. Learning forward models in time series is a central task in artificial intelligence, with applications in unsupervised learning, planning and compression. A major challenge in this task is how to handle the multi-modal nature of many time series. When there are multiple valid ways in which a time series can evolve, training a model using classical 1 or 2 losses produces predictions which are the average or median of the different outcomes across each dimension, which is itself often not a valid prediction.In recent years, Generative Adversarial Networks BID9 have been introduced, a general framework where the prediction problem is formulated as a minimax game between the predictor function and a trainable discriminator network representing the loss. By using a trainable loss function, it is in theory possible to handle multiple output modes since a generator which covers each of the output modes will fool the discriminator leading to convergence. However, a generator which covers a single mode can also fool the discriminator and converge, and this behavior of mode collapse has been widely observed in practice. Some workarounds have been introduced to resolve or partially reduce mode-collapsing, such as minibatch discrimination, adding parameter noise BID23 , backpropagating through the unrolled discriminator BID18 and using multiple GANs to cover different modes BID27 . However, many of these techniques can bring additional challenges such as added complexity of implementation and increased computational cost. The mode collapsing problem becomes even more pronounced in the conditional generation setting when the output is highly dependent on the context, such as video prediction BID12 .In this work, we introduce a novel architecture that allows for robust multimodal conditional predictions in time series data. It is based on a simple intuition of separating the future state into a deterministic component, which can be predicted from the current state, and a stochastic (or difficult to predict) component which accounts for the uncertainty regarding the future mode. By training a deterministic network, we can obtain this factorization in the form of the network's prediction together with the prediction error with respect to the true state. This error can be encoded as a lowdimensional latent variable which is fed into a second network which is trained to accurately correct the determinisic prediction by incorporating this additional information. We call this model the Error Encoding Network (EEN). In a nutshell , this framework contains three function mappings at each timestep: (i) a mapping from the current state to the future state, which separates the future state into deterministic and non-deterministic components; (ii) a mapping from the non-deterministic component of the future state to a low-dimensional latent vector; (iii) a mapping from the current state to the future state conditioned on the latent vector, which encodes the mode information of the future state. While the training procedure involves all these mappings, the inference phase involves only (iii).Both networks are trained end-to-end using a supervised learning objective and latent variables are computed using a learned parametric function, leading to easy and fast training. We apply this method to video datasets from games, robotic manipulation and simulated driving, and show that the method is able to consistently produce multimodal predictions of future video frames for all of them. Although we focus on video in this work, the method itself is general and can in principle be applied to any continuous-valued time series. In this work, we have introduced a new framework for performing temporal prediction in the presence of uncertainty by disentangling predictable and non-predictable components of the future state. It is fast, simple to implement and easy to train without the need for an adverserial network or al- ternating minimization, and does not require additional tuning to prevent mode collapse. We have provided one instantiation in the context of video prediction using convolutional networks, but it is in principle applicable to different data types and architectures. There are several directions for future work. Here, we have adopted a simple strategy of sampling uniformly from the z distribution without considering their possible dependence on the state x, and there are likely better methods. In addition, one advantage of our model is that it can extract latent variables from unseen data very quickly, since it simply requires a forward pass through a network. If latent variables encode information about actions in a manner that is easy to disentangle, this could be used to extract actions from large unlabeled datasets and perform imitation learning. Another interesting application would be using this model for planning and having it unroll different possible futures.
A simple and easy to train method for multimodal prediction in time series.  This paper introduces a times-series prediction model that that learns a deterministic mapping and trains another net to predict future frames given the input and residual error from the first network. The paper proposes a model for prediction under uncertainty where they separate out deterministic component prediction and uncertain component prediction.