This work studies the problem of modeling non-linear visual processes by leveraging deep generative architectures for learning linear, Gaussian models of observed sequences. We propose a joint learning framework, combining a multivariate autoregressive model and deep convolutional generative networks. After justification of theoretical assumptions of inearization, we propose an architecture that allows Variational Autoencoders and Generative Adversarial Networks to simultaneously learn the non-linear observation as well as the linear state-transition model from a sequence of observed frames. Finally, we demonstrate our approach on conceptual toy examples and dynamic textures. While classification of image and video with Convolutional Neural Networks (CNN) is becoming an established practice, unsupervised learning and generative modeling remain to be challenging problems in deep learning. A generative model of a visual process enables the possibility of generating sequences of video frames such that the appearance as well as the dynamics approximately resemble the original training process without copying it. This procedure is typically referred to as video generation BID25 BID6 ) or video synthesis BID19 ). More technically, this means that in addition to a suitable probability model for the individual frames, a probabilistic description for the frame-to-frame transition is also necessary. Analysis and reproduction of visual processes simplifies considerably, if this transition can be described as a multivariate autoregressive (MAR) model, i.e., as a combination of linear transformations and Gaussian noise. For instance, linear transformations are easily invertible and by means of spectral analysis, it can be studied how such a process behaves in the long term.Realistically, most frame transitions in real-world visual processes unlikely are linear functions. Nevertheless, unsupervised learning has come up with many approaches to fit MAR models to realworld processes, for instance by using linear low-rank approximations, as proposed by BID8 , or sparse approximations of the frames, as proposed by BID28 , or applying the kernel trick to them BID3 ).The success of Generative Adversarial Networks (GAN) introduced by BID9 and Variational Autoencoders (VAE) introduced by BID15 has led to an increased interest in deep generative learning and it seems natural to apply such techniques to sequential processes. We approach this idea from the perspective of linearization, in order to keep the model as simple as possible. In an analogous way as physicists transforming non-linear differential equations into linear ones by means of an appropriate change of variables, our approach is to learn latent representations of visual processes, such that the latent state-to-state transition can be described by an MAR model. To this end, we jointly learn a non-linear observation and a linear state transition function by introducing a dynamic layer that can be used in conjunction with deep generative architectures such as GANs and VAEs. This work presents an approach to learn embedded MAR models from image sequences. We motivate the feasibility of this approach by introducing the concept of local linearizability and propose a joint learning procedure that employs deep generative models in combination with an additional linear component, the dynamic layer. We report first positive results on low-resolution visual processes, where a first-order Markov property can be assumed, and hope to shed some light on the nature of linearization. A possible future research direction is improving the theoretical understanding of linearizing representations and their applicability outside of stationary visual processes. Let Φ ∈ R n×n be a matrix. Since Γ is a diffeomorphism, we define DISPLAYFORM0 holds. Let us denote the Jacobian of φ at y * by J φ . Because Γ maps y * to the origin, we can reformulate the requirement as DISPLAYFORM1 This requirement is fulfilled if the Jacobi matrices of ϕ and φ coincide, i.e. DISPLAYFORM2 The Jacobian of φ at y * is given, according to he chain rule, by DISPLAYFORM3 where J Γ −1 ∈ R d×n is the Jacobian matrix of Γ −1 at Γ(y * ). A matrix Φ ∈ R n×n can be always found, such that Eq. FORMULA2 is fulfilled, if the columns and rows of J ϕ lie in the column space of J Γ −1 and the row space of J Γ , respectively. The column space of J Γ −1 coincides with the row space of J Γ , due to the identity J Γ J Γ −1 = I n . The statement of the proposition follows.
We model non-linear visual processes as autoregressive noise via generative deep learning. Proposes a new method that models non-linear visual process with a deep version of a linear process (Markov process). This paper proposes a new deep generative model for sequences, particularly image sequences and video, which uses a linear structure in part of the model.