Lifelong learning is the problem of learning multiple consecutive tasks in a sequential manner where knowledge gained from previous tasks is retained and used for future learning. It is essential towards the development of intelligent machines that can adapt to their surroundings. In this work we focus on a lifelong learning approach to generative modeling where we continuously incorporate newly observed streaming distributions into our learnt model. We do so through a student-teacher architecture which allows us to learn and preserve all the distributions seen so far without the need to retain the past data nor the past models. Through the introduction of a novel cross-model regularizer, the student model leverages the information learnt by the teacher, which acts as a summary of everything seen till now. The regularizer has the additional benefit of reducing the effect of catastrophic interference that appears when we learn over streaming data. We demonstrate its efficacy on streaming distributions as well as its ability to learn a common latent representation across a complex transfer learning scenario.
 Deep unsupervised generative learning allows us to take advantage of the massive amount of unlabeled data available in order to build models that efficiently compress and learn an approximation of the true data distribution. It has numerous applications such as image denoising, inpainting, super-resolution, structured prediction, clustering, pre-training and many more. However, something that is lacking in the modern ML toolbox is an efficient way to learn these deep generative models in a sequential, lifelong setting.In a lot of real world scenarios we observe distributions sequentially. Examples of this include streaming data from sensors such as cameras and microphones or other similar time series data. A system can also be resource limited wherein all of the past data or learnt models cannot be stored. We are interested in the lifelong learning setting for generative models where data arrives sequentially in a stream and where the storage of all data is infeasible. Within the stream, instances are generated according to some non-observed distribution which changes at given time-points. We assume we know the time points at which the transitions occur and whether the latent distribution is a completely new one or one that has been observed before. We do not however know the underlying identity of the individual distributions. Our goal is to learn a generative model that can summarize all the distributions seen so far in the stream. We give an example of such a setting in figure 1(a ) using MNIST BID19 , where we have three unique distributions and one that is repeated.Since we only observe one distribution at a time we need to develop a strategy of retaining the previously learnt knowledge (i.e. the previously learnt distributions) and integrate it into future learning. To accumulate additional distributions in the current generative model we utilize a student-teacher architecture similar to that in distillation methods BID9 ; BID4 . The teacher contains a summary of all past distributions and is used to augment the data used to train the student model. The student model thus receives data samples from the currently observable distribution as well as synthetic data samples from previous distributions. This allows the student model to learn a distribution that summarizes the current as well as all previously observed distributions. Once a new distribution shift occurs the existing teacher model is discarded, the student becomes the teacher and a new student is instantiated.We further leverage the generative model of the teacher by introducing a regularizer in the learning objective function of the student that brings the posterior distribution of the latter close to that of the former. This allows us to build upon and extend the teacher's generative model in the student each time the latter is re-instantiated (rather than re-learning it from scratch). By coupling this regularizer with a weight transfer from the teacher to the student we also allow for faster convergence of the student model. We empirically show that the regularizer allows us to learn a much larger set of distributions without catastrophic interference BID23 .We build our lifelong generative models over Variational Autoencoders (VAEs) BID17 . VAEs learn the posterior distribution of a latent variable model using an encoder network; they generate data by sampling from a prior and decoding the sample through a conditional distribution learnt by a decoder network.Using a vanilla VAE as a teacher to generate synthetic data for the student is problematic due to a couple of limitations of the VAE generative process. 1) Sampling the prior can select a point in the latent space that is in between two separate distributions, causing generation of unrealistic synthetic data and eventually leading to loss of previously learnt distributions. 2) Additionally, data points mapped to the posterior that are further away from the prior mean will be sampled less frequently resulting in an unbalanced sampling of the constituent distributions. Both limitations can be understood by visually inspecting the learnt posterior distribution of a standard VAE evaluated on test images from MNIST as shown in figure 1(b). To address the VAE's sampling limitations we decompose the latent variable vector into a continuous and a discrete component. The discrete component is used to summarize the discriminative information of the individual generative distributions while the continuous caters for the remaining sample variability. By independently sampling the discrete and continuous components we preserve the distributional boundaries and circumvent the two problems above.This sampling strategy, combined with the proposed regularizer allows us to learn and remember all the individual distributions observed in the past. In addition we are also able to generate samples from any of the past distributions at will; we call this property consistent sampling. In this work we propose a novel method for learning generative models over streaming data following the lifelong learning principles. The principal assumption for the data is that they are generated by multiple distributions and presented to the learner in a sequential manner (a set of observations from a single distribution followed by a distributional transition). A key limitation for the learning is that the method can only access data generated by the current distribution and has no access to any of the data generated by any of the previous distributions.The proposed method is based on a dual student-teacher architecture where the teacher's role is to preserve the past knowledge and aid the student in future learning. We argue for and augment the standard VAE's ELBO objective by terms helping the teacher-student knowledge transfer. We demonstrate on a series of experiments the benefits this augmented objective brings in the lifelong learning settings by supporting the retention of previously learned knowledge (models) and limiting the usual effects of catastrophic interference.In our future work we will explore the possibilities to extend our architecture to GAN-like BID8 learning with the prospect to further improve the generative abilities of our method. GANs, however, do not use a metric for measuring the quality of the learned distributions such as the marginal likelihood or the ELBO in their objective and therefore the transfer of our architecture to these is not straightforward. Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
Lifelong distributional learning through a student-teacher architecture coupled with a cross model posterior regularizer.