A disentangled representation of a data set should be capable of recovering the underlying factors that generated it. One question that arises is whether using Euclidean space for latent variable models can produce a disentangled representation when the underlying generating factors have a certain geometrical structure. Take for example the images of a car seen from different angles. The angle has a periodic structure but a 1-dimensional representation would fail to capture this topology. How can we address this problem? The submissions presented for the first stage of the  NeurIPS2019 Disentanglement Challenge consist of a Diffusion Variational Autoencoder ($\Delta$VAE) with a hyperspherical latent space which can for example recover periodic true factors. The training of the $\Delta$VAE is enhanced by incorporating a modified version of the Evidence Lower Bound (ELBO) for tailoring the encoding capacity of the posterior approximate. Variational Autoencoders (VAEs) proposed by BID4 are an unsupervised learning method that can estimate the underlying generative model that produced a data set in terms of the so-called latent variables. In the context of VAEs, a disentangled representation is obtained when the latent variables represent the true independent underlying factors, which usually have a semantic meaning, that generated the data set.VAEs assume that a data set X = {x i } N i=1 consists of N independent and identically distributed data points belonging to a set X. A set Z of unobserved latent variables is proposed and the main goal is to maximize the log-likelihood via variational inference using an approximate to the posterior distribution Q X|z with parameters a, b calculated by neural networks. A prior distribution P Z is selected before training such that the training of the VAE is carried out by maximizing for each data point the Evidence Lower Bound (ELBO) w.r.t. the neural network weights that calculate a, b given by DISPLAYFORM0 To accomplish the disentanglement of latent variables BID3 proposed to weight the contribution of both terms in the ELBO by using a parameter β ∈ R + to change the capacity of encoding of the posterior distribution. The idea of changing the capacity of the encoding distribution was further explored in BID0 where the KullbackLeibler divergence term is pushed towards a certain value C ∈ R + in each training step. The combination of both approaches led to a to the following training objective to be maximized, DISPLAYFORM1 The value of β is fixed before training and C is increased linearly each epoch of training from a minimum value C min to C max . We refer to this procedure as capacity annealing.In some cases the underlying factors that generated a data set have a certain geometrical/topological structure that cannot be captured with the traditional Euclidean latent variables as has been mentioned in and in . This problem is referred to as manifold mismatch.For the NeurIPS2019 Disentanglement challenge, datasets for local evaluation are provided based on the paper by BID5 . It is important to note that in such datasets there is at least one underlying factor that has a periodic structure. Take for example the Cars3D dataset consisting of images of cars. In particular, one factor of variation is the azimuthal angle of rotation of the car. The geometrical structure of this factor is circular and thus it is better represented with a periodical latent variable.The Diffusion Variational Autoencoders ∆VAE presented by Pérez Rey et al. FORMULA0 provide a versatile method that can be used to implement arbitrary closed manifolds for a latent space, in particular, hyperspheres.
Description of submission to NeurIPS2019 Disentanglement Challenge based on hyperspherical variational autoencoders