Recent work has shown increased interest in using the Variational Autoencoder (VAE) framework to discover interpretable representations of data in an unsupervised way. These methods have focussed largely on modifying the variational cost function to achieve this goal. However, we show that methods like beta-VAE simplify the tendency of variational inference to underfit causing pathological over-pruning and over-orthogonalization of learned components. In this paper we take a complementary approach: to modify the probabilistic model to encourage structured latent variable representations to be discovered. Specifically, the standard VAE probabilistic model is unidentifiable: the likelihood of the parameters is invariant under rotations of the latent space. This means there is no pressure to identify each true factor of variation with a latent variable.
 We therefore employ a rich prior distribution, akin to the ICA model, that breaks the rotational symmetry.
 Extensive quantitative and qualitative experiments demonstrate that the proposed prior mitigates the trade-off introduced by modified cost functions like beta-VAE and TCVAE between reconstruction loss and disentanglement. The proposed prior allows to improve these approaches with respect to both disentanglement and reconstruction quality significantly over the state of the art. Recently there has been an increased interest in unsupervised learning of disentangled representations. The term disentangled usually describes two main objectives: First, to identify each true factor of variation with a latent variable, and second, interpretability of these latent factors (Schmidhuber, 1992; Ridgeway, 2016; BID0 . Most of this recent work is inspired by the β-VAE concept introduced in BID11 , which proposes to re-weight the terms in the evidence lower bound (ELBO) objective. In BID11 a higher weight for the Kullback-Leibler divergence (KL) between approximate posterior and prior is proposed, and putative mechanistic explanations for the effects of this modification are studied in BID4 ; BID5 . An alternative decomposition of the ELBO leads to the recent variant of β-VAE called β-TCVAE BID5 , which shows the highest scores on recent disentanglement benchmarks.These modifications of the evidence lower bound however lead to a trade-off between disentanglement and reconstruction loss and therefore the quality of the learned model. This trade-off is directly encoded in the modified objective: by increasing the β-weight of the KL-term, the relative weight of the reconstruction loss term is more and more decreased. Therefore, optimization of the modified ELBO will lead to latent encodings which have a lower KL-divergence from the prior, but at the same time lead to a higher reconstruction loss. Furthermore, we discuss in section 2.4 that using a higher weight for the KL-term amplifies existing biases of variational inference, potentially to a catastrophic extent.There is a foundational contradiction in many approaches to disentangling deep generative models (DGMs): the standard model employed is not identifiable as it employs a standard normal prior which then undergoes a linear transformation. Any rotation of the latent space can be absorbed into the linear transform and is therefore statistically indistinguishable. If interpretability is desired, the modelling choices are setting us up to fail.We make the following contributions:• We show that the current state of the art approaches employ a trade-off between reconstruction loss and disentanglement of the latent representation.• In section 2.3 we show that variational inference techniques are biased: the estimated components are biased towards having orthogonal effects on the data and the number of components is underestimated.• We provide a novel description of the origin of disentanglement in β-VAE and demonstrate in section 2.4 that increasing the weight of the KL term increases the over-pruning bias of variational inference.• To mitigate these drawbacks of existing approaches, we propose a family of rotationally asymmetric distributions for the latent prior, which removes the rotational ambiguity from the model. This approach resembles independent component analysis (ICA) for variational autoencoders.• We propose to use a prior which allows a decomposition of the latent space using independent subspace analysis (ISA) and demonstrate that this prior leads to disentangled representations even for the unmodified ELBO objective. This removes the trade-off between disentanglement and reconstruction loss of existing approaches.• An even higher disentanglement of the latent space can be achieved by incorporating the proposed prior distribution into the existing approaches β-VAE and β-TCVAE. Since the prior distribution already favours a disentangled representation, the new method dominates previous in terms of the trade-off between disentanglement and model quality. We presented a structured prior for unsupervised learning of disentangled representations in deep generative models. We choose the prior from the family of L p -nested symmetric distributions which enables the definition of a hierarchy of independent subspaces in the latent space. In contrast to the standard normal prior that is often used in training of deep generative models the proposed prior is not rotationally invariant and therefore enhances the interpretability of the latent space. We demonstrate in our experiments, that a combination of the proposed prior with existing approaches for unsupervised learning of disentangled representations allows a significant improvement of the trade-off between disentanglement and reconstruction loss. We vary l 0 between 4 and 10 and choose the same value for l 1 = l 2,...,l0 between 2 and 10. We set the parameter range of the exponents p i to p i ∈ [0.9, 2.4] with a discretization step size of 0.1, which includes lepto-and platokyrtic distributions. Fig. 2 depicts how lepto-and platykurtic distributions at the child subspaces lead to different representations of the x and y coordinate. Because the MIG metric evaluates axis-alignment of the latent dimensions to the underlying factors, here the x and y coordinate, platykurtic priors in general achieve a higher MIG score. The child subspaces share the same parameter p 1 = p 2,...,l0 and we choose the exponent of the root node as p 0 = p 1 to ensure independence of the subspaces. To study the influence of the layout on the reconstruction quality and MIG score we compare the results for different values of p 0 , p 1,...,5 and l 1 , and vary the value of β in the interval β ∈ [1, 4] with a step size of 0.5 and repeat each experiment four times. We compare four layouts with the highest MIG score for each subspace layout in FIG6 where we plot the mean and standard error of MIG score and reconstruction loss. For this dataset, the confguration p 0 = 2.1, p 1,...,5 = 2.2 and l 1 = 4 (denoted in black) is most appropriate as it achieves high MIG scores while maintaining a good reconstruction quality, both for the ISA-VAE and the ISA-TCVAE model. DISPLAYFORM0 2. For each inner node i of the tree associated with f , sample the auxiliary variable s i from a Dirichlet distribution Dir 5. Sample a new radiusṽ 0 from the radial distribution of the target radial distribution ψ 0 and obtain the sample viax =ṽ 0 · u 6. Multiply each entry x i ofx by and independent sample z i from the uniform distribution over {−1, 1}.
We present structured priors for unsupervised learning of disentangled representations in VAEs that significantly mitigate the trade-off between disentanglement and reconstruction loss. A general framework to use the family of L^p-nested distributions as the prior for the code vector of VAE, demonstrating a higher MIG. The authors point out issues in current VAE approaches and provide a new perspective on the tradeoff between reconstruction and orthogonalization for VAE, beta-VAE, and beta-TCVAE.