What would be learned by variational autoencoder(VAE) and what influence the disentanglement of VAE? This paper tries to preliminarily address VAE's intrinsic dimension, real factor, disentanglement and indicator issues theoretically in the idealistic situation and implementation issue practically through noise modeling perspective in the realistic case.   On intrinsic dimension issue, due to information conservation, the idealistic VAE learns and only learns intrinsic factor dimension. Besides, suggested by mutual information separation property, the constraint induced by Gaussian prior to the VAE objective encourages the information sparsity in dimension. On disentanglement issue,   subsequently, inspired by information conservation theorem the clarification on disentanglement in this paper is made. On real factor issue, due to factor equivalence, the idealistic VAE possibly learns any factor set in the equivalence class.   On indicator issue, the behavior of current disentanglement metric is discussed, and several performance indicators regarding the disentanglement and generating influence are subsequently raised to evaluate the performance of VAE model and to supervise the used factors. On implementation issue, the experiments under noise modeling and constraints empirically testify the theoretical analysis and also show their own characteristic in pursuing disentanglement. Variational AutoEncoder(VAE)s BID9 , Rezende et al. (2014) ) have shown their powerful human-like abilities: modelling causal relationship, unsupervisedly extracting disentangled factors/representation BID1 ) and generating signals with abundant diversities in a "latent-factor-controllable" way. Those capabilities enable the knowledge transferring through shared causes/factors among different tasks/experiences, emphasized as the important human advantages against the current machine by BID12 and compling with the ideal mental imagery mechanism in memory and thinking. Benefitted from those capabilities, VAEs have been widely applied to various applications, including disentangled representations learning of images and time series BID6 , BID11 , Mathieu et al. (2016) , BID3 ), few-shot and transfer learning (Rezende et al. (2016) , BID8 , BID7 ), causal relationships modeling (Louizos et al. (2017) ), pixel trajectory predicting (Walker et al. (2016) ), joint multi-modal inference learning (Suzuki et al. (2016) ), increasing diversity in imitation learning (Wang et al. (2017) ), generation with memory BID16 ) and etc. However, the lack of public theoretical study regarding the generating and inference procedure induced by VAEs is tripping the research process: Idealistic VAE learns and only learns the intrinsic factor dimension. The illustration of the information conservation theorem 1. Suppose that the oracle data, denoted by random variable x, is generated by y (with P independent unit Gaussian random variables) with a homeomorphism mapping x = φ(y). Idealistic VAE will be forced to learn the factor z (with H independent unit Gaussian random variables) that generates the x with a homeomorphism mapping x = ψ(z). It yields z = ψ −1 • φ(y) and y = φ −1 • ψ(z). Then according to the information conservation theorem, it must hold that H = P . Gaussian-VAE (Kingma & Welling (2013) , Rezende et al. (2014) ) is an scalable unsupervised representation learning model BID6 ), and since Gaussian distribution can be continuously and reversibly mapping to many other distributions, the theoretical analysis on it is also instructive for other continuous latent factors VAE. DISPLAYFORM0 Gaussian-VAE assume that input x is generated by several independent Gaussian factors z, that is p θ (z) = N (z|0, I H ). The generating/decoding process is modeled as p θ (x|z) and the inference/encoding process q φ (z|x) is treated as the approximate posterior distribution. Both of them are parameterized by the neural network with parameter θ and φ. In VAE setting, the approximate inference method is applied to maximizing the variational lower bound of p θ (x) = p θ (x|z)p θ (z)dz, DISPLAYFORM0 log p θ (x|z) − D KL (q φ (z|x)||p θ (z)) ≤ log p θ (x).(1) In order to get assess to aforemention issues, we will start the analysis from the idealistic situation: An idealistic VAE model means that it can perfectly encode the signal into "used" factors and perfectly decode the "used" factors to original input signal and the factors follows i.i.d unit Gaussian distribution. The idealistic VAE discussed in this literature should also under the following the Deterministic Assumption on q φ (x|z) and p θ (z|x). If the factors of x are well understood, the generation process should be deterministic, that is p θ (x|z) = δ(x = ψ(z)). We limit the consideration that q φ (z|x) = δ(z = ψ −1 (x)) is also deterministic as well for simplicity of analysis in this paper. For more complex situation, this consideration could be also basic and instructive. We try to address aforemention issues by disregarding the training procedure and direct considering the idealistic VAE's behavior. In order to get asses to the intrinsic dimension issue, we will present the information conservation theorem. It states some basic truths, e.g. two independent Gaussian and three independent Gaussian cannot be the generating factor of each other under continuous mapping. The theorem thus further illustrates that idealistic VAE learns and only learns the intrinsic factor dimension. From the perspective of VAE objective, we will also show that the constraint induced by Gaussian prior, plays the lasso on the mutual information which encourage to clip down the small information dimension and promotes information sparsity in factors. In order to derive this perspective, the mutual information separation theorem and objective decomposition theorem are subsequently raised. Theorem 1 (Information Conservation). Suppose that z = (z 1 , · · · , z H ) and y = (y 1 , · · · , y P ) are sets of H and P (H = P ) independent unit Gaussian random variables, respectively, then these two sets of random variables can not be the generating factor of each other. That is, there are no continuous functions f : R H → R P and g : R P → R H such that z = g(y) and y = f (z).Proof in Appendix B. The principle of the theorem is visually illustrated in FIG0 . The mutual information regarding the factors learned by the inference/encoder network and the signal x can be a good quantity for evaluating the generating influence. That is, DISPLAYFORM0 In order to understand and estimate which factor of the VAE was learnt and influenced the generating process, I encoder (x; z h ) can be taken as a rational indicator 2 . If we assume that z 1 , z 2 , · · · , z H is conditional independent given x 3 , it can yield a useful result as the following. Theorem 2 (Mutual Information Separation). Let z 1 , · · · , z H be independent unit Gaussian distribution, and z 1 , z 2 , · · · , z H be conditional independent given x. Then DISPLAYFORM1 Proof in Appendix B. This theorem suggests that if the learnt q φ (z) can factorize and the q φ (z|x) can factorize, then the consideration of each I encoder (z h ; x) won't be excess or lose information. Besides, when those term comes in the optimization objective, then it can start the negotiation between the information preservation and dimension reduction and play the role of lasso that clip down the factor in dimension with small mutual information. Theorem 3 (Objective Decomposition). The terminology follows the aforemention definitions and if the involved KL-divergence and mutual information is well defined, then DISPLAYFORM2 Proof in Appendix B. The theorem demonstrates that the second term in variation lower bound in Eq. FORMULA12 is capable of controlling both the mutual information of x and z induced by the encoder network as well as the similarity of the learnt q φ (z) and the prior p θ (z). Further, the theorem suggests that it possesses the lasso capacity of clipping down the non-intrinsic factor dimension to some extent, visually demonstrated in Fig. (2). The sparsity of mutual information occurs;Ĩ encoder (x; z h ) determines the "used" factors; disentangled VAE pursues the intrinsic factors dimensions; generating factor exists equivalence class. Noise learning β-VAE (β = 10, equivalent σ 2 = 0.112):Ĩ encoder (x; z h ), σ 2 z h and qualitatively influential factor traversals. The top pulse subgraph:Ĩ encoder (x; z h ) of each factor. The bottom reverse pulse subgraph: the estimated variance σ 2 z h of each factor. The montages: influential factor traversals. We select those factor traversals with visually most interpretable/comprehensive effects to present and the whole influential factor traversals are listed in appendix 13. The phenomenon of the multiple semantic change induced by the same learnt factor and the encoding of same semantic among different learnt factor tallies with factor equivalence class theorem 4. The similar plot of its counterpart with specified normalized noise can be found in FIG0 in Appendix. Inspiration on Disentanglement: According to the information conservation theorem 1, the independent unit Gaussian factor assumption forms a strong inductive bias and facilitates the model incline to achieve most efficient coding. Under this assumption, the number of the learnt "used" factors of idealistic VAE should be the same as the true factors number under some assumptions such as the learnt q φ (z) should equal p θ (z) and decode/encode process is continuous and reversible. Empirically, sometimes, though the number latent of factors sometimes is pre-specified larger, only a small amount of unit Gaussian variables regarding the factors of VAE have been used while q φ (z|x) is close to deterministic. The theorem helps provide an interpretation to explain this phenomenon. Here, in order to avoid the ambiguity of the terminology of disentanglement in this paper, we make the following clarification. • The disentanglement of the learnt representation/factors in this literature refers to two parts depicted in Theorem 1: -the factors are closer to be independent with each other, -the factors incline to be able to generate the oracle signal and to be inferred perfectly from the oracle signal through a continuous procedure/mapping.• The "disentanglement" refers to the closeness of the learnt factors to the pre-specified independent factors/concpets that can generate the oracle signal and be perfect inferred through a continuous procedure/mapping such as the independent semantic/visual factors. Therefore, the estimation for D KL (q φ (z)||p θ (z)) that reflects the divergence of the learnt factor distribution and the i.i.d. unit Gaussian prior can be good a indicator to supervise the independence of the factors and served to quantitatively assess the disentanglement of each extracted factor, while the similarity regarding the original signal and reconstruction place another part of the disentanglement. The "disentanglement" will be shown hard to be obtained in an unsupervised manner. Concretely, even in the idealistic cases, the extracted factors tend to possess the intrinsic number of latent factors of the model, while there are still possibly large variations of these factors due to it can be obtained only in the equivalent class induced by the pre-specified factors as proved in the next section. As for real factor issue, Gaussian Factor Equivalence theorem, (i.e. linear orthogonal transformation of Gaussian factor set are still gaussian factor set.), and Linear Factor Equivalence Class will be presented. They states that idealistic VAE are possibly learns any factors set in the factors equivalence class, and we should not expect "one-to-one" correspondence by disentanglement. Theorem 4 (Gaussian Factor Equivalence). Suppose that z = (z 1 , · · · , z H ) is a set of H independent unit Gaussian random variables. Let Q ∈ R H×H be an orthogonal matrix and then y = Qz is also a set of H independent unit Gaussian random variables. Besides, z and y can generate each other through a linear homeomorphism mapping. Proof in Appendix B. This theorem implies that there are a class of unit Gaussian random variables which can generate each other and have equivalent conservation information, as indicated by the following theorem. Theorem 5 (Linear Gaussian Factor Equivalence Class).[z] = {y|y = Qz, Q ∈ R H×H be the orthogonal mapping.} Then ∀y ∈ [z], y is a set of H independent unit Gaussian random variables and can generate z through an linear homeomorphism mapping. The theorem clarifies that if Gaussian-VAEs have an linear matrix multiplication freedom degree of learning the factors, then the factors in the equivalence class can all be possibly learnt. The empirically results tally with the above analysis(see FIG2 ). Suppose the visual semantic concepts can be viewed as a set of independent Gaussian variables (z = 4 Notice this clarification is based on the assumption that q φ (z|x) is deterministic. If not, then continuous and reversible mapping of encoder constrain should be loosen and also the reversibility of the decoder should be loosen. If we further demand the enhancement of the pattern separation and completion ability, that is, to make the hyperspheres induced by the observation points in the factor domain fully fill up the whole compact factor ball, then some auxiliary constrains including the restriction on mutual information I encoder (x; z) (defined in Section 3.2) induced by the encoder network need to be introduced. It can be seen that changing one factor results in multiple semantic factor changes in a comprehensible manner rather than the "one-to-one" correspondence which reflexed analysis regarding generating factor equivalence.(z rotation , z gender , z with−glass , · · · ) T ) which are desired to be captured and learnt by VAEs, while the model is also possible to learn the independent factor set y = (y 1 , · · · ) T = Qz in the equivalence class [z] . This explains why changing one factor like y 1 sometimes empirically results in change in multiple visual concepts. This perspective suggests that it's actually hard to obtain the "disentangled representation" that exactly "one-to-one" corresponds to the "independent semantic representation" even though they are in the same equivalence class. As a result, the idealistic VAE model just tend to learn the "entangle representation" if we do preset "oracle generating factors" belonging to the equivalent class. However, though those conclusions might be upsetting, it seems not be biology impossible. A neuron in hippocampus of animals was suggested to combinatorially possess several representation capabilities. E.g., BID0 found that some neurons in rat's hippocampus involved in spatial representation also were involved in representing sound frequencies after training rats by a tasks that required them to use a joystick to manipulate sound in frequency continuously. 6.1 LIMITATION OF THE EXISTING DISENTANGLEMENT METIC BID6 proposed a "simulated factor" based "disentanglement" metric on the simulation datasets. However, according to Gaussian factor equivalence theorem 4 that even idealistic VAE will still learn the factors in the equivalence class, their metric could be effective sometimes for disentanglement but might suffer instability when evaluating the VAE in different trials (detailed in Appendix C).Further, this metric could be hardly calculated in the real datasets to provide direct feedback of the "disentanglement". The reason is that it must pre-know the generating factors expected to be learnt. In order to quantify the disentanglement performance 5 as well as the I encoder (x; z), we assume that q * (z) is a factorized zero mean Gaussian estimation for q φ (z).We can then list the indicators for assessing latent factor disentanglement: DISPLAYFORM0 Definition 2 (Estimation for I encoder (x; z)). DISPLAYFORM1 Definition 3 (Estimation for I encoder (x; z h ) which quantifies the influence of each factor). DISPLAYFORM2 Note that the above indicators 2-4 need the value of q * (z), we now introduce how to calculate this term based on Theorem 3. Through the minimization equivalence, we know that DISPLAYFORM3 the q * (z) can then be obtained by gradient method from solving the following optimization problem. DISPLAYFORM4 7 IMPLEMENTATION ISSUE In order to testify the idealistic consideration in real situation, we are not going to learn all the factors or equivalently we assume that datasets have noise 6 (this situation correspond to that p θ (x|z) is not deterministic since the major factors z only forms a subset of the whole factors. 7 ), we integrate the noise modeling into our model: DISPLAYFORM0 where σ 2 is either manually enumerated or adaptive learned. Noise modeling can be found crucial in influencing disentanglement in experiment since it would actually define the factors aimed to be learnt and subsequently influence the learnt intrinsic dimension suggested by information conservation property of VAE. The entangled representation can be caused by the over-large of searching space of q φ (z|x). If the learned q φ (z) = q φ (z|x)p data (x)dx has a big divergence to p θ (z), then the VAE model tends to learn the entangle representation as it violates the one part of the disentanglement (clarified in section 4). Actually, in the VAE model, what we want is to search in the space that BID6 , BID8 , BID13 , Mathieu et al. (2016) ) suggest it an effective way.6 Noise can be viewed as the generating factor that we are not interested in. 7 Notice the indeterministic p θ (x|z) could lead to the indeterministic p θ (z|x), but since the minor factors are supposed to have less influence on x, it could not bother the deduction using the knowledge we derived too much. q φ (z) is possibly similar to p θ (z) 8 . By implementing this ideal, we add auxiliary upper bound DISPLAYFORM0 to the original objective. This equivalently leads to the approach of β-VAE raised by BID6 . DISPLAYFORM1 where β > 1. Equivalent objective of σ 2 pre-specified Gaussian noise VAE: DISPLAYFORM2 Equivalent objective of σ 2 pre-specified as σ 2 pre Gaussian noise β-VAE: DISPLAYFORM3 where we call βσ 2 pre the normalized variance. It's shown that when the σ 2 is pre-specified, manually tuning it is the same as manually tuning β with a fixed σ 2 pre (for example σ 2 pre = 1) under Gaussian noise assumption. This equivalence saves our time for extra experiment studying the behaviors of this two cases and we call those two case noise specified β-VAE. MNIST is a database of handwritten digits BID14 ). CelebA BID17 ) is a largescale celebfaces attributes datasets and only its images are used in our experiments. More details are in Appendix D. The extensive comparison of Gaussian-noise modeling β-VAE and Gaussiannoise with specified variance (β)-VAE is made on MNIST based on our indicators to exploring the disentanglement as well as to testify the theorem and analysis from the idealistic case to the realistic sampling case. The experiments on CelebA will be auxiliary to further support the generating factor equivalence class theorem 9 .By setting β as different values, we compare the performance of β-VAE with and without prespecified noise on MNIST. We specifically listed the result of β(=1)-VAE in all cases. More details can be found in Appendix D.2. • Noise modeling/specification influence the disentanglement. The noise specifications and modeling significantly influence the model evidence quantitatively and reconstruction qualitatively, as clearly shown in FIG4 .The noise specifications significantly influence the divergence regarding q φ (z) and p θ (z) and noise specified and noise learning β-VAE achieve similar disentanglement quantitatively based on the similar indicator behaviours ofD KL (q φ (z)||p θ (z)) in • Auxiliary constraints can influence the disentanglement in a different way. The prominent difference induced by auxiliary constraints would be its stronger suppression oñ I encoder (x; z) and the number of influential factors although changing the noise level also possess this capability indirectly. It's somewhat obvious to see and compare value of different indictors under the different β setting by sliding on the green/blue line. The bigger β, the lowerD KL (q φ (z)||p θ (z)) and roughly the better reconstruction and hypothesis learnt. However, it's more interesting that in regard to the normalized variance, noise learning β-VAE enhances the suppression onĨ encoder (x; z), as depicted in FIG6 and that is comprehensible since β-VAE is minimizing the auxiliary constraints both I encoder (x; z) + D KL (q(z)||p(z)) based on Theorem 3. •Ĩ encoder (x; z h ) effectively determines the "used" factors and VAEs incline to learn the intrinsic factor dimension in realistic sampling case when the disentanglement achieves in some extent. As shown in FIG1 and FORMULA6 , the indicatorĨ encoder (x; z h ) determines the "used" factors. According to those figures, under the suitable noise assumption, VAEs automatically suppress the auxiliary factors and learn the intrinsic factor dimension as it still capable to have good reconstruct abilities and its D KL (q φ (z)||p θ (z)) closer to zero. This phenomenon has already suggested by the information conservation theorem 1. • Factor equivalence is generally hold and VAEs possibly learn any factor set in the equivalence class. The reflections of the generating equivalence properties 5.1, that is, single factor could result in multiple semantic concepts change and same semantic concept could be encoded in different factors, are again well demonstrated by FIG11 , FIG1 , FIG0 and TAB1 . From the perspective of representation learning:• It is interesting that the topology properties of oracle signal are used to obtain the proof for the information conservation theorem. Other situations including that data owns several connected components can be further considered and would uncover the efficient coding properties of discrete factors.• When q φ (z|x) is far from deterministic, the discussion would be crucial for many other general purposes induced by word disentanglement. Those study may further extend to the case that data containing different dimension manifolds and to the core VAEs' pattern separation/completion/generalization capabilities. Thanks for Haodong Sun( Georgia Tech)'s twice greeting me after this paper being rejected. Here is the acknowledgment to memorize our relationship. ;-) The VAE objective (i.e., the variational lower bound) can be treated as function of the θ and φ where noise variance parameters σ 2 are contained in θ, DISPLAYFORM0 Here the SGVB estimator in BID9 DISPLAYFORM1 ) is used. Note that the noise variance σ 2 is also taken as an optimization variable in the model, making the model capable of better adapting noise variation of data in practical cases in a totally automatic way, instead of a manually set manner. Given multiple data points from a dataset X, we can construct an estimator of the mean marginal likelihood lower bound of the full dataset, based on minibatches DISPLAYFORM2 where the minibatch DISPLAYFORM3 is a randomly drawn sample set of M datapoints from the full dataset X. Such a lower bound also constitutes an important indicator for model evidence in latter experiment. We call it the empirical variational lower bound (EVLB) in the following. B (θ, φ, x m ) L(θ, φ, x m ) and we can deduce that DISPLAYFORM0 The last inequality holds due to DISPLAYFORM1 The alternative optimization strategy can be readily utilized to design the algorithm for solving the model by iteratively updating the noise parameter and the network ones. During the optimization process, the objective can be monotonically increasing, and thus the algorithm can be guaranteed to be convergent. The algorithm is summarized as follows:Optimization for parameters except σ 2 :Optimization for DISPLAYFORM2 Direct gradient method to the transformed variable log simga = log σ ∈ R can be implemented to lift the lower boundL M as a result to increase the likelihood as well. The noise ε in real situation might be more complex than a simple Gaussian, like that existed in real photographs (Plotz & Roth (2017) ). We thus try to further ameliorate the noise setting as a mixture of Gaussian(MoG) noise. Such noise modeling strategy has been widely verified to be effective in applications, like matrix factorization (Meng & Torre FORMULA2 ) and robust principal component analysis (Zhao et al. (2014) ). That is, we assume that DISPLAYFORM0 Let c d ∈ {0, 1} K be the latent indicator random one-hot variable, DISPLAYFORM1 The posterior distribution q φ (z, c|x) can be factorized as q φ (z|x)q(c|x, z), where q φ (z|x) will be direct learnt and the alternative of q(c|x, z), q(c|x, e) will be set to the last step p θ (c|x, e) in regard to EM procedure. The lower bound of log p θ (x) is then reformulated as follows: DISPLAYFORM2 Similar to the Gaussian case, the reparamerization trick is implemented, DISPLAYFORM3 DISPLAYFORM4 By utilizing the SGVB estimator, we get, DISPLAYFORM5 Given an input dataset X, we can then construct an estimator to the mean marginal likelihood lower bound of the full dataset, based on minibatches, as follows: DISPLAYFORM6 where DISPLAYFORM7 is a randomly drawn sample of M datapoints from the full dataset X. where z old = En old (x) + Σ old z|x 1/2 (x)e, and we can get DISPLAYFORM0 The EM algorithm can be naturally employed to solve the model. The implementation steps are listed as follows:Step 1. Expectation Step. DISPLAYFORM1 Calculate the expectation of the latent variable c: DISPLAYFORM2 where DISPLAYFORM3 Step is obtained as the following, DISPLAYFORM0 Step 2. Maximization Step:Fix: q(c|x, e) determined in the Expectation Step. DISPLAYFORM1 Update [Π, Σ] and {θ, φ}/[Π, Σ] by alternative optimization strategy. Update Π, Σ: note here z m,l : z DISPLAYFORM2 , and we can easily get the closed-form updating formula for these parameters: DISPLAYFORM3 Update {θ, φ}/[Π, Σ]: gradient methods with respect to {θ, DISPLAYFORM4 The algorithm can then be summarized as follows:1. Initialize the coefficient of {θ, φ}/[Π, Σ] and the coefficient of noise ε: Π,Σ.2. Sample e from N (0, I H ) to obtain e 1 , · · · , e M [One for each element sample in the mini batch in the next step (L here is set to 1)].3. Sample a mini batch X M from p data (x). Proof. For theorem 1. Proof by Contradiction. Suppose those two function exist, and we will show that they will be inverse mapping of each other and the homeomorphism mapping of R H and R P . Since R H and R P have different topology structures (P = H), the homeomorphism mapping will not exist. DISPLAYFORM0 Since both f and g are continuous, there is a homeomorphism mapping between R H and R P and it leads to the contradiction. Proof. For theorem 4. We only need to test the mean and variance of y. DISPLAYFORM1 Therefore, y is another set of H independent unit Gaussian random variables. Since x = Q T y, z and y can generate each other with an linear homeomorphism mapping. Proof. For theorem 2, DISPLAYFORM2 Proof. For theorem 3. DISPLAYFORM3 Corollary 1. The terminology follows the aforemention definitions and if the involved KLdivergence and mutual information be well defined then DISPLAYFORM0 The proof of corollary 1 is the same as that of theorem 3. This corollary suggests that the estimation in definition 2 provides another upper bound for the capacity of the encoder network. Empirically, this estimation is a much tighter estimation than using the estimation in definition 1.Corollary 2. The terminology follows the aforemention definitions and if the involved KLdivergence and mutual information be well defined then DISPLAYFORM1 The corollary is an direct result of theorem 3 and corollary 1. It suggests that the estimation in definition 4 is a lower bound for D KL (q φ (z)||p θ (z)). DISPLAYFORM2 DISPLAYFORM3 shown the same estimation results on M-NIST.Definition 6 (Another estimation for I encoder (x; z)). DISPLAYFORM4 Definition 7 (Another estimation for I encoder (x; z h ) which quantifies the influence of each factor). DISPLAYFORM5 C APPENDIX: ANALYSIS ON THE "DISENTANGLEMENT" METRIC RAISED IN β-VAE (HIGGINS ET AL. FORMULA2 The terminology inherits those in the β-VAE paper. The main idea of that "disentanglement" metric is to create a statistic point z dif f relevant to the model for each simulated factor respectively and then to use a linear classifier to project the statistic point to the corresponding index of the simulated factor. If the statistic points induced by the model are easy to be separated then the model is thought to learn "disentangled" representation. Here, we will argue that even for the idealistic VAE model that follows the disentanglement conditons 4 could still receive bad score under that performance metric in some situations. Suppose that the true simulated factors v follows N (0, I H ). Then the learnt "used" 10 factor z can be in the equivalence class [v] according to theorem 4. Concretely, there exists an orthogonal transformation Q such that z = Qv. y−f ixed . In order to calculate the statistic point z dif f (y) = E |z 1 − z 2 |, we first calculate the mean and variance of (z 1 − z 2 ). DISPLAYFORM6 Therefore, z dif f (y) can be obtained through the diagonal value of 2I − 2q y q T y . That is, DISPLAYFORM7 From the above equation, the location of statistic point is unique determined by (q DISPLAYFORM8 is close to the vertex of the unit cubic for each y then all the statistic points turn to be easily separated. However, from the perspective of the objective, all the orthogonal Qs are with the same potential to be learnt. It seems not to be with a small probability that statistic points of different indexes take similar location. For instance, if H = 2 and Q = DISPLAYFORM9 cannot be separated while the representation still follows the disentanglement conditions. Among different trials, the Q might contribute to that "disentanglement" metric but also might not. That explains why that metric is unstable. We set L to 1, and minibatch size M to be 100 in all practices. All the pixels value have been linear normalized in to [0, 1] . We find in the last version that the code on calculating q * (z) is wrong. Concretely, the objective of the KL-divergence of two Gaussian is incorrect calculated. That influences the estimation of z H but has little influence on the determination of the "used" factors. So we redo all the experiments on MNIST and delete the relevant results regarding those wrong-calculated indictors on CelebA and Extended Yale Face B.The new experiment results solve many our past confusions due to the wrong experiment. They are DISPLAYFORM0 is strongly correlated withĨ(x; z h )? • Why σ 2 z h of the used factor is always relative small?• Why can VAE still learn the "disentangled" representation when the learntD KL (q φ (z)||p θ (z)) is such big?Now we know that they are actually not the cases. Something better is that we find our experiment results are more close to our theoretical analysis: when guaranteeing the reconstruction quality in a tolerance range, the smallerD KL (q φ (z)||p θ (z)) the less "used" factors are learnt. That coincides with information conservation theorem: the independent unit Gaussian of the factors assumption facilitates the most efficient coding. We split randomly 7000 datapoints according to ratio [0.6 : 0.2 : 0.2] into training set, validation set (no use), testing set. All the indicators and q * (z) are evaluated/calculated on 10000 datapoints belonging to the testing set. All the seed images used to infer latent code and to draw the traversal come from the testing set. In all figures of latent code traversal each block corresponds to the traversal of a single latent variable while keeping others fixed to either their inferred ( β-VAE, VAE). Each row represents a different seed image used to infer the latent values in the VAE-based models. β-VAE and VAE traversal is over the [−3, 3] range. The assumed variance σ 2 of noise specified Gaussian of VAE models is enumerated from We split randomly 2424 datapoints according to ratio [0.8 : 0.1 : 0.1] into training set, validation set (no use), testing set. The model is training on the training set. All the seed images used to infer latent code and to draw the traversal come from the 100 datapoints from the testing set. In all figures of latent code traversal each block corresponds to the traversal of a single latent variable while keeping others fixed to either their inferred ( β-VAE, VAE). Each row represents a different seed image used to infer the latent values in the VAE-based models.β-VAE and VAE traversal is over the [−3, 3] range. The β setting for the noise learning β-VAE is enumerated from [1, 40, 80, 120, 160] . We split randomly roughly 200000 datapoints according to ratio [0.8 : 0.1 : 0.1] into training set, validation set (no use), testing set. The model is training on the training set. All the indicators and q * (z) are evaluated/calculated on 10000 datapoints selected from testing set. All the seed images used to infer latent code and to draw the traversal come from the 100 datapoints from the testing set. In all figures of latent code traversal each block corresponds to the traversal of a single latent variable while keeping others fixed to either their inferred ( β-VAE, VAE). Each row represents a different seed image used to infer the latent values in the VAE-based models.β-VAE and VAE traversal is over the [−3, 3] range. The β setting for the noise learning β-VAE is enumerated from [1, 30, 40] . BID9 and Rezende et al. (2014) to implement the efficient learning and inference in directed probabilistic models regarding continuous latent variables with intractable posterior distributions and in scalable datasets. They introduced a network inference/recoginition model to represent the approximate posterior distribution and utilized reparameterization trick for stochastic joint optimization of a variational lower bound containing the parameters of both the generative/decoder and inference/recoginition/encoder models. After being raised, many VAE variations have been proposed to boost VAE's capabilities in generation quality and/or disentanglement of the learned representation. In these methods, multiple efforts were made by improving the generative and inference network structures. Typical works along this line include the convolution/de-convolution structure raised by BID11 and ladder structure raised by Zhao et al. FORMULA2 ). Some other works advanced the mechanism under the VAE generation/inference processes. Typical works include the iterative attention generation/inference mechanism raised by BID5 , normalizing flow proposed by Rezende & Mohamed (2015) that enhanced the expressive ability of the approximate posterior and its variants BID10 ).Despite the improvement to the VAE itself, some other efforts were made by the ensemble between GAN with VAE. E.g., BID13 unified GAN and VAE to obtain a better reconstruction and a high-level abstracts visual features embedding. Mathieu et al. (2016) also unified GAN and VAE but put emphasis on disentangling factors of variation. GANs without auxiliary design would learn the data distribution disregarding its noise level though suffer from unstable training and mode collapsing BID10 ) while VAEs would assume a decomposition of the noise and oracle clean datapoint regarding the noise data with an auxiliary prior on the distribution regarding the factors. Besides, many efforts were made by regularization on the factor distribution or factor generating effect. E.g., Makhzani et al. (2015) introduced an adversarial loss into the latent space of the autoencoder which in idealistic case could learn any kind factor/lantent distribution including those contributing to the disentangled factors/representation. InfoGAN, raised by , introduced the infomax principle to GAN by adding an auxiliary mutual information regularization which enabled the inference of GANs and led to a better disentangled representation as well. Recently, there is a new VAE variation is proposed by BID6 who introduced the β-VAE framework which enhanced the constraints regarding the KL-divergence of the posterior and prior distribution of VAE and showd a novel disentanglement performance. This method has obtained a better performance as compared with conventional VAE methods, especially on its flexible tuning a compromising a parameter beta between the KL-divergence term and the likelihood term (the variational lower bound). and qualitatively influential factor traversals. The mutual information of "used" factor learnt by noise specified β-VAE can be found more diverse than that in figure 2.
This paper tries to preliminarily address the disentanglement theoretically in the idealistic situation and practically through noise modelling perspective in the realistic case. Studies the importance of the noise modelling in Gaussian VAE and proposes to train the noise using Empirical-Bayes like fashion. Modifying how noise factors are treated when developing VAE models