Learning disentangled representation from any unlabelled data is a non-trivial problem. In this paper we propose Information Maximising Autoencoder (InfoAE) where the encoder learns powerful disentangled representation through maximizing the mutual information between the representation and given information in an unsupervised fashion. We have evaluated our model on MNIST dataset and achieved approximately 98.9 % test accuracy while using complete unsupervised training. Learning disentangled representation from any unlabelled data is an active area of research ). Self supervised learning BID3 ; BID15 ; BID11 ) is a way to learn representation from the unlabelled data but the supervised signal is needed to be developed manually, which usually varies depending on the problem and the dataset. Generative Adversarial Neural Networks (GANs) BID4 ) is a potential candidate for learning disentangled representation from unlabelled data BID12 ; BID7 ; BID2 ). In particular, InfoGAN BID1 ), which is a slight modification of the GAN, can learn interpretable and disentangled representation in an unsupervised fashion. The classifier from this model can be reused for any intermediate task such as feature extraction but the representation learned by the classifier of the model is fully dependent on the generation of the model which is a major shortcoming. Because if the generator of the InfoGAN fails to generate any data manifold, the classifier is unable to perform well on any sample from that manifold. Tricks from Mutual Information Neural Estimation paper BID0 ) might help to capture the training data distribution, yet learning all the training data manifold using GAN is a challenge for the research community ). Adversarial autoencoder (AAE) BID10 ) is another successful model for learning disentangled representation. The encoder of the AAE learns representation directly from the training data but it does not utilize the sample generation power of the decoder for learning the representations. In this paper, we aim to address this challenge. We aim to build a model that utilizes both training data and the generated samples and thereby learns more accurate disentangled representation maximizing the mutual information between the random condition/information and representation space. InfoAE consists of an encoder E, a decoder D and a generator G. G network produces latent variable space from a random latent distribution and a given condition/information. D is used to generate samples from the latent variable space generated by the generator. It also maximizes the mutual information between the condition and the generated samples. E is forced to learn the mapping of the train samples to the latent variable space generated by the generator. The model has three other networks for regulating the whole learning process: a classifier C, a discriminator D i and a self critic S. FIG0 shows the architecture of the model. The encoding network E, takes any sample x ∈ p(x), where p(x) is the data distribution. E outputs latent variable z e = E(x), where z e ∈ q(z) and q(z) can be any continuous distribution learned by E. This z e is feed to decoder network D to get samplex r so thatx r ∈ p(x) andx r ≈ x. Generator network G generates latent variable z g = G(z, c) ∈ p(z), from any sample z and c where z ∈ u(z) and c ∈ Cat(c). Here p(z) can be any continuous distribution learned by G, u(z) is random continuous distribution (e.g., continuous uniform distribution) and Cat(c) is random categorical distribution. To validate z g , decoder D learns to generate samplex g = D(G(z g , c)) so thatx g ∈ p(x). The discriminator network D i forces decoder to create sample from the data distribution. While generator generates z g from z, it can easily ignore the given condition c. To maximise the Mutual Information (MI) between c and z g , we use classifier network C to classify z g intoĉ g = C(G(z g , c)) according to the given condition c. We also want encoder E to learn encodingẑ e = E(x g ) so thatẑ e ∈ p(z) and MI(ẑ e , c) is maximised. To ensure MI(ẑ e , c) is maximised again the classifier network C is utilised to classifyẑ e intoĉ e = C(ẑ e ) according to given condition c. To make sure q(z) ≈ p(z), we use a discriminator network S, which forces E to encode x into p(z). S learns through discriminating (x, z e ) as fake and (x g , z g ) as real sample. We named this discriminator as Self Critic as it criticises two generations from the sub networks of a single model where they are jointly trained. The InfoAE is trained based on multiple losses. The losses are : Reconstruction loss, R l = (x r − x) 2 for both Encoder and Decoder; Discriminator loss, DISPLAYFORM0 Decoder has loss D lg , for the generated imagex g and loss D le for the reconstructed imagex, where DISPLAYFORM1 Encoder loss E l = log(1 − S(z e , x)) ; Self Critic loss S l = log S(z g ,x g ) + log(1 − S(z e , x)) ; Two classification losses C lg , C le respectively for Generator and Encoder where C lg = − c log(ĉ g ) and C le = − c log(ĉ e ). We get our total loss, T l in equation 1 where α , β and γ are hyper parameters. DISPLAYFORM2 All the networks are trained together and the weights of the E, D, C, and G are updated to minimise the total loss, T l while the weights of the S and D i are updated to maximise the loss S l , D il , respectively. So the training objective can be express by the equation 2 DISPLAYFORM3 3 IMPLEMENTATION DETAILS Our model has different components as shown in FIG0 . We used Convolutional Neural Network (CNN) for E, D i and S. Batch Normalization BID6 ) is used except for the first and the last layer. We did not use any maxpool layer and the down sampling is done through increasing the stride. For classifier C and generator G we used simple two layers feedforward network with hidden layer. For Decoder D we used Transpose CNN.Our experiments show that the training of the whole model is highly sensitive to α, β and γ. After experimenting with different values of α, β and γ, we received best result for α = 1, β = 1 and γ = 0.4. For c variable we used random one hot encoding of size 10(c ∼ Cat(K = 10, p = 0.1)) and z ∈ R 100 , which is randomly sampled from a uniform distribution U (−1, 1). The weights of all the networks are updated with Adam Optimizer (Kingma & Ba FORMULA2 ) and the learning rate of 0.0002 is used for all of them. We have evaluated the model on MNIST dataset and received outstanding results. InfoAE is trained on MNIST training data without any labels. After trainning, We encoded the test data with Encoder, E and got classification label with the Classifier, C. Then we clustered the test data according to label and received classification accuracy of 98.9 (±.05), which is better than the popular methods as shown in TAB0 . InfoGAN BID1 ) 5 Adversarial Autoencoder BID10 4.10 (± 1.12) Convolutional CatGAN (Springenberg FORMULA2 4.27 PixelGAN Autoencoders BID9 ) 5.27 (± 1.81) InfoAE 1.1 (± .1)The latent variable produced by the encoder on test data is visualized in FIG1 . For visualization purpose we reduced the dimension of the latent vector with T-distributed Stochastic Neighbor Embedding or t-SNE (Van der Maaten & Hinton FORMULA3 ). In the visualization, we can observe that representation of similar digits are located nearby in the 2D space while different digits. This suggests that the encoder was able to disentangle the digits category in the representation space, which has eventually resulted in the superior performance. Also, the generator was able to generate latent space according to the condition and the decoder was able to generate samples from that latent variable space, disentangling the digit category as shown in FIG1 .Let us consider two latent variables z 1 = E(x 1 ) and z 2 = E(x 2 ) where x 1 , x 2 are two sample images from the test data. Now let us do a linear interpolation between z 1 toward z 2 with z 1 + (s/n) * (z 2 − z 1 ) where s ∈ {1, 2, ...., n} and n is the number of steps and feed the latent variables to the Decoder for generating sample. In this paper we present and validate InfoAE, which learns the disentangled representation in a completely unsupervised fashion while utilizing both training and generated samples. We tested InfoAE on MNIST dataset and achieved test accuracy of 98.9 (±.1), which is a very competitive performance compared to the best reported results including InfoGAN. We observe that the encoder is able to disentangle the digit category and styles in the representation space, which results in the superior performance. InfoAE can be used to learn representation from unlabelled dataset and the learning can be utilized in a related problem where limited labeled data is available. Moreover, its power of representation learning can be exploited for data augmentation. This research is currently in progress. We are currently attempting to mathematically explain the results. We are also aiming to analyze the performance of InfoAE on large scale audio and image datasets.
Learn disentangle representation in an unsupervised manner. The authors present a framework in which an auto encoder (E, D) is regularized such that its latent representation to share mutual information with a generated latent space representation.