
We study the problem of building models that disentangle independent factors of variation. Such models encode features that can efficiently be used for classification and to transfer attributes between different images in image synthesis. As data we use a weakly labeled training set, where labels indicate what single factor has changed between two data samples, although the relative value of the change is unknown. This labeling is of particular interest as it may be readily available without annotation costs. We introduce an autoencoder model and train it through constraints on image pairs and triplets. We show the role of feature dimensionality and adversarial training theoretically and experimentally. We formally prove the existence of the reference ambiguity, which is inherently present in the disentangling task when weakly labeled data is used. The numerical value of a factor has different meaning in different reference frames. When the reference depends on other factors, transferring that factor becomes ambiguous. We demonstrate experimentally that the proposed model can successfully transfer attributes on several datasets, but show also cases when the reference ambiguity occurs.
 One way to simplify the problem of classifying or regressing attributes of interest from data is to build an intermediate representation, a feature, where the information about the attributes is better separated than in the input data. Better separation means that some entries of the feature vary only with respect to one and only one attribute. In this way, classifiers and regressors would not need to build invariance to many nuisance attributes. Instead, they could devote more capacity to discriminating the attributes of interest, and possibly achieve better performance. We call this task disentangling factors of variation, and we identify attributes with the factors. In addition to facilitating classification and regression, this task is beneficial to image synthesis. One could build a model to render images, where each input varies only one attribute of the output, and to transfer attributes between images.When labeling is possible and available, supervised learning can be used to solve this task. In general, however, some attributes may not be easily quantifiable (e.g., style). Therefore, we consider using weak labeling, where we only know what attribute has changed between two images, although we do not know by how much. This type of labeling may be readily available in many cases without manual annotation. For example, image pairs from a stereo system are automatically labeled with a viewpoint change, albeit unknown. A practical model that can learn from these labels is an encoder-decoder pair subject to a reconstruction constraint. In this model the weak labels can be used to define similarities between subsets of the feature obtained from two input images.We introduce a novel adversarial training of autoencoders to solve the disentangling task when only weak labels are available. Compared to previous methods, our discriminator is not conditioned on class labels, but takes image pairs as inputs. This way the number of parameters can be kept constant.We describe the shortcut problem, where all the the information is encoded only in one part of the feature, while other part is completely ignored, as FIG0 illustrates. We prove our method solves this problem and demonstrate it experimentally.We formally prove existence of the reference ambiguity, that is inherently present in the disentangling task when weak labels are used. Thus no algorithm can provably learn disentangling. As FIG0 shows, the reference ambiguity means that a factor (for example viewpoint) can have different meaning when using a different reference frame that depends on another factor (for example car type). We show experimentally that this ambiguity rarely arise, we can observe it only when the data is complex. In this paper we studied the challenges of disentangling factors of variation, mainly the shortcut problem and the reference ambiguity. The shortcut problem occurs when all information is stored in only one feature chunk, while the other is ignored. The reference ambiguity means that the reference in which a factor is interpreted, may depend on other factors. This makes the attribute transfer ambiguous. We introduced a novel training of autoencoders to solve disentangling using image triplets. We showed theoretically and experimentally how to keep the shortcut problem under control through adversarial training, and enable to use large feature dimensions. We proved that the reference ambiguity is inherently present in the disentangling task when weak labels are used. Most importantly this can be stated independently of the learning algorithm. We demonstrated that training and transfer of factors of variation may not be guaranteed. However, in practice we observe that our trained model works well on many datasets and exhibits good generalization capabilities.
It is a mostly theoretical paper that describes the challenges in disentangling factors of variation, using autoencoders and GAN. This paper considers disentangling factors of variation in images, shows that in general, without further assumptions, one cannot tell apart two different variation factors, and suggests a novel AE+GAN architecture to try and disentangle variation factors. This paper studies the challenges of disentangling independent factors of variation under weakly labeled data and introduces the term reference ambiguity for data point mapping.