Latent Space Factorisation and Manipulation via Matrix Subspace Projection
Xiao Li Chenghua Lin Ruizhe Li Chaozheng Wang Frank Guerin
cs.LG stat.ML


[
Latent Space Factorisation and Manipulation via Matrix Subspace Projection
















equal*


Xiao Lito,goo
Chenghua Lingoo
Ruizhe Ligoo
Chaozheng Wangto
Frank Guerined


toDepartment of Computing Science, University of Aberdeen, UK
gooDepartment of Computer Science, University of Sheffield, UK
edDepartment of Computer Science, University of Surrey, UK

Chenghua Linc.lin@sheffield.ac.uk





MSP, generative models, synthesis, disentanglement, facial attribute editing,
attribute style manipulation

0.3in
]









  



We tackle the problem disentangling the latent space of an autoencoder in order to separate labelled attribute information from other characteristic information. This then allows us to change selected attributes while preserving other information. Our method, matrix subspace projection, is much simpler than previous approaches to latent space factorisation, for  example not requiring multiple discriminators or a careful weighting among their loss functions.
Furthermore our new model can be applied to autoencoders as a plugin, and works across diverse domains such as images or text. We demonstrate the utility of our method for attribute manipulation in autoencoders trained across varied domains, using both human evaluation and automated methods. The quality of generation of our new model (e.g. reconstruction, conditional generation) is highly competitive to a number of strong baselines. 






§ INTRODUCTION



We investigate the problem of manipulating multiple attributes of  data samples. This can be applied to image data, for example to manipulate a picture of a face to add a beard, change gender, or age. It can also be applied to text, for example to change the style or sentiment of a text. We assume that we have a training dataset where attributes are labelled. However there is an unsupervised aspect because we do not have samples of the same individual with different attribute combinations, e.g., the same person with and without a beard. Furthermore the training samples have some attribute combinations that are highly correlated, while other combinations are completely absent; e.g., in the CelebA dataset blond hair and earrings are highly correlated with female <cit.>, while a female with beard is absent. Nevertheless we would like our system to somehow isolate the explanatory factors in pixel space, to understand, e.g., that blond hair corresponds only to colour changes to hair pixels, and no change elsewhere in the face.


This challenge of isolating multiple explanatory factors poses interesting problems for generative models.
In images of faces for example, even if the training data has no bearded lady, a good generative model should be able to `imagine' novel examples that combine attributes in ways not present in training data.
As noted by <cit.> “Models are unable to generalise to data outside of the convex hull of the training
distribution …unless they learn about the data generative factors and recombine them
in novel ways.” 
Ideally we should fully disentangle and isolate the data generative factors, so that we can represent the generative factors of a sample with a vector that has one part labelled attribute information, and another part with the other characteristic information of the sample. 
This is in a small way part of a general trend  to try to move deep neural network research towards explanatory models of the world <cit.>, which requires disentanglement.
The problem is important because isolating explanatory factors is a way to overcome the combinatorial explosion  of required training examples if such factors are not isolated <cit.>.


A typical approach to the problem uses an autoencoder (AE) which encodes a given input (e.g. picture, text, etc.) into a  latent vector, and then restores (decodes) the latent vector to the given input <cit.>. The latent vector contains the attribute information as well as other characteristic information of the  input. If one can change the attribute information in the latent space, then one can generate examples with the altered attributes. The difficulty here is twofold: (1) learn a latent space representation which separates the attributes from all other characteristic information, and (2) fully disentangle the attributes. If we fail in the separation part, then efforts to generate with specific attributes may conflict with other information in the latent space (as in <cit.> etc., see <ref>). If we fail in the second part then  examples generated with specified attributes will also be contaminated with spurious attributes (see Fig. <ref> Left).


Many recent approaches make use of auxiliary neural network structures with adversarial training in the style of
Generative Adversarial Networks (GANs). These new networks can be used to remove attribute information from the latent space <cit.>, or to feedback a loss term to impose the attributes they want to appear in the output <cit.>.
These adversarial approaches have competing loss terms (for example reconstruction loss, attribute classification loss, realistic output loss), and require a careful choice of hyperparameters to weight these loss functions. In the case of <cit.>  a slowly increasing loss was critical. These hyperparameters and training schedules must be determined by trial and error, to avoid training instability. Even after successful training we have found that some models ignore the desired attributes and put too much weight on reconstruction and realistic output (see <ref>). This is partly because we push systems to the very difficult setting of training for multiple attributes together (e.g. 40 attributes for CelebA). This is a very demanding setting for disentanglement, e.g. to dissociate lipstick, make-up, and blond hair from female, and to dissociate beard, bushy eyebrows, and 5 o'clock shadow
from male.




We propose a simple and generic method, Matrix Subspace Projection (MSP), which directly separates the attribute information from all other non-attribute information, without relying on weighting loss terms from auxiliary neural networks. 
Our variables representing attributes  are fully disentangled, with one isolated variable for each attribute of the training set.
Therefore, when we do  conditional generation, we can assign  pure attributes combined with other latent data which does not conflict, so that the generated pictures are of high quality and not
contaminated with spurious attributes.
Meanwhile, our model is a universal plugin. In theory, it can be applied to any existing AEs (if and only if the AEs use a latent vector). If the AE is a generative model (such as VAE), with our approach, it becomes a conditional generative model that generates content based on the given condition constraints.
In the case of images, we add a PatchGAN at the end of our generator to sharpen the image, but this is not connected with the attribute manipulation task and is not core to our model; it could be replaced with any super resolution and sharpening method.



Our plugin has two uses: (1)   samples can be  generated from a random seed, but with given attributes; (2) a given sample can be modified to have desired  specified attributes. Our key contributions are:
(1) A simple and universal plugin for conditional generation and content replacement, directly applicable to any AE architectures (e.g., image or text). (2) Strong performance on learning disentangled latent representations of multiple (e.g. 40) attributes. (3) A principled weighting strategy for combining loss terms for training. The code for our model is available online[Code: <https://xiao.ac/proj/msp>].




§ RELATED WORK




The first approaches to control of generation by attributes (conditional VAEs <cit.>) simply added attribute  information as an extra input to the encoder or the decoder. These approaches generate using a latent vector 𝐳 and also an attribute vector 𝐲, where the 𝐳 often conflicts with 𝐲, because attribute information has not been removed from 𝐳.
With conflicting inputs the best the VAE can do is to produce a blurry image. 

Generative Adversarial Networks (GANs) can be augmented with encoders. IcGAN  trains separate encoders for the 𝐲 and 𝐳 vectors, but does not try to remove potentially conflicting information <cit.>. The IcGAN authors also note that it can fail to generate unusual attribute combinations such as a woman with a moustache, because the GAN discriminator  discourages the generator from generating samples outside the training distribution.
  
  
  
More recent work tackled the problem of separating the attribute information from the latent vector, using a new auxiliary network  (like a GAN  discriminator)  <cit.>,  which attempts to guess the attribute of the latent vector 𝐳, and penalise the generator if attribute information remains. A significant drawback of these  adversarial approaches is that great care must be taken in training so that the loss from the discriminator (which is trying to remove attribute information) does not disturb the training to produce a good reconstruction. In the case of 
Fader networks <cit.> it was necessary to start with a discriminator loss weight of zero, and linearly increase to 0.0001 over the first 500,000 iterations; the authors state “This scheduling turned out to be critical in our experiments.
Without it, we observed that the encoder was too affected by the loss coming from the discriminator, even for low values of [loss coefficient].”

While this adversarial approach can successfully remove attribute information from 𝐳, there is nothing to stop the decoder (generator) from associating other spurious information with the attribute. For example the decoder might associate the attribute intended to be for `glasses' with an older or more masculine face. This is what we see in the results of two of the adversarial approaches (see Fig. <ref>). 
Most of the results in <cit.> focus on the attribute `smiling' (not reproduced here), and this is very well disentangled. It is only when the training dataset associates other attributes with the trained attribute that entanglement will arise.
In <cit.> the attribute vector is a single binary variable so that the system can only be trained to control (or classify) one attribute. 
It is not unexpected that a generator will associate spurious information with an attribute if the association is present in the training data and the system has  been trained only on examples labelling a single attribute, e.g., glasses. 
 The system cannot know that it should isolate `wearing glasses', and not `wearing glasses and older'.
Fader Networks  <cit.>  can train for multiple attributes together, however
<cit.> state that “Although
Fader Networks is capable for multiple attribute editing
with one model, in practice, multiple attribute setting makes
the results blurry.”



The most recent works (2018-19) are GAN-based. They do not try to remove attribute information from the latent space, but instead add an additional attribute classifier after generation, and impose an attribute classification loss. This is in addition to a typical GAN discriminator for realistic images.
AttGAN  <cit.> uses an endoder, decoder (generator), and the attribute classifier and discriminator applied to the output of the generator.
StarGAN  <cit.> and RelGAN <cit.> use no encoder, but use a singe generator twice, in a cycle; the first direction alters attributes like a conditional GAN, the second one attempts to reconstruct the image (using original attributes), and so requires that non-attribute information has been preserved. StarGAN uses a  discriminator and attribute classifier, like AttGAN, while 
RelGAN adds a third network for interpolation.

All the works cited from 2017 to 2019 have an  adversarial component (in the style of a GAN); they train auxiliary classifiers to feed back loss terms, to ensure they remove undesirable attributes, or enforce desired ones. They need a careful weighting among loss terms, but there is no principled method for determining these weighting hyperparameters. Our work does not rely on an adversarial component to manipulate attributes; we use a more direct method of matrix projection onto subspaces, in order to factorise the latent representation and separate attributes from other information. 
Furthermore, unlike the above works[Not mentioned in the Fader networks paper, but in the published code: https://github.com/facebookresearch/FaderNetworks]
 we do not use any skip connections. Skip connections can introduce errors when a region of the source and target image is quite different, we illustrate this further in Fig. <ref> Right.





In addition to the above works using labelled attributes there is also work on the more difficult problem of unsupervised learning of disentangled generative factors of data
<cit.>.
However the supervised (labelled) approaches generate much clearer samples of selected attributes, and superior disentanglement. 
An alternative approach to controlled generation is to simply train a deep convolutional network and do linear interpolation in deep feature space <cit.>. This shows surprisingly good results, but in changing an attribute that should only affect a local area it can affect more image regions, and can produce unrealistic results for more rare face poses. 







§ METHOD





 §.§ Problem Formulation
 
We are interested in factorising and manipulating multiple attributes from a latent representation learned by an arbitrary Autoencoder (AE). Suppose we are given a dataset 𝒟 of elements (𝐱, 𝐲) with 𝐱∈ℝ^n and 𝐲∈ Y = { 0, 1}^k representing k attributes of 𝐱. 

Let an arbitrary AE be represented by 𝐳=F(𝐱) and 𝐱'=G(𝐳), where F(·) is the encoder, G(·) is the decoder, 𝐳 is the latent vector encoding 𝐱, and 𝐱' is the reconstruction of 𝐱 (see Fig. <ref>). Note that when 𝐱' is a good approximation of 𝐱 (i.e., 𝐱'≈𝐱), the attribute information of 𝐱 represented in 𝐲 will also be captured in the latent encoding 𝐳. 

Attribute manipulation means that we 
replace the attributes 𝐲 captured by 𝐳 with new attributes 𝐲_𝐧. Let K(·) be a replacement function, then we have new latent space 𝐳_𝐧 = K(𝐳, 𝐲_𝐧) and 𝐱_𝐧=G(K(𝐳, 𝐲_𝐧)), where the attribute information encoded in 𝐲_𝐧 can be predicted from 𝐱_𝐧 and the non-attribute information of  𝐱 will be preserved. To give a concrete example, given an image of a face, 𝐱, we wish to manipulate 𝐱 w.r.t. the presence or absence of a set of desired attributes encoded in 𝐲_𝐧 (e.g., a face with or without smiles, wearing or not wearing glasses), producing the manipulated image 𝐱_𝐧, without changing the identify of the face (i.e., preserving the non-attribute information of 𝐱). 




 §.§ Learning Disentangled Latent Representations via Matrix Subspace Projection



To tackle the problem formulated in <ref>, we propose a generic method to factor out the information about attributes 𝐲 from 𝐳 based on the idea of performing orthogonal matrix projection onto subspaces.  Our model works as a universal plugin and in theory, it can be applied to any existing AEs. The general architecture of the proposed MSP model is depicted in Fig. <ref> (a). 
Given a latent vector 𝐳 encoding 𝐱 and an arbitrarily complex invertible function H(·), H(·) transforms 𝐳 to a new linear space (ẑ=H(𝐳)) such that one can find a matrix 𝐌 where  
(a) the projection of ẑ on 𝐌 (denoted by ŷ) approaches  𝐲 (i.e., ŷ captures attribute information), 

    𝐌·ẑ=ŷ;   ŷ→𝐲

and (b) there is an orthogonal matrix 𝐔≡  [𝐌;𝐍], where 𝐍 is the null space of 𝐌 (i.e., 𝐌⊥𝐍) and the projection of ẑ on 𝐍 (denoted by ŝ) captures non-attribute information. 
As 𝐔 is orthogonal, we also have 𝐔^T ≡𝐔^-1. 


Fig. <ref> (b) presents a simplified architecture of our MSP model, which is equivalent to the general architecture. This simplification exists because as explained earlier H(·) is invertible. So  when the encoder and decoder of an AE have enough capacity, they can essentially absorb H and H^-1. In other words, rather than fitting F and G, the encoder and decoder will fit H· F(·) and G· H^-1(·) instead. As 𝐌 itself is an incomplete orthogonal matrix, similar operations cannot be applied to 𝐌. 



Our main learning objective, in addition to the original AE objective (i.e. reconstruction loss ℒ_AE; see <ref> and Eq. <ref>), is then to estimate 𝐌 which is nontrivial. We turn the problem of finding an optimal solution for 𝐌 into an optimisation problem, in which we need to (1) enforce ŷ to be as close to 𝐲 (i.e., the vector encoding the ground truth attributes) as possible; and (2) minimise ||ŝ||^2  so that ŝ contains as little information from ẑ as possible. This can be formulated into the following loss function 

    ℒ_𝑀𝑆𝑃    =  ℒ_1+ℒ_2 
    ℒ_1     = ||ŷ-𝐲||^2 = ||𝐌·ẑ-𝐲||^2 
    ℒ_2     = ||ŝ||^2

Here ℒ_1 and ℒ_2 encode the above two constraints, respectively, and ŷ is the predicted attributes. 
Given that the AE relies on the information of ẑ to reconstruct 𝐱, the optimisation constraints of ℒ_AE and ℒ_2 essentially introduce an adversarial process: on the one hand, it discourages any information of ẑ to be stored in ŝ due to the penalty from ℒ_2; on the other hand, the AE requires information from ẑ to reconstruct 𝐱.
So, the best solution is to only restore the essential information for reconstruction (except the attribute information) in ŝ. 
By optimising ℒ_𝑀𝑆𝑃, we cause ẑ to be factorised, with the attribute information stored in ŷ, while ŝ only retains non-attribute information. 


The first part of our loss function ℒ_1 is relatively straightforward. The main obstacle here is to compute ℒ_2 as ŝ is unknown. We develop a strategy to compute ||ŝ||^2 indirectly. According to the definition of ŷ and ŝ we can derive:

    ℒ_2 =     ||ŝ||^2=||ŝ - 0||^2 
    
        =     ||[ŷ;ŝ]-[ŷ;0]||^2   Identical deformation
     
        =     ||𝐔·ẑ-[ŷ;0]||^2

where the square brackets represent the vector concatenation.
Because 𝐔 is orthogonal, we have

    ℒ_2 =    ||𝐔·ẑ-[ŷ;0]||^2 
    
        =     ||𝐔^-1· (𝐔·ẑ-[ŷ;0])||^2 
     
       =     ||ẑ-𝐔^-1·[ŷ;0]||^2 
       = ||ẑ-𝐔^T·[ŷ;0]||^2 
    
       =     ||ẑ-[𝐌;𝐍]^T·[ŷ;0]||^2 
    
       =     ||ẑ-𝐌^T·ŷ||^2 
       ≈ ||ẑ-𝐌^T·𝐲||^2

With Eq. <ref> (which makes use of the properties of orthogonal matrices), we avoid computing ŝ and 𝐍 directly when minimising ||ŝ||^2, and turn the minimisation problem into optimising 𝐌 instead. 
Finally, we have:

    ℒ_𝑀𝑆𝑃=    ℒ_1+ℒ_2 
    
       =    ||𝐌·ẑ - 𝐲||^2 + ||ẑ-𝐌^T·ŷ||^2  
    ≈    ||𝐌·ẑ - 𝐲||^2 + ||ẑ-𝐌^T·𝐲||^2


The loss function in Eq. <ref> also guarantees that after training, the solution for 𝐌 will be part of the orthogonal matrix 𝐔 (see <ref>). When ℒ_𝑀𝑆𝑃 is small, the transposition of 𝐌 becomes the inverse of 𝐌.





 §.§ Applying the Matrix Subspace Projection Framework to an AE
 


To apply our matrix subspace projection (MSP) framework to an existing AE, one only needs to derive a final loss function by combining the loss of the AE and the loss of our MSP framework. 

    ℒ = ℒ_𝐴𝐸 + αℒ_𝑀𝑆𝑃

where α is the weight for ℒ_𝑀𝑆𝑃. As illustrated in Fig. <ref> (a) and (b), one should note that applying our framework will not change the structure of the AE, where our MSP component simply takes the latent vector ẑ of the AE as input. ℒ_𝐴𝐸 hopes that ŝ can store more information to reconstruct 𝐱, but ℒ_𝑀𝑆𝑃 wants ŝ to contain less information. When α is small, the model becomes a standard AE. When α is too large, the non-attribute information in ẑ is reduced excessively, resulting in the generated products tending to the average of the training samples. Therefore, another challenge we face is how to set α appropriately. 


We propose a principled strategy for effectively determining the value of α (this strategy is used in all experiments in this paper). Since ℒ_𝐴𝐸 and ℒ_𝑀𝑆𝑃 essentially represent a competing relationship for ẑ resources, we specify that ℒ_𝐴𝐸 and ℒ_𝑀𝑆𝑃 have the same influence on updating ŝ. We use α to represent the “intensity” with which the AE updates ẑ during each back-propagation process. This intensity depends on the structure of the model and the loss function used by the model.
For example, suppose an AE (for picture generation) uses a CNN decoder and L2-loss. During the training process, the error of each pixel between the generated picture and the true picture is backpropagated to ẑ as the gradients of ẑ. The sum of these gradients is the final gradient of ẑ (i.e., corresponding to ℒ_𝐴𝐸). For a picture with h× w pixels and c colour channels, there are h× w× c parts of gradients accumulated, so the intensity is h× w× c. The intensity of ŷ for updating ẑ is the total amount of attributes (i.e. the dimension of ŷ), because the error for each attribute is propagated back to ẑ and accumulated (i.e., corresponding to ℒ_𝑀𝑆𝑃). Therefore, in order to balance the influence of ℒ_𝐴𝐸 and ℒ_𝑀𝑆𝑃 on updating ẑ during training, we have: 

    α≈h × w × c/size(attribute) + size(ẑ)



When using the cross-entropy-loss (or NLL loss etc.), which is usually for textual generative models (e.g. the seq2seq model), each generated word produces only one intensity, regardless of the word embedding size. 
Meanwhile, loss values returned by the cross-entropy-loss are proportional to the error, but the loss values returned by the MSP loss (which is a MSE loss) are proportional to the error's square.
Therefore, for a sentence of length k, the intensity of the entire sentence to ẑ is k^2, so that for cross-entropy-loss,

    α≈k^2/size(attribute) + size(ẑ)






 §.§ Content Replacement and Conditional Generation

After MSP is trained (i.e., 𝐌 is estimated), there are two ways to perform content replacement or change of attributes. One way is to derive the orthogonal matrix 𝐔 = [𝐌;𝐍] by solving the null space  𝐍 of 𝐌 (i.e., the null space is constituted of all the specific solutions for 𝐧 w.r.t. equation 𝐌·𝐧=0, where 𝐧 is an independent variable). Given an input 𝐱, we first encode it as ẑ. We then use 𝐔 to obtain the attribute vector ŷ of 𝐱 and the non-attribute information vector ŝ as follows.

    [ŷ;ŝ]=[𝐌;𝐍]·ẑ= 𝐔·ẑ=𝐔·encoder(𝐱)

At this point, we can directly replace [ŷ;ŝ] with [𝐲_𝐧;ŝ], where 𝐲_𝐧 is the new attribute vector. With [𝐲_𝐧;ŝ] and 𝐔^T (note that 𝐔^T approaches  𝐔^-1 during training), we can derive the new latent code 𝐳_𝐧 and then decode it into 𝐱_𝐧, which ultimately captures the desired new attributes. 

    𝐱_𝐧 = decoder(𝐳_𝐧)=decoder(𝐔^T· [𝐲_𝐧;ŝ])


Alternatively, we can avoid explicitly calculating  matrix 𝐍 (i.e. avoid calculating ŝ), for content replacement. According to Eqs.<ref> and <ref>, we define 𝐝 as the distance between ẑ and 𝐳_𝐧.

    𝐝    = ẑ - 𝐳_𝐧 = 𝐔^T· [ŷ;ŝ] - 𝐔^T· [𝐲_𝐧;ŝ] 
        = 𝐔^T· ([ŷ;ŝ] - [𝐲_𝐧;ŝ]) = 𝐔^T· [ŷ-𝐲_𝐧;0] 
        = [𝐌;𝐍]^T· [ŷ-𝐲_𝐧;0] 
        = 𝐌^T· (ŷ-𝐲_𝐧) = 𝐌^T· (𝐌·ẑ-𝐲_𝐧)

It should be noted that here ẑ≠𝐌^T·𝐌·ẑ because the reconstruction loss does not allows ŝ to be zero. Thus, we have:

    𝐳_𝐧   = ẑ-𝐝 = ẑ - 𝐌^T· (𝐌·ẑ-𝐲_𝐧) 
    𝐱_𝐧   = decoder(ẑ-𝐌^T· (𝐌·ẑ-𝐲_𝐧))



If the AE itself is a generative model (such as VAE), then the AE+MSP structure becomes a conditional generative model. Given a randomly sampled 𝐬_𝐫 and an attribute vector 𝐲_𝐫, the model can  generate new sample 𝐱_𝐫 with the desired attributes with Eq.<ref>.































§ EVALUATION



Here we evaluate  the ability to disentangle. 
We  also evaluate the orthogonality of  𝐌 as it is an important indicator  of how well our algorithm can approximate 𝐌. 







 §.§ Matrix Subspace Projection in VAE


We apply our model on a vanilla VAE <cit.> with the standard CNN encoder and decoder (the architectures are same as <cit.>).
We used the ADAM optimiser with learning rate = 0.0002, mini-batch size of 256, and images are upsampled to 256 × 256.
We add an additional  PatchGAN <cit.> to make the produced images sharp. The architecture of the PatchGAN discriminator also adopts the version of <cit.>. 
Our baselines are Fader networks <cit.> and
AttGAN  <cit.>, based on their published code and settings.
We did not compare StarGAN <cit.> because we feel it is superseded by AttGAN, which demonstrated better performance.
We did not compare RelGAN <cit.> as it does not disentangle attributes (see Fig. <ref> (left), also RelGAN cannot add a moustache or beard to a female face, instead it will transform it to a male face with beard). 









We evaluated on the CelebA dataset <cit.> (202,600 images) and trained one model on all 40 labelled attributes. 
The generated examples are shown in Fig. <ref>.
Qualitatively  we see clear examples of what Fader networks and AttGan cannot do: For the woman with glasses (middle) FaderNetwork and AttGan show complete inability to remove the glasses; FaderNetwork completely fails to add glasses to the other two faces, and AttGan can only manage weak rims on the final woman (bottom). FaderNetwork in general struggles to change attributes, especially for the two women, while AttGan does better, but struggles with certain attributes, e.g. mostly it fails to change the final woman to male, and struggles to remove makeup.





For a quantitative evaluation we trained a classifier (ResNet-CNN) to measure the accuracy with which attributes are changed.
Table <ref> shows that our MSP approach outperforms the competitors.
Finally we calculated the average Fréchet Inception Distance
(FID) <cit.>
for each method: MSP=35.0, Fader=26.3, AttGAN=7.3 (lower is better, 0 is the best).
The FID score tries to calculate the similarity of original images and generated images. 
Clearly AttGAN is a winner for quality while our MSP is a winner for accuracy of attribute modification.
When AttGAN  cannot handle the attribute modification it  generates  unchanged images and can get  lower FID scores.

The results of attribute manipulation (both qualitative and quantitative) are surprisingly bad for Fader networks and AttGAN, especially relative to the examples displayed in their original papers. The primary reason for this is that we trained those models on all 40 attributes together. Fader networks works best when trained on a single attribute, as noted in Sec. <ref>.  The original AttGAN paper trained on 13 attributes, and indeed it performs better at attribute manipulation than Fader in our pictures. For the 40-attribute-together version, when any attribute is changed all others must be unchanged. For example, when we transition from male to female (Fig. <ref> left), it is implicit that the female should keep no make-up or lipstick, etc. In the direction from female to male the male should keep no bushy eyebrows or 5 o'clock shadow. The original Fader network paper displays a beautiful example of interpolating between male and female, but the female does have make-up and lipstick and the male does have bushy eyebrows and 5 o'clock shadow.
Our difficult 40-attribute setting is central to our aims, as stated in our introduction:
we want to fully disentangle multiple attributes, because this gives a
generative model the ability to `imagine' novel examples that combine attributes in ways not present in training data.






 §.§ Human Evaluation of Generated Example Quality


We evaluated whether our MSP model reduces the quality of  generated examples, using human evaluation via Amazon Mechanical Turk (hiring 150 participants in total).

For each model, we randomly generated 1,000 example pairs. Each pair contains a reconstructed example (from AE) and an example generated by AE+MSP with one or two random attribute modification (attributes were changed to -1 if they were originally > 0, or changed to 1 if they were < 0)[The generated examples were automatically filtered to prevent conflict attributes; e.g. images of woman with beard are not provided to the participants in this experiment.]. The participants were shown the examples pair-by-pair in the blind test, and they were asked to please choose the one with better text/image quality, or choose both if you think they perform similarly. The participants were told that the text quality means the fluency, semantic accuracy, and syntactic accuracy, and the image quality means the clarity and (face) recognisability. The results are shown in Table <ref>.

We treat the scores (i.e. participants' choices) of the result as a Likert scale, and we set our null hypothesis to be H_0: generation quality of AE+MSP is worse than using the AE only, and, H_1: generation quality of AE+MSP is equal or higher than using the AE only. The hypotheses are tested by a discrete Mann-Whitney U-test, rejecting H_0  with p<0.03.








 §.§ Evaluation of Disentanglement


Disentanglement is also an important feature of our model. It means that when an attribute is modified, other attributes remain unchanged. 
We make the three models (VAE+GAN+MSP, AttGAN, and Fader Networks) generate images by manipulating two groups of highly correlated attributes, openness of mouth / smiling, and male / beard. 
For the two groups, the three models should respectively generate the images with closed mouth × no smiling, closed mouth × smiling, opened mouth × no smiling, opened mouth × smiling, female × no beard, female × beard, male × no beard, and male × beard. We hired 50 participants in Amazon Mechanical Turk; each of them was given 40 image blocks. A block contains four images, which were from the mouth / smiling group or the male / beard group, and which were generated by one of the three models. The participants were told which image should represent which attributes, and the participants evaluated whether it did for  each image in the block by using a 3-level Likert scale (perfect, recognisable, and unrecognisable/unchanged). The results are shown in Table <ref>. It shows that our model performs significantly better than the baseline. (p<0.0001).







In addition to human evaluation, we also conducted a quantitative evaluation to test how well a model can change an attribute in isolation. For some selected  highly correlated attributes we change one target attribute, and measure the change in another non-target attribute. For instance (the row of gender/beard in Table <ref>), when the gender attribute is changed manually, we measure the amount by which the beard attribute is consequently changed. The results are shown in Table <ref>. 
Note that, the scores show how much the non-target attributes are affected, but not whether the target attributes are correctly changed in the generated pictures. Therefore the scores need to  be read in conjunction with Table <ref>. According to both Table <ref> and Table <ref>, we can conclude that in both of the aspects of the manipulation of attributes and avoiding influence on non-target attributes, the performance of our model exceeds the baselines.








 §.§ Matrix Subspace Projection in Seq2seq




We apply our model to a classic seq2seq model for textual content replacement, in order to determine if we can replace words according to the given attributes and keep other words unchanged.
In this task, we adopt the E2E corpus <cit.>, which contains 50k+ reviews of restaurants (E2E dataset is developed for Natural Language Generation, but here we use it for content replacement). Each review is a single sentence that is labelled by the attribute-value pairs, for example, “name=[The Eagle]”, “food=[French]”, and “customerRating=[3/5]”. We regard each attribute-value pair as a unique label. All the attributes constitute y whose entries are 1 or 0 to represent each value  (the correct texts of the attribute name or value are NOT used).

Both the encoder and decoder of the seq2seq model are formed by two-layer LSTMs. The model is trained for 1000 epochs (on a Tesla T4 around 12 hours). After training, we reconstruct the review sentences with randomly replaced attributes, for example replacing “name=[The Eagle]” by “name=[Burger King]”, “customerRating=[3/5]” by “customerRating=[1/5]”. 50% of attributes are changed in each sentence.
The outcomes are shown in Table <ref>.





 §.§ Orthogonality Evaluation
 


The ability to disentangle attributes is ensured by the orthogonality of 𝐌 in our model. Instead of directly using an orthogonal matrix, we train 𝐌 to be orthogonal. Thus, we evaluate how close 𝐌 is to the orthogonal matrix.
Fig. <ref> shows the heat map of 𝐌^T·𝐌 and 𝐔^T·𝐔, which indicates that the production is fairly close to a unit matrix. It visualises 𝐌^T·𝐌 in the seq2seq version of our model (Fig. <ref> (a)) and in the VAE version (Fig. <ref> (c)). The matrix 𝐔^T·𝐔 (𝐔 is formed by 𝐌 concatenating its null space) is also visualised (in Fig. <ref> (b) and (d)). It is clear that when the matrices are multiplied by their transposes, the products do approximate the unit matrix.
Although Fig. <ref> (c) shows that a small number of attributes remain slightly entangled (by the green and deep purple pixels), this is mainly caused by the few conflicting attributes in CelebA, for example the receding-hairline × bald × bangs, and the straight-hair × wavy-hair.
Thus, 𝐌 is indeed trained to be a (partial) orthogonal matrix.  




§ CONCLUSION

We propose a matrix projection plugin that can be attached to various autoencoders (e.g. Seq2seq, VAE) to make the latent space factorised and  disentangled, based on  labelled attribute information, which ensures that manipulation in the latent space is much easier. 
We test the attribute manipulation ability of our model on an image dataset and text corpus, obtaining results that show clean disentanglement. 
In addition our model involves a simpler training process than adversarial approaches which need a long training with a very low weight on the loss coming from the discriminator that removes attribute information, to avoid the encoder being too affected by this loss term <cit.>.




§ ACKNOWLEDGEMENT

We would like to thank all the anonymous reviewers for their insightful comments. This work is supported by the award made by the
UK Engineering and Physical Sciences Research
Council (Grant number: EP/P011829/1).



icml2020


