Especially with high probability we will have that
$|\langle\bm{\xi},{\bm{v}}\rangle|\lesssim\log n/\sqrt{n}$. Then we conclude
from (21), with high probability it holds

$\displaystyle\widehat{\beta}=\beta_{j_{*}}+\operatorname{O}\left(\frac{1}{\beta_{j_{*}}}+\frac{\log
n}{\sqrt{n}}\right).$ (180)

With the bound (180), we can replace
$\langle{\bm{a}},{\bm{v}}\rangle/(2\beta^{2})$ on the righthand side of (20)
by $\langle{\bm{a}},{\bm{v}}\rangle/(2\widehat{\beta}^{2})$, which gives an
error

$\displaystyle\left|\frac{\langle{\bm{a}},{\bm{v}}\rangle}{2\beta_{j_{*}}^{2}}-\frac{\langle{\bm{a}},{\bm{v}}\rangle}{2\widehat{\beta}^{2}}\right|=\operatorname{O}\left(|\langle{\bm{a}},{\bm{v}}\rangle|\left(\frac{1}{|\beta_{j_{*}}|^{4}}+\frac{\log
n}{|\beta_{j_{*}}|^{3}\sqrt{n}}\right)\right).$ (181)

Combining the above discussion together, we can rewrite (20) as

$\displaystyle\begin{split}\langle{\bm{a}},\widehat{\bm{v}}\rangle-\left(1-\frac{1}{2\widehat{\beta}^{2}}\right)\langle{\bm{a}},{\bm{v}}\rangle&=\frac{\langle{\bm{a}},{\bm{\xi}}\rangle-\langle{\bm{a}},{\bm{v}}_{j_{*}}\rangle\langle{\bm{v}}_{j_{*}},{\bm{\xi}}\rangle}{\beta_{j_{*}}}\\\
&+\operatorname{O}_{\mathbb{P}}\left(\frac{\log n}{\sqrt{n}}\left(\frac{\log
n}{\sqrt{n}|\beta_{1}|}\right)^{k-1}+\frac{\log
n}{|\beta_{1}|^{2}\sqrt{n}}+\frac{(\log
n)^{3/2}}{|\beta_{1}|^{3/2}n^{3/4}}+\frac{1}{|\beta_{1}|^{4}}\right),\end{split}$
(182)

with high probability, where we used that $|\beta_{j_{*}}|\gtrsim|\beta_{1}|$.

Again thanks to the definition of ${\bm{\xi}}$ in (20) of Theorem 2.1, i.e.
${\bm{\xi}}=\bm{Z}[{\bm{v}}^{\otimes(k-1)}]$, is an $n$-dim vector, with each
entry i.i.d. ${\mathcal{N}}(0,1/n)$ Gaussian random variable, we see that

$\displaystyle\langle{\bm{a}},{\bm{\xi}}\rangle-\langle{\bm{a}},{\bm{v}}_{j_{*}}\rangle\langle{\bm{v}}_{j_{*}},{\bm{\xi}}\rangle=\langle{\bm{a}}-\langle{\bm{a}},{\bm{v}}_{j_{*}}\rangle{\bm{v}}_{j_{*}},\bm{\xi}\rangle,$
(183)

is a Gaussian random variable, with mean zero and variance

$\displaystyle\mathbb{E}[\langle{\bm{a}}-\langle{\bm{a}},{\bm{v}}_{j_{*}}\rangle{\bm{v}}_{j_{*}},\bm{\xi}\rangle^{2}]$
$\displaystyle=\frac{1}{n}\|{\bm{a}}-\langle{\bm{a}},{\bm{v}}_{j_{*}}\rangle{\bm{v}}_{j_{*}}\|_{2}^{2}=\frac{1}{n}\langle{\bm{a}},(\bm{I}_{n}-{\bm{v}}_{j_{*}}{\bm{v}}_{j_{*}}^{\top}){\bm{a}}\rangle$
(184)
$\displaystyle=\frac{1+\operatorname{o}(1)}{n}\langle{\bm{a}},(\bm{I}_{n}-\widehat{\bm{v}}_{j_{*}}\widehat{\bm{v}}_{j_{*}}^{\top}){\bm{a}}\rangle.$
(185)

This together with (180), (182) as well as our assumption (22)

$\frac{\sqrt{n}\widehat{\beta}}{\sqrt{\langle{\bm{a}},(\bm{I}_{n}-\widehat{\bm{v}}\widehat{\bm{v}}^{\top}){\bm{a}}\rangle}}\left[\big{(}1-\frac{1}{2\widehat{\beta}^{2}}\big{)}^{-1}\langle{\bm{a}},\widehat{\bm{v}}\rangle-\langle{\bm{a}},{\bm{v}}_{j_{*}}\rangle\right]\xrightarrow{d}{\mathcal{N}}(0,1).$
(186)

Under the same assumption, we have similar results for Cases 2, 3, 4, by
simply changing $(\beta_{j_{*}},{\bm{v}}_{j_{*}})$ in the righthand side of
(7) and (8) to the corresponding expression. ∎

###### Proof of Corollary 2.10.

For $k\geqslant 3$ and $|\beta_{1}|\geqslant n^{(k-2)/2+\varepsilon}$, the
assumption 22 holds trivially. The claim (29) follows from (24). For (30), we
recall that in (28), ${\bm{\xi}}=\bm{Z}[{\bm{v}}_{i}^{\otimes(k-1)}]$, is an
$n$-dim vector, with each entry i.i.d. ${\mathcal{N}}(0,1/n)$ Gaussian random
variable. We see that

$\langle\bm{\xi},{\bm{v}}_{i}\rangle\stackrel{{\scriptstyle
d}}{{=}}{\mathcal{N}}\left(0,1/n\right).$

Especially with high probability we will have that
$|\langle\bm{\xi},{\bm{v}}\rangle|\lesssim\log n/\sqrt{n}$. Then we conclude
from (28), with high probability it holds

$\displaystyle\widehat{\beta}=\beta_{i}+\operatorname{O}\left(\frac{1}{\beta_{i}}+\frac{\log
n}{\sqrt{n}}\right).$ (187)

With the bound (187), we can replace $(k/2-1)/\beta_{i}$ on the righthand side
of (28) by $(k/2-1)/\widehat{\beta}$, which gives an error

$\displaystyle\left|\frac{k/2-1}{\beta_{i}}-\frac{k/2-1}{\widehat{\beta}}\right|=\operatorname{O}\left(\frac{1}{|\beta_{1}|^{2}}+\frac{\log
n}{|\beta_{1}|\sqrt{n}}\right),$ (188)

where we used that $|\beta_{i}|\gtrsim|\beta_{1}|$. Combining the above
discussion together, we can rewrite (28) as

$\displaystyle\beta_{i}=\widehat{\beta}+\frac{k/2-1}{\widehat{\beta}}-\langle\bm{\xi},{\bm{v}}_{i}\rangle+\operatorname{O}_{\mathbb{P}}\left(\frac{\log
n}{\sqrt{n}}\left(\frac{\log n}{\sqrt{n}|\beta_{1}|}\right)^{k-1}+\frac{\log
n}{|\beta_{1}|\sqrt{n}}+\frac{(\log
n)^{3/2}}{|\beta_{1}|^{1/2}n^{3/4}}+\frac{1}{|\beta_{1}|^{2}}\right).$ (189)

Since $\langle\bm{\xi},{\bm{v}}_{i}\rangle\stackrel{{\scriptstyle
d}}{{=}}{\mathcal{N}}\left(0,1/n\right)$, and the error term in (189) is much
smaller than $1/\sqrt{n}$. We conclude from (189)

$\displaystyle\sqrt{n}\left(\beta_{i}-\widehat{\beta}+\frac{k/2-1}{\widehat{\beta}}\right)\xrightarrow{d}{\mathcal{N}}(0,1).$
(190)

This finishes the proof of (30).

∎

###### Proof of Corollary 2.11.

Given the significance level $\alpha$, the asymptotic confidence intervals in
Corollary 2.11 can be calculated from Corollary 2.10 by bounding the absolute
values of the left hand sides of (29) and (30) at $z_{\alpha}$. ∎

## References

  * [1] E. Abbe, J. Fan, K. Wang, Y. Zhong, et al.  Entrywise eigenvector analysis of random matrices with low expected rank.  Annals of Statistics, 48(3):1452–1474, 2020.
  * [2] A. Anandkumar, R. Ge, D. Hsu, and S. M. Kakade.  A tensor approach to learning mixed membership community models.  The Journal of Machine Learning Research, 15(1):2239–2312, 2014\.
  * [3] A. Anandkumar, R. Ge, D. Hsu, S. M. Kakade, and M. Telgarsky.  Tensor decompositions for learning latent variable models.  Journal of Machine Learning Research, 15:2773–2832, 2014.
  * [4] G. B. Arous, R. Gheissari, A. Jagannath, et al.  Algorithmic thresholds for tensor pca.  Annals of Probability, 48(4):2052–2087, 2020.
  * [5] Z. Bai and J. Yao.  On sample eigenvalues in a generalized spiked population model.  Journal of Multivariate Analysis, 106:167–177, 2012.
  * [6] J. Baik, G. B. Arous, S. Péché, et al.  Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices.  The Annals of Probability, 33(5):1643–1697, 2005.
  * [7] J. Baik and J. W. Silverstein.  Eigenvalues of large sample covariance matrices of spiked population models.  Journal of multivariate analysis, 97(6):1382–1408, 2006.
  * [8] F. Benaych-Georges and R. R. Nadakuditi.  The singular values and vectors of low rank perturbations of large rectangular random matrices.  Journal of Multivariate Analysis, 111:120–135, 2012.
  * [9] A. Birnbaum, I. M. Johnstone, B. Nadler, and D. Paul.  Minimax bounds for sparse pca with noisy high-dimensional data.  Annals of statistics, 41(3):1055, 2013.
  * [10] T. Cai, Z. Ma, and Y. Wu.  Optimal estimation and rank detection for sparse spiked covariance matrices.  Probability theory and related fields, 161(3-4):781–815, 2015.
  * [11] T. T. Cai, Z. Ma, Y. Wu, et al.  Sparse pca: Optimal rates and adaptive estimation.  The Annals of Statistics, 41(6):3074–3110, 2013.
  * [12] W.-K. Chen et al.  Phase transition in the spiked random tensor with rademacher prior.  The Annals of Statistics, 47(5):2734–2756, 2019.
  * [13] W.-K. Chen, M. Handschy, and G. Lerman.  Phase transition in random tensors with multiple spikes.  arXiv preprint arXiv:1809.06790, 2018.
  * [14] Y. Chen, C. Cheng, and J. Fan.  Asymmetry helps: Eigenvalue and eigenvector analyses of asymmetrically perturbed low-rank matrices.  arXiv preprint arXiv:1811.12804, 2018.
  * [15] C. Cheng, Y. Wei, and Y. Chen.  Inference for linear forms of eigenvectors under minimal eigenvalue separation: Asymmetry and heteroscedasticity.  arXiv preprint arXiv:2001.04620, 2020.
  * [16] A. Cichocki, D. Mandic, L. De Lathauwer, G. Zhou, Q. Zhao, C. Caiafa, and H. A. Phan.  Tensor decompositions for signal processing applications: From two-way to multiway component analysis.  IEEE signal processing magazine, 32(2):145–163, 2015.
  * [17] P. Comon.  Tensors: a brief introduction.  IEEE Signal Processing Magazine, 31(3):44–53, 2014.
  * [18] D. L. Donoho, M. Gavish, and I. M. Johnstone.  Optimal shrinkage of eigenvalues in the spiked covariance model.  Annals of statistics, 46(4):1742, 2018.
  * [19] O. Duchenne, F. Bach, I.-S. Kweon, and J. Ponce.  A tensor-based algorithm for high-order graph matching.  IEEE transactions on pattern analysis and machine intelligence, 33(12):2383–2395, 2011.
  * [20] N. El Karoui et al.  Spectrum estimation for large dimensional covariance matrices using random matrix theory.  The Annals of Statistics, 36(6):2757–2790, 2008.
  * [21] E. Frolov and I. Oseledets.  Tensor methods and recommender systems.  Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 7(3):e1201, 2017.
  * [22] W. Hackbusch.  Tensor spaces and numerical tensor calculus, volume 42.  Springer, 2012.
  * [23] C. J. Hillar and L.-H. Lim.  Most tensor problems are np-hard.  Journal of the ACM (JACM), 60(6):1–39, 2013.
  * [24] S. B. Hopkins, T. Schramm, J. Shi, and D. Steurer.  Fast spectral algorithms from sum-of-squares proofs: tensor decomposition and planted sparse vectors.  In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 178–191, 2016.
  * [25] S. B. Hopkins, J. Shi, and D. Steurer.  Tensor principal component analysis via sum-of-square proofs.  In Conference on Learning Theory, pages 956–1006, 2015.
  * [26] D. Hsu and S. M. Kakade.  Learning mixtures of spherical gaussians: moment methods and spectral decompositions.  In Proceedings of the 4th conference on Innovations in Theoretical Computer Science, pages 11–20, 2013.
  * [27] A. Jagannath, P. Lopatto, and L. Miolane.  Statistical thresholds for tensor pca.  arXiv preprint arXiv:1812.03403, 2018.
  * [28] I. M. Johnstone.  On the distribution of the largest eigenvalue in principal components analysis.  Annals of statistics, pages 295–327, 2001.
  * [29] I. M. Johnstone and A. Y. Lu.  On consistency and sparsity for principal components analysis in high dimensions.  Journal of the American Statistical Association, 104(486):682–693, 2009.
  * [30] I. M. Johnstone and D. Paul.  PCA in high dimensions: An orientation.  Proceedings of the IEEE, 106(8):1277–1292, 2018.
  * [31] A. Karatzoglou, X. Amatriain, L. Baltrunas, and N. Oliver.  Multiverse recommendation: n-dimensional tensor factorization for context-aware collaborative filtering.  In Proceedings of the fourth ACM conference on Recommender systems, pages 79–86, 2010.
  * [32] C. Kim, A. S. Bandeira, and M. X. Goemans.  Community detection in hypergraphs, spiked tensor models, and sum-of-squares.  In 2017 International Conference on Sampling Theory and Applications (SampTA), pages 124–128. IEEE, 2017.
  * [33] T. G. Kolda and B. W. Bader.  Tensor decompositions and applications.  SIAM review, 51(3):455–500, 2009.
  * [34] O. Ledoit, M. Wolf, et al.  Nonlinear shrinkage estimation of large-dimensional covariance matrices.  The Annals of Statistics, 40(2):1024–1060, 2012.
  * [35] T. Lesieur, L. Miolane, M. Lelarge, F. Krzakala, and L. Zdeborová.  Statistical and computational phase transitions in spiked tensor estimation.  In 2017 IEEE International Symposium on Information Theory (ISIT), pages 511–515. IEEE, 2017.
  * [36] Y. Luo, G. Raskutti, M. Yuan, and A. R. Zhang.  A sharp blockwise tensor perturbation bound for orthogonal iteration.  arXiv preprint arXiv:2008.02437, 2020.
  * [37] Y. Luo and A. R. Zhang.  Open problem: Average-case hardness of hypergraphic planted clique detection.  In Conference on Learning Theory, pages 3852–3856. PMLR, 2020.
  * [38] Z. Ma et al.  Sparse principal component analysis and iterative thresholding.  The Annals of Statistics, 41(2):772–801, 2013.
  * [39] S. O’Rourke, V. Vu, and K. Wang.  Random perturbation of low rank matrices: Improving classical bounds.  Linear Algebra and its Applications, 540:26–59, 2018.
  * [40] D. Paul.  Asymptotics of sample eigenstructure for a large dimensional spiked covariance model.  Statistica Sinica, pages 1617–1642, 2007.
  * [41] S. Péché.  The largest eigenvalue of small rank perturbations of hermitian random matrices.  Probability Theory and Related Fields, 134(1):127–173, 2006.
  * [42] A. Perry, A. S. Wein, A. S. Bandeira, et al.  Statistical limits of spiked tensor models.  In Annales de l’Institut Henri Poincaré, Probabilités et Statistiques, volume 56, pages 230–264. Institut Henri Poincaré, 2020.
  * [43] S. Rendle and L. Schmidt-Thieme.  Pairwise interaction tensor factorization for personalized tag recommendation.  In Proceedings of the third ACM international conference on Web search and data mining, pages 81–90, 2010.
  * [44] E. Richard and A. Montanari.  A statistical model for tensor pca.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2897–2905. Curran Associates, Inc., 2014.
  * [45] N. D. Sidiropoulos, L. De Lathauwer, X. Fu, K. Huang, E. E. Papalexakis, and C. Faloutsos.  Tensor decomposition for signal processing and machine learning.  IEEE Transactions on Signal Processing, 65(13):3551–3582, 2017\.
  * [46] E. Simony, C. J. Honey, J. Chen, O. Lositsky, Y. Yeshurun, A. Wiesel, and U. Hasson.  Dynamic reconfiguration of the default mode network during narrative comprehension.  Nature communications, 7:12141, 2016.
  * [47] V. Vu.  Singular vectors under random perturbation.  Random Structures & Algorithms, 39(4):526–538, 2011.
  * [48] V. Q. Vu, J. Lei, et al.  Minimax sparse principal subspace estimation in high dimensions.  The Annals of Statistics, 41(6):2905–2947, 2013.
  * [49] A. Zhang, T. T. Cai, and Y. Wu.  Heteroskedastic pca: Algorithm, optimality, and applications.  arXiv preprint arXiv:1810.08316, 2018.
  * [50] A. Zhang and D. Xia.  Tensor svd: Statistical and computational limits.  IEEE Transactions on Information Theory, 64(11):7311–7338, 2018\.
  * [51] Y. Zhong.  Eigenvector under random perturbation: A nonasymptotic rayleigh-schr$\backslash$”$\\{$o$\\}$ dinger theory.  arXiv preprint arXiv:1702.00139, 2017.
  * [52] H. Zhou, L. Li, and H. Zhu.  Tensor regression with applications in neuroimaging data analysis.  Journal of the American Statistical Association, 108(502):540–552, 2013.


capbtabboxtable[][]

11institutetext: Bilkent University
11email: {yusuf.dalva<EMAIL_ADDRESS>
11email<EMAIL_ADDRESS>
# VecGAN: Image-to-Image Translation with Interpretable Latent Directions

Yusuf Dalva  Said Fahri Altındiş  Aysegul Dundar

###### Abstract

We propose VecGAN, an image-to-image translation framework for facial
attribute editing with interpretable latent directions. Facial attribute
editing task faces the challenges of precise attribute editing with
controllable strength and preservation of the other attributes of an image.
For this goal, we design the attribute editing by latent space factorization
and for each attribute, we learn a linear direction that is orthogonal to the
others. The other component is the controllable strength of the change, a
scalar value. In our framework, this scalar can be either sampled or encoded
from a reference image by projection. Our work is inspired by the latent space
factorization works of fixed pretrained GANs. However, while those models
cannot be trained end-to-end and struggle to edit encoded images precisely,
VecGAN is end-to-end trained for image translation task and successful at
editing an attribute while preserving the others. Our extensive experiments
show that VecGAN achieves significant improvements over state-of-the-arts for
both local and global edits.

###### Keywords:

Image translation, generative adversarial networks, latent space manipulation,
face attribute editing.

##  1 Introduction

There has been a significant progress in image-to-image translation methods
[15, 26, 39, 8, 23, 41, 22, 25] especially for facial attribute editing [7,
27, 37, 42, 21] powered with generative adversarial networks (GANs). A main
challenge of facial attribute editing methods is to be able to change only one
attribute of an image without affecting others such as global lighting
parameters of the images, identity of the persons, background, or their other
attributes. The other challenge is the interpretability of the style codes so
that one can control the attribute intensity of the edit, e.g. increase the
intensity of smile or aging.

To achieve the targeted attribute editing while preserving the others, many
works set a separate style encoder and an image editing network where modified
styles are injected into it [7, 21]. During image-to-image translation, a
style encoded from another image or a newly sampled style latent code can be
used to output diverse images. To disentangle attributes, works focus on style
encoding and progress from a shared style code, SDIT [34], to mixed style
codes, StarGANv2 [7], to hierarchical disentangled styles, HiSD [21]. Among
these works, HiSD independently learn styles of each attribute, bangs, hair
color, glasses and introduces a local translator which uses attention masks to
avoid global manipulations. HiSD showcases successes on those three local
attribute editing task and is not tested for global attribute editing, e.g.
age, smile. Furthermore, one limitation of these works is the
uninterpretablity of style codes as one cannot control the intensity of
attribute (e.g. blondness) in a straight-forward manner.

To overcome the challenges of facial attribute editing task, we propose a
novel framework, VecGAN, and image-to-image translation framework with
interpretable latent directions. Our framework does not require a separate
style encoder as in the previous works since we achieve the translation in the
encoded latent space directly. The attribute editing directions are learned in
the latent space and regularized to be orthogonal to each other for style
disentanglement. The other component of our framework is the controllable
strength of the change, a scalar value. This scalar can be either sampled from
a distribution or encoded from a reference image by projection in the latent
space. Our framework not only achieves significant improvements over state-of-
the-arts for both local and global edits but also provides a knob to control
the editing attribute intensity via its design.

Figure 1: Attribute editing results of VecGAN. The first column shows the
source images, and other columns show the results of editing a specific
attribute. Each edited image has an attribute value opposite to that of the
source one. For hair color, sources are translated to brown, black, and blonde
hair, respectively.

VecGAN is encouraged by the findings that well-trained generative models
organize their latent space as disentangled representations with meaningful
directions in a completely unsupervised way. Exploring these interpretable
directions in latent codes has emerged as an important research endeavor on
the fixed pretrained GANs [28, 31, 11, 29, 36]. These works show that images
can be mapped to the GANs latent space and edits can be achieved by
manipulations in the latent space. However, since these models are not trained
end-to-end, the results are sub-optimal as will also be shown in our
experiments.

To enable VecGAN, different than previous works of image-to-image translation
networks, we use a deeper neural network architecture. Image-to-image
translation methods, such as state-of-the-art HiSD [21] uses a network with
small receptive fields that decreases the image resolution only by four times
in the encoder. However, we want an organization in a latent space such that
we can take meaningful linear directions. Therefore, images should be encoded
to a spatially smaller feature space and a network should have a full
understanding of an image. For that reason, we set a deep encoder and decoder
network architecture but then this network faces the challenges of
reconstructing all the details from the input image. To solve this problem, we
use a skip connection between the encoder and decoder but only at lower
resolution to find the optimal equilibrium of the information flow between
with and without dimensionality reduction bottleneck. In summary, our main
contributions are:

  * •

We propose VecGAN, a novel image-to-image translation network that is trained
end to end with interpretable latent directions. Our framework does not employ
a separate style network as in the previous works and translations are
achieved with a single deep encoder-decoder architecture.

  * •

VecGAN enables both reference attribute copy and attribute strength
manipulation. Reference style encoding is designed in a novel way by using the
same encoder from the translation pipeline. First, encoder is used to obtain
latent codes of a reference image and it is followed by the projection of the
codes into learned latent directions for different attributes.

  * •

We conduct extensive experiments to show the effectiveness of our framework
and achieve significant improvements over state-of-the-art for both local and
global edits. Qualitative results of our framework can be seen in Fig. 1.

##  2 Related Works

Image to Image Translation. Image-to-image translation algorithms aim at
preserving a given content while changing targeted attributes. Examples range
from translating semantic maps into RGB images [33], to translating summer
images into winter images [15], to portrait drawing [40] and very popularly to
editing faces [7, 27, 37, 42, 21, 35, 9, 13, 2]. These algorithms powered with
GAN loss [10] set an encoder-decoder architecture. In models that learn a
deterministic mapping from one domain to the other, images are processed with
encoder and decoder to output translated images [33, 26]. In multi-modal
image-to-image translation methods, style is encoded separately from an
another image or sampled from a distribution [14, 7]. In the generator, style
and content are either combined with concatenation [45], or combined with a
mask [21] or fed separately through instance normalization blocks [14, 46].
The generator also uses an encoder-decoder architecture [21, 38] that is
seperate than the style encoder. In our work, we are interested in designing
the attribute as a learnable linear direction in the latent space and we do
not employ a separate style encoder which results in a more intuitive
framework.

Learning interpretable latent directions. In another line of research, it is
shown that GANs that are trained to synthesize faces can also be used for face
attribute manipulations [17, 5, 18]. Initially, these networks are not
designed or trained to translate images but rather to synthesize high fidelity
images. However, it is shown that one can embed existing images into the GAN’s
embedding space [1] and further one can find latent directions to edit those
images [28, 31, 11, 29, 36]. These directions are explored in supervised [28]
and unsupervised ways [31, 11, 29, 36]. It is quite remarkable when the
generative network is only taught to synthesize realistic images, it organizes
the use of latent space such that linear shifts on them change a specific
attribute. Inspired by these findings, we design our image to image
translation such that a linear shift in the encoded features is expected to
change a single attribute of an image. Different than previous works, our
framework is trained end-to-end for translation task and allows for reference
guided attribute manipulation via projection.

##  3 Method

We follow the hierarchical labels defined by [21]. For a single image, its
attribute for tag $i\in\\{1,2,...,N\\}$ can be defined as
$j\in\\{1,2,...,M_{i}\\}$, where N is the number of tags and $M_{i}$ is the
number of attributes for tag $i$. For example $i$ can be tag of hair color,
and attribute $j$ can take the value of black, brown, or blond.

Our framework has two main objectives. As the main task, we aim to be able to
perform the image-to-image translation task in a feature (tag) specific
manner. While performing this translation, as the second objective, we also
want to obtain an interpretable feature space which allows us to perform tag-
specific feature interpolation.

###  3.1 Generator Architecture

For image to image translation task, we set an encoder-decoder based
architecture and latent space translation in the middle as given in Fig. 2. We
perform the translation in the encoded latent space, $e$, which is obtained by
$e=E(x)$ where $E$ refers to the encoder. The encoded features go through a
transformation $T$ which is discussed in the next section. The transformed
features are then decoded by $G$ to reconstruct the translated images. The
image generation pipeline following feature encoding is described in Eq. 1.

$\displaystyle e^{\prime}=T(e,\alpha,i)$ $\displaystyle
x^{\prime}=G(e^{\prime})$ (1)

Previous image-to-image translation networks [21, 38, 7] set a shallow encoder
decoder architecture to translate an image and a separate deep network for
style encoding. In most cases, the style encoder includes separate branches
for each tag. The shallow architecture that is used to translate images
prevents the model from making drastic changes in the images and this helps
preserving the identity of the persons. Our framework is different as we do
not employ a separate style encoder and instead have a deep encoder-decoder
architecture for translation. That is because to be able to organize the
latent space in an interpretable way, our framework requires a full
understanding of the image and therefore a larger receptive field; deeper
network architecture. A deep architecture with decreasing size of feature
size, on the other hand, faces the challenges of reconstructing all the fine
details from the input image.

With the motivation of helping the network to preserve tag independent
features such as the fine details from background, we use skip connections
between our encoder and decoder. However, we observe that the flow of
information should be limited to force the encoder-decoder architecture learn
facial attributes and well-organized latent representations. Because of that
reason, we only allow skip connection at low resolution. This design is
extensively justified in our Ablation Studies.

Figure 2: VecGAN pipeline. Our translator is built on the idea of
interpretable latent directions. We encode images with an Encoder to a latent
representation from which we change a selected tag ($i$), e.g. hair color with
a learnable direction $A_{i}$ and a scale $\alpha$. To calculate the scale, we
subtract the target style scale from the source style. This operation
corresponds to removing an attribute and adding an attribute. To remove the
image’s attribute, source style is encoded and projected from the source
image. To add the target attribute, target style scale is sampled from a
distribution mapped for the given attribute ($j$), e.g. blonde, brown or
encoded and projected from a reference image.

###  3.2 Translation Module

To achieve a style transformation, we perform the tag-based feature
manipulation in a linear fashion in the latent space. First, we set a feature
direction matrix $A$ which contains learnable feature directions for each tag.
In our formulation $A_{i}$ denotes the learned feature direction for tag $i$.
Direction matrix $A$ is randomly initialized and learned during the training
process.

Our translation module is formulated in Eq. 2, which adds the desired shift on
top of the encoded features $e$ similar to [31].

$T(e,\alpha,i)=e+\alpha\times A_{i}$ (2)

We compute the shift by subtracting target style from the source style as
given in Eq 3.

$\alpha=\alpha_{t}-\alpha_{s}$ (3)

Since the attributes are designed as linear steps in the learnable directions,
we find the style shift by subtracting the target attribute scale from source
attribute scale. This way the same target attribute $\alpha_{t}$ can have the
same impact on the translated images no matter what the attributes were of the
original images. For example, if our target scale corresponds to brown hair,
the source scale can be coming from an image with blonde or back hair but
since we take a step for difference of the scales, they can be both translated
to an image with the same shade of brown hair.

To extract the target shifting scale for feature (tag) $i$, $\alpha_{t}$,
there are two alternative pathways. The first pathway, named as latent-guided
path, samples a $z\in\mathcal{U}[0,1)$ and applies a linear transformation
$\alpha_{t}=w_{i,j}\cdot z+b_{i,j}$, where $\alpha_{t}$ denotes sampled
shifting scale for tag $i$ and attribute $j$. Here tag $i$ can be hair color
and attribute $j$ can be blonde, brown, or back hair. For each attribute we
learn a different transformation module which is denoted as $M_{i,j}(z)$.
Since we learn a single direction for every tag for example for hair color,
this transformation module can put the initially sampled $z$’s into correct
scale in the linear line based on the target hair color attribute. As the
other alternative pathway, we encode the scalar value $\alpha_{t}$ in a
reference-guided manner. We extract $\alpha_{t}$ for tag $i$ from a provided
reference image by first encoding it into the latent space, $e_{r}$, and
projecting $e_{r}$ via by $A_{i}$ as given in Eq. 4.

$\alpha_{t}=P(e_{r},A_{i})=\dfrac{e_{r}\cdot A_{i}}{||A_{i}||}$ (4)

In the reference guidance set-up, we do not use the information of attribute
$j$, since it is encoded by the tag $i$ features of the image.

The source scale, $\alpha_{s}$, is obtained by the same way we obtain
$\alpha_{t}$ from reference image. We perform the projection for the
corresponding tag we want to manipulate, $i$, by $P(e,A_{i})$. We formulate
our framework with the intuition that the scale controls the amount of feature
to be added. Therefore, especially when the attribute is copied over from a
reference image, the amount of features that will be added will be different
based on the source image. It is for this reason, we find the amount of shift
by subtraction as given in Eq. 3. Our framework is intuitive and relies on a
single encoder-decoder architecture. Fig. 2 shows the overall pipeline.

###  3.3 Training pathways

Modifying the translation paths defined by [21], we train our network using
two different paths. For each iteration to optimize our model, we sample a tag
$i$ for shift direction, a source attribute $j$ as the current attribute and a
target attribute $\hat{j}$.

Non-translation path. To ensure that the encoder-decoder structure preserves
details of the images, we perform a reconstruction of the input image without
applying any style shifts. The resulting image is denoted as $x_{n}$ as given
in Eq. 5.

$x_{n}=G(E(x))$ (5)

Figure 3: Overview of cycle translation path.

Cycle-translation path. We apply a cyclic translation to ensure that we get a
reversible translation from a latent guided scale. In this path, as shown in
Fig. 3, we first apply a style shift by sampling $z\in\mathcal{U}[0,1)$ and
obtaining target $\alpha_{t}$ with $M_{i,\hat{j}}(z)$ for target attribute
$\hat{j}$. The translation uses $\alpha$ that is obtained by subtracting
$\alpha_{t}$ from the source style. Decoder generates an image, $x_{t}$ as
given in Eq. 6 where $e$ is encoded features from input image $x$, $e=E(x)$.
$x_{t}$ refers to the image without glasses in Fig. 3.

$\displaystyle x_{t}=G(T(e,M_{i,j}(z)-P(e,i),i))$ (6)

Then by using the original image, $x$, as a reference image, we aim to
reconstruct the original image by translating $x_{t}$. Overall, this path
attempts to reverse a latent-guided style shift with a reference-guided shift.
The second translation is given in Eq. 7 where $e_{t}=E(x_{t})$.

$\displaystyle x_{c}=G(T(e_{t},P(e,i)-P(e_{t},i),i))$ (7)

In our learning objectives, we use $x_{n}$ and $x_{c}$ for reconstruction and
$x_{t}$ and $x_{c}$ for adversarial losses, and $M_{i,j}(z)$ for the shift
reconstruction loss. Details about the learning objectives are given in the
next section.

###  3.4 Learning objectives

Given an input image $x_{i,j}\in\mathcal{X}_{i,j}$, where $i$ is the tag to
manipulate and $j$ is the current attribute of the image, we optimize our
model with the following objectives. In our equations, $x_{i,j}$ is shown as
$x$.

Adversarial Objective. During training, our generator performs a style-shift
either in a latent-guided way or a reference-guided way, which results in a
translated image. In our adversarial loss, we receive feedback from the two
steps of cycle-translation path. As the first component of the adversarial
loss, we feed a real image $x$ with tag $i$ and attribute $j$ to the
discriminator as the real example. To give adversarial feedback to latent-
guided path, we use the intermediate image generated in cycle-translation
path, $x_{t}$. Finally, to provide adversarial feedback to reference-guided
path, we use the final outcome of the cycle-translation path $x_{c}$. Only $x$
acts as real image, both $x_{t}$ and $x_{c}$ are translated images, and they
are treated as fake images with different attributes. The discriminator aims
at classifying whether an image, given its tag and attribute, is real or not.
The objective is given in Eq 8.

$\begin{split}\mathcal{L}_{adv}=2log(D_{i,j}(x))+log(1-D_{i,\hat{j}}(x_{t}))+log(1-D_{i,j}(x_{c}))\end{split}$
(8)

Shift Reconstruction Objective. As the cycle-consistency loss performs
reference-guided generation followed by latent-guided generation, we utilize a
loss function to make these two methods consistent with each other [19, 14,
20, 21]. Specifically, we would like to obtain the same target scale,
$\alpha_{t}$, both from the mapping and from the encoded reference image
generated by the mapped $\alpha_{t}$. The loss function is given in Eq. 9.

$\mathcal{L}_{shift}=||M_{i,j}(z)-P(e_{t},i)||_{1}$ (9)

Those parameters, $M_{i,j}(z)$ and $P(e_{t},i)$, are calculated for the cycle-
translation path as given in Eq. 6 and 7.

Image Reconstruction Objective. In all of our training paths, the purpose it
to be able to re-generate the original image again. To supervise this desired
behavior, we use $L_{1}$ loss for reconstruction loss. In our formulation
$x_{n}$ and $x_{c}$ are outputs of non-translation path and cycle-translation
path, respectively. Formulation of this objective is provided in Eq. 10.

$\begin{split}\mathcal{L}_{rec}=||x_{n}-x||_{1}+||x_{c}-x||_{1}\end{split}$
(10)

Orthogonality Objective. To encourage the orthogonality between directions, we
use soft orthogonality regularization based on Frobenius norm, which is given
in Eq. 11. This orthogonality further encourages a disentanglement in the
learned style directions.

$\mathcal{L}_{ortho}={\|A^{T}A-I\|_{F}}$ (11)

Full Objective. Combining all of the loss components described, we reach to
the overall objective for optimization as given in Eq. 12. We additionally add
L1 loss on the matrix $A$ parameters to encourage its sparsity.

$\begin{split}\underset{E,G,M,A}{\min}\underset{D}{\max}\lambda_{a}\mathcal{L}_{adv}+\lambda_{s}\mathcal{L}_{shift}+\lambda_{r}\mathcal{L}_{rec}+\lambda_{o}\mathcal{L}_{ortho}+\lambda_{sp}\mathcal{L}_{sparse}\end{split}$
(12)

To control the dominance of each loss component, we use
$\lambda_{a},\lambda_{s}$, $\lambda_{r}$, $\lambda_{o}$, and $\lambda_{sp}$
hyperparameters. These hyperparameter values and training details are given in
Supplementary.

##  4 Experiments

###  4.1 Dataset and Settings

We train our model on CelebA-HQ dataset [24] which contains 30,000 face
images. To extensively compare with state-of-the-arts, we follow two training-
evaluation protocols as follows:

Setting A. In our first setting, we follow the set-up from HiSD [21].
Following HiSD, we use the first 3000 images of CelebA-HQ dataset as the test
set and 27000 as the training set. These images include annotations for
different attributes from which we use hair color, presence of glass, and
bangs attributes for translation task in this setting. Hair color attribute
includes 3 tags, black, brown, and blonde whereas the other attributes are
binary. The images are resized to $128\times 128$. Following the evaluation
protocol proposed by HiSD [21], we compute FID scores on bangs addition task.
For each test image without bangs, we translate them to images with bangs with
latent and reference guidance. In latent guidance, 5 images are generated for
each test image by randomly sampling scale from a uniform distribution. Then
this generated set of images are compared with images that have attribute
bangs in terms of their FIDs. FIDs are calculated for these 5 sets and
averaged. For reference guidance, we randomly pick 5 references images to
extract the style scale. FIDs are calculated for these 5 sets separately and
averaged.

Setting B. In this setting, we follow the set-up from L2M-GAN [38]. The
training/test split is obtained by re-indexing each image in CelebA-HQ back to
the original CelebA and following the standard split of CelebA. This results
in 27,176 training and 2,824 test images. Models are trained for hair color,
presence of glasses, bangs, age, smiling, and gender attributes. Images are
resized to $256\times 256$ resolution. For evaluation, smiling attribute is
used following L2M-GAN [38]. It is noted that smiling is one of the most
challenging among the CelebA facial attributes because adding/removing a smile
requires high-level understanding of the input face image for modifying
multiple facial components simultaneously. FIDs are calculated for adding and
removing the smile attribute.

###  4.2 Results

We extensively compare our results with other competing methods in Table 1. In
Setting A, as given in Table 1(a), we compare with SDIT [34], StarGANv2 [7],
Elegant [37], and HiSD [21] models. Among these methods, HiSD learns a
hierarchical style disentanglement whereas StarGANv2 learns a mixed style
code. Therefore, StarGANv2 when translating images also does other unnecessary
manipulations and does not strictly preserve the identity. Our work is most
similar to HiSD as we also learn disentangled style directions. However, HiSD
learns feature based local translators which is an approach known to be
successful on local edits, e.g. bangs. Ours results show that VecGAN achieves
significantly better quantitative results than HiSD both in latent guided and
reference guided evaluations even though they are compared on a local edit
task.

Method | Lat. | Ref.
---|---|---
SDIT [34] | 33.73 | 33.12
StarGANv2 [7] | 26.04 | 25.49
Elegant [37] | - | 22.96
HiSD [21] | 21.37 | 21.49
VecGAN (Ours) | 20.17 | 20.72
(a) Quantitative results for Setting A. Lat: Latent guided, Ref: Reference
guided. FID scores are given. Lower is better.

Method | FID (+) | FID (-) | Avg
---|---|---|---
StarGAN [6] | 32.6 | 38.6 | 35.6
CycleGAN [44] | 22.5 | 24.4 | 23.5
Elegant [37] | 39.7 | 42.9 | 41.3
PA-GAN [12] | 20.5 | 21.4 | 21.0
InterFaceGAN [28] | 24.8 | 24.9 | 24.9
L2M-GAN [38] | 17.9 | 23.3 | 20.6
VecGAN (Ours) | 17.7 | 20.3 | 19.0
(b) Quantitative results for Setting B. FID (+) (or FID (-)) denotes the FID
score for adding (or removing) a smile.

Table 1: Comparisons with state-of-the-art competing methods. Please refer to Section 4 for details on training and evaluation protocol of Setting A and B. | Smiling (+) | Smiling (-) | Smiling (Avg)
---|---|---|---
Comparisons | Quality | Fidelity | Quality | Fidelity | Quality | Fidelity
VecGAN (Ours) vs L2M-GAN | 57.96% | 70.94% | 60.93% | 77.50% | 59.45% | 74.22%
VecGAN (Ours) vs InterFaceGAN | 88.13% | 91.56% | 77.50% | 90.62% | 82.82% | 91.09%
Table 2: User study results conducted with smiling attribute. Smiling (+)
denotes the results of adding a smile, Smiling (-) refers to the results of
removing a smile, and Smiling (avg) denotes the average of Smiling (+) and
Smiling (-). Percentages show the preference rates of our method versus the
other competing method. Figure 4: Qualitative results of bangs attribute of
our model (VecGAN) and HiSD. In the second example, we provide a very
challenging sample where VecGAN even though not perfect achieves significantly
better results than HiSD. Figure 5: Qualitative results of smile attribute of
our model (VecGAN), L2M-GAN, and InterFaceGAN. The first four examples show
smile addition and the other four shows smile removal manipulations.

Fig. 4 shows reference guided results of our model versus HiSD. We compare
with HiSD since it provides with the best results after ours. As can be seen
from Fig. 4, both methods achieve attribute disentanglement, they do not
change any other attribute of the image than the bangs tag. However, HiSD
outputs artifacts especially for the reference image from the last column. On
the other hand, VecGAN outputs higher quality results. As the second example,
we pick a very challenging example to compare these methods. Even though, our
results can be further improved to look more realistic, it achieves
significantly better outputs with no artifacts compared to HiSD.

In our second set-up of evaluation, we compare our method with many state-of-
the-art methods as given in Table 1(b). We compare with StarGAN [6], CycleGAN
[44], Elegant [37], PA-GAN [12], InterFaceGAN [28], and L2M-GAN [38]. For
InterFaceGAN, we use the GAN Inversion [43] as the encoder and pretrained
StyleGAN [17] as the generator backbone. As can be seen from Table 1(b), we
achieve significantly better scores on both settings and in average.

In our visual comparisons, we mainly focus on L2M-GAN and InterFaceGAN since
L2M-GAN is the second best model after ours and InterFaceGAN shares the same
intuition with our model and performs edits by latent code manipulation. The
results are shown in Fig. 5 where the first four examples show smile addition
and the other four examples show smile removal manipulations. The most
prominent limitation of L2M-GAN and InterFaceGAN is that they do not preserve
the other attributes of images, especially on the background whereas VecGAN
does a very good job at that. Smile attribute addition and removal of L2M-GAN
is better than InterFaceGAN, however, worse than ours. VecGAN is the only
method among them that can produce manipulated images with high fidelity to
the originals with only targeted attribute manipulated in a natural and
realistic way.

We also conduct a user study on the first 64 images of validation set among 10
users. We set an A/B test and provide users with input images and translations
obtained by VecGAN and other competing methods. The left-right order is
randomized to ensure fair comparisons. We perform two separate tests. 1)
Quality: We ask users to select the best result according to i) whether the
smile attribute is correctly added, ii) whether irrelevant facial attributes
preserved, iii) and overall whether the output image looks realistic and high
quality. 2) Fidelity: We ask users to pay attention if details from the input
image is preserved in addition to the quality. When only asked for quality,
users pay attention to facial attributes and do not pay much attention to the
background, ornament, details of hair of the image, and so on. In this test,
we remind the users to pay attention to those as well. Table 2 shows the
results of the user study. Users preferred our method as opposed to L2M-GAN
$59.45\%$ of the time ($50\%$ is tie), and as opposed to InterFaceGAN
$82.82\%$ of the time for the quality measure in average of smile addition and
removal results. When users asked to pay attention to non-facial attributes as
well, they preferred our method as opposed to L2M-GAN $74.22\%$ of the time,
and as opposed to InterFaceGAN $91.09\%$ of the time in average.

###  4.3 Ablation Study

We conduct ablation studies for network architecture and loss objectives as
given in Table 7. We first experiment with a shallower architecture where
encoder decreases the input dimension of $128\times 128$ to a spatial
dimension of $8\times 8$. This version gives reasonable scores, however, we
are interested in a better latent space organization. For that, we use a
deeper encoder-decoder architecture where encoded latent space goes as low as
$1\times 1$ which we refer as deep architecture. Deep architecture without
skip connections is not able to minimize the reconstruction objective and
results in a high FID. On the other hand, deep architecture with a skip
connection at each resolution from encoder to decoder can minimize the
reconstruction loss however the latent space is not well organized since the
model tends to pass all the information from the encoder which instabilizes
the training. Our architecture with single skip layer at resolution $32\times
32$ provides a good balance between the information flow from encoder-decoder
and the latent space bottleneck.

Figure 6: Qualitative results of ablation study of orthogonality loss. Bangs
tag transferred from the reference image.

Method | Lat. | Ref.
---|---|---
Shallow | 21.30 | 20.94
Deep w/o skip | 88.62 | 127.65
Deep all skip | 273.80 | 273.97
Ours | 20.17 | 20.72
w/o Orthogonality | 21.98 | 22.50
w/o Sparsity | 24.07 | 22.43

Figure 7: FID results of ablation study with Setting A. Lat: Latent guided,
Ref: Reference guided.

Next, we experiment the effect of loss functions. First, we remove the
orthogonality loss of $A$ directions. This results in worse FID scores but
more importantly we observe that the styles are not disentangled, e.g.
changing bangs attribute changes the gender as can be seen in Fig. 7. Even
without this loss function, we observe that during training the orthogonality
loss of $A$ decreases but to a higher value than when this loss is added to
the final objective. That is because the framework and other loss objectives
also encourage the disentanglement of attribute manipulations and it shows in
the orthogonality of direction vectors. This also shows the importance of
orthogonality in style disentanglement and this targeted loss helps improve
that significantly. We also observe that sparsity loss applied on the
directional vectors stabilizes the training and without that FIDs are much
higher.

Figure 8: Results of changing the strength of a manipulation gradually. Each
example shows a different attribute manipulation. Rows show bangs, hair color,
gender, smile, glasses, and age manipulations in this order.

###  4.4 Other Capabilities of VecGAN

Gradually Increased Scale. We translate images with gradually increased
attribute strength as shown in Fig. 8. We plot the manipulation results on six
different attributes. These results show that attributes that are designed as
linear transformations are disentangled, and changing one attribute does not
affect the other components. In these results, as scales are gradually
increased, the strength of the tag smoothly increases with the identity of the
person preserved.

Multi-tag Edits. We additionally experiment with multi-tag manipulation. To
change two attributes, instead of encoding and decoding the image twice with a
translation in between each time, we perform two translation operations in the
latent code simultaneously. That is we apply Eq. 2 twice for two different
$i$. Fig. 9(a) shows results of the multi-tag edits. In the first row, we
consider gender and smile tags, and first edit those attributes individually.
In the last coloumn, we edit the image with these two tags simultaneously. The
second row shows a similar experiment with smile and age tags. We observe that
VecGAN provides with disentangled tag control and can successfully edit tags
independently.

(a) Multi-attribute editing results.

(b) Generalization results.

Figure 9: Results of multi-attribute editing and cross-dataset generalization
results of VecGAN.

Generalization to other domains. We apply VecGAN model to MetFace dataset [16]
without any retraining. The results are provided in Fig. 9(b). The first row
shows source images, and the second row shows outputs of our model. In the
first two examples, we increase the smile attribute, and in the other two, we
decrease it. The results show that VecGAN has a good generalization ability
and works reasonably well across datasets.

##  5 Conclusion

This paper introduces VecGAN, an image-to-image translation framework with
interpretable latent directions. This framework includes a deep encoder and
decoder architecture with latent space manipulation in between. Latent space
manipulation is designed as vector arithmetic where for each attribute, a
linear direction is learned. This design is encouraged by the finding that
well-trained generative models organize their latent space as disentangled
representations with meaningful directions in a completely unsupervised way.
Each change in the architecture and loss functions is extensively studied and
compared with state-of-the-arts. Experiments show the effectiveness of our
framework.

##  Appendix 0.A More comparisons

Figure 10: Qualitative results of smile attribute of our model (VecGAN) and
other StyleGAN based models.

In Fig. 10, we compare our method with other methods that are proposed to
invert images to StyleGANv2 space and perform edits via the pretrained
StyleGANv2. We compare with e4e [30], HyperStyle [3], and HFGI [32]. Same
input examples are used from Fig. 5 main paper. e4e as also stated in their
paper outputs results with worse distortion (input-output similarity) but
better edits. HyperStyle and HFGI are concurrent works with improved fidelity
to the input image but still significantly worse than our method both in edit
quality and reconstruction quality of the input details.

Figure 11:  Qualitative results of smile attribute of our model (VecGAN) and
other StyleGAN based editing models.

We additionally compare with StyleFlow [2] and StyleSpace [36] in Fig. 11. For
both examples, we take their real image editing example from their papers and
feed the input crops to VecGAN for comparison. As can be seen from Fig. 11,
both methods suffer from the limitations of the projection method as inputs
are not faithfully reconstructed. Additionally, the edit is not perfectly
disentangled in StyleFlow example as the strap of the top changes when smile
is modified. VecGAN achieves significantly better results in these examples.

##  Appendix 0.B Additional Quantitative Results

Method | KID(+) | KID(-) | KID (Avg)
---|---|---|---
L2M-GAN | 0.01010 | 0.00942 | 0.00976
InterfaceGAN | 0.00603 | 0.00671 | 0.00637
VecGAN | 0.00188 | 0.00328 | 0.00258
Table 3: Quantitative results for Setting B - Smile attribute.

In Table 3, we compare VecGAN and other competing methods with KID metric [4].
Same as in FID evaluation, VecGAN achieves significantly better results.

##  Appendix 0.C Model Architecture

In this section, we provide architectural details of VecGAN.

#### Generator.

Our generator is composed of an encoder and decoder as shown in Fig. 2. For
encoder, we use 8 successive blocks that perform downsampling which reduce
feature map dimensions to 1x1. In our decoder, we have an architecture
symmetric to encoder, which is composed of 8 successive upsampling blocks.
Except the last downsampling block and the first upsampling block, we use
instance normalization denoted as (+IN). The channels increase as {64, 64,
128, 256, 512, 512, 512, 1024, 2048} (for output resolution 256x256) in the
encoder and decrease in a symmetric way in the decoder. In addition to these
building blocks, we use a skip connection between the encoder and decoder as
shown in Fig. 12.

Figure 12: Generator architecture. Numbers correspond to the output channels
of each block.

#### Residual Blocks.

Each DownBlock and UpBlock has a residual block with $3x3$ convolutional
filters followed by a downsampling and upsampling layer, respectively. For
downsampling, we use average pooling and for upsampling, we use nearest-
neighbor. We use LeakyReLU activation layer and instance normalization layer
in each convolutional module.

#### Discriminator.

Discriminator also employs an architecture with decreasing resolution and
increasing channel size as given in Fig. 13. Just like the generator, we build
our discriminator with channel sizes of {64, 64, 128, 256, 512, 512, 512,
1024, 2048}, that reduces the feature map dimensions to 1x1. At the end, we
concatenate the extracted style $\alpha_{t}$ from the input image to this
latent code and apply a 1x1 convolution. This final convolution is specific to
each tag-attribute pair so that the model can use this information.

Figure 13: Architecture of the discriminator. Discriminator takes an input
image and processes it with downsampling blocks with increased number of
channels. Towards the end, the extracted feature map with 1x1 feature
dimensions is concatenated with the scale of the input image. As we perform
scale extraction for the image in the cycle-translation path, no additional
scale extraction is needed.

#### Hyperparameters.

For training our framework, we set the following parameters; $\lambda_{a}=1$,
$\lambda_{rec}=1.5$, $\lambda_{s}=1$, $\lambda_{o}=1$ and $\lambda_{sp}=0.05$.
We use a learning rate of $10^{-4}$ and train our model for 500K iterations
with a batch size of 8 on a single GPU. For the feature encoding and feature
directions in matrix $A$, we use a 2048 dimensional vector representation same
as the channel size of the last convolutional layer from the encoder.

##  Appendix 0.D Additional Results

We provide additional qualitative results of our method in Fig. 14, 15, 16,
17, 18, and 19.

Figure 14: Smile tag manipulation results. First and third rows show input
images. Second and forth rows show image translation results. Figure 15:
Glasses tag manipulation results. First and third rows show input images.
Second and forth rows show image translation results. Figure 16: Gender tag
manipulation results. First and third rows show input images. Second and forth
rows show image translation results. Figure 17: Bangs tag manipulation
results. First and third rows show input images. Second and forth rows show
image translation results. Figure 18: Age tag manipulation results. First and
third rows show input images. Second and forth rows show image translation
results. Figure 19: Hair tag manipulation results. First and third rows show
input images. Second and forth rows shows image translation results.

## References

  * [1] Abdal, R., Qin, Y., Wonka, P.: Image2stylegan: How to embed images into the stylegan latent space? In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4432–4441 (2019)
  * [2] Abdal, R., Zhu, P., Mitra, N.J., Wonka, P.: Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics (ToG) 40(3), 1–21 (2021)
  * [3] Alaluf, Y., Tov, O., Mokady, R., Gal, R., Bermano, A.: Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18511–18521 (2022)
  * [4] Bińkowski, M., Sutherland, D.J., Arbel, M., Gretton, A.: Demystifying mmd gans. arXiv preprint arXiv:1801.01401 (2018)
  * [5] Brock, A., Donahue, J., Simonyan, K.: Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018)
  * [6] Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8789–8797 (2018)
  * [7] Choi, Y., Uh, Y., Yoo, J., Ha, J.W.: Stargan v2: Diverse image synthesis for multiple domains. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2020)
  * [8] Dundar, A., Sapra, K., Liu, G., Tao, A., Catanzaro, B.: Panoptic-based image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8070–8079 (2020)
  * [9] Gao, Y., Wei, F., Bao, J., Gu, S., Chen, D., Wen, F., Lian, Z.: High-fidelity and arbitrary face editing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16115–16124 (2021)
  * [10] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Advances in neural information processing systems 27 (2014)
  * [11] Härkönen, E., Hertzmann, A., Lehtinen, J., Paris, S.: Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems 33 (2020)
  * [12] He, Z., Kan, M., Zhang, J., Shan, S.: Pa-gan: Progressive attention generative adversarial network for facial attribute editing. arXiv preprint arXiv:2007.05892 (2020)
  * [13] Hou, X., Zhang, X., Liang, H., Shen, L., Lai, Z., Wan, J.: Guidedstyle: Attribute knowledge guided style manipulation for semantic face editing. Neural Networks 145, 209–220 (2022)
  * [14] Huang, X., Liu, M.Y., Belongie, S., Kautz, J.: Multimodal unsupervised image-to-image translation. Eur. Conf. Comput. Vis. (2018)
  * [15] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Conf. Comput. Vis. Pattern Recog. (2017)
  * [16] Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., Aila, T.: Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems 33, 12104–12114 (2020)
  * [17] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4401–4410 (2019)
  * [18] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8110–8119 (2020)
  * [19] Lee, H.Y., Tseng, H.Y., Huang, J.B., Singh, M., Yang, M.H.: Diverse image-to-image translation via disentangled representations. In: Proceedings of the European conference on computer vision (ECCV). pp. 35–51 (2018)
  * [20] Li, X., Hu, J., Zhang, S., Hong, X., Ye, Q., Wu, C., Ji, R.: Attribute guided unpaired image-to-image translation with semi-supervised learning. arXiv preprint arXiv:1904.12428 (2019)
  * [21] Li, X., Zhang, S., Hu, J., Cao, L., Hong, X., Mao, X., Huang, F., Wu, Y., Ji, R.: Image-to-image translation via hierarchical style disentanglement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8639–8648 (2021)
  * [22] Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Adv. Neural Inform. Process. Syst. (2017)
  * [23] Liu, M.Y., Tuzel, O.: Coupled generative adversarial networks. Advances in neural information processing systems 29, 469–477 (2016)
  * [24] Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV) (December 2015)
  * [25] Mardani, M., Liu, G., Dundar, A., Liu, S., Tao, A., Catanzaro, B.: Neural ffts for universal texture image synthesis. Advances in Neural Information Processing Systems 33, 14081–14092 (2020)
  * [26] Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2337–2346 (2019)
  * [27] Shen, W., Liu, R.: Learning residual images for face attribute manipulation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4030–4038 (2017)
  * [28] Shen, Y., Gu, J., Tang, X., Zhou, B.: Interpreting the latent space of gans for semantic face editing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9243–9252 (2020)
  * [29] Shen, Y., Zhou, B.: Closed-form factorization of latent semantics in gans. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1532–1540 (2021)
  * [30] Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG) 40(4), 1–14 (2021)
  * [31] Voynov, A., Babenko, A.: Unsupervised discovery of interpretable directions in the gan latent space. In: International Conference on Machine Learning. pp. 9786–9796. PMLR (2020)
  * [32] Wang, T., Zhang, Y., Fan, Y., Wang, J., Chen, Q.: High-fidelity gan inversion for image attribute editing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11379–11388 (2022)
  * [33] Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional gans. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8798–8807 (2018)
  * [34] Wang, Y., Gonzalez-Garcia, A., van de Weijer, J., Herranz, L.: Sdit: Scalable and diverse cross-domain image translation. In: Proceedings of the 27th ACM International Conference on Multimedia. pp. 1267–1276 (2019)
  * [35] Wu, P.W., Lin, Y.J., Chang, C.H., Chang, E.Y., Liao, S.W.: Relgan: Multi-domain image-to-image translation via relative attributes. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 5914–5922 (2019)
  * [36] Wu, Z., Lischinski, D., Shechtman, E.: Stylespace analysis: Disentangled controls for stylegan image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12863–12872 (2021)
  * [37] Xiao, T., Hong, J., Ma, J.: Elegant: Exchanging latent encodings with gan for transferring multiple face attributes. In: Proceedings of the European conference on computer vision (ECCV). pp. 168–184 (2018)
  * [38] Yang, G., Fei, N., Ding, M., Liu, G., Lu, Z., Xiang, T.: L2m-gan: Learning to manipulate latent space semantics for facial attribute editing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2951–2960 (2021)
  * [39] Yi, R., Liu, Y.J., Lai, Y.K., Rosin, P.L.: Apdrawinggan: Generating artistic portrait drawings from face photos with hierarchical gans. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10743–10752 (2019)
  * [40] Yi, R., Liu, Y.J., Lai, Y.K., Rosin, P.L.: Unpaired portrait drawing generation via asymmetric cycle mapping. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8217–8225 (2020)
  * [41] Yi, Z., Zhang, H., Tan, P., Gong, M.: Dualgan: Unsupervised dual learning for image-to-image translation. In: Int. Conf. Comput. Vis. (2017)
  * [42] Zhang, G., Kan, M., Shan, S., Chen, X.: Generative adversarial network with spatial attention for face attribute editing. In: Proceedings of the European conference on computer vision (ECCV). pp. 417–432 (2018)
  * [43] Zhu, J., Shen, Y., Zhao, D., Zhou, B.: In-domain gan inversion for real image editing. In: European conference on computer vision. pp. 592–608. Springer (2020)
  * [44] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Int. Conf. Comput. Vis. (2017)
  * [45] Zhu, J.Y., Zhang, R., Pathak, D., Darrell, T., Efros, A.A., Wang, O., Shechtman, E.: Multimodal image-to-image translation by enforcing bi-cycle consistency. In: Advances in neural information processing systems. pp. 465–476 (2017)
  * [46] Zhu, P., Abdal, R., Qin, Y., Wonka, P.: Sean: Image synthesis with semantic region-adaptive normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5104–5113 (2020)