# CFASL: Composite Factor-Aligned Symmetry Learning for Disentanglement in
Variational AutoEncoder

Hee-Jun Jung1 Jaehyoung Jeong1&Kangil Kim1111corresponding author 1Gwangju
Institute of Science and Technology
{jungheejun93<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Symmetries of input and latent vectors have provided valuable insights for
disentanglement learning in VAEs. However, only a few works were proposed as
an unsupervised method, and even these works require known factor information
in training data. We propose a novel method, Composite Factor-Aligned Symmetry
Learning (CFASL), which is integrated into VAEs for learning symmetry-based
disentanglement in unsupervised learning without any knowledge of the dataset
factor information. CFASL incorporates three novel features for learning
symmetry-based disentanglement: 1) Injecting inductive bias to align latent
vector dimensions to factor-aligned symmetries within an explicit learnable
symmetry codebook 2) Learning a composite symmetry to express unknown factors
change between two random samples by learning factor-aligned symmetries within
the codebook 3) Inducing group equivariant encoder and decoder in training
VAEs with the two conditions. In addition, we propose an extended evaluation
metric for multi-factor changes in comparison to disentanglement evaluation in
VAEs. In quantitative and in-depth qualitative analysis, CFASL demonstrates a
significant improvement of disentanglement in single-factor change, and multi-
factor change conditions compared to state-of-the-art methods.

##  1 Introduction

Disentangling representations by intrinsic factors of datasets is a crucial
issue in machine learning literature Bengio et al. (2013). In Variational
Autoencoder (VAE) frameworks, a prevalent method to handle the issue is to
factorize latent vector dimensions to encapsulate specific factor information
Kingma and Welling (2013); Higgins et al. (2017); Chen et al. (2018); Kim and
Mnih (2018); Jeong and Song (2019); Shao et al. (2020, 2022). Although their
effective disentanglement learning methods, Locatello et al. (2019) raises the
serious difficulty of disentanglement without sufficient inductive bias.

In VAE literature, recent works using group theory offer a possible solution
to inject such inductive bias by decomposing group symmetries Higgins et al.
(2018) in the latent vector space. To implement group equivariant VAE, Winter
et al. (2022a); Nasiri and Bepler (2022) achieve the translation and rotation
equivariant VAE. The other branch implements the group equivariant function
Yang et al. (2022); Keller and Welling (2021a) over the pre-defined group
elements. All of the methods effectively improve disentanglement by adjusting
symmetries, but they focused on learning symmetries among observations to
inject inductive bias rather than factorizing group elements to align them on
a single factor and a single dimension changes, as introduced in the
definition provided in Higgins et al. (2018).

In current works, unsupervised learning approaches of group equivariant models
are introduced. Miyato et al. (2022); Quessard et al. (2020) represent the
symmetries on the latent vector space, which correspond to the symmetries on
the input space, by considering the sequential observations. Also, Winter et
al. (2022b) proposes the group invariant and equivariant representations with
different modules to learn the different groups of dataset structure. However,
these approaches, despite being unsupervised learning, require the factor
information of the dataset to construct the sequential input and to set
different modules for learning symmetries.

This paper introduces a novel disentanglement method for Composite Factor-
Aligned Symmetry Learning (CFASL) within VAE frameworks, aimed at addressing
the challenges encountered in unsupervised learning scenarios, particularly
the absence of explicit knowledge about the factor structure in datasets. Our
methodology follow as 1) a network architecture to learn an explicit codebook
of symmetries, responsible for each single factor change, called factor-
aligned symmetries, 2) training losses to inject inductive bias to be an
explicit codebook where each factor-aligned symmetry only impacts to a single
dimension value of latent vectors for disentangled representations, 3)
learning composite symmetries by prediction of single factor changes itself
without information of factor labels for unsupervised learning, 4)
implementing group equivariant encoder and decoder functions that factor-
aligned symmetry affects latent vector space, 5) an extended metric (m-FVMk)
to evaluate disentanglement in the multi-factor change condition. We conduct
quantitative and qualitative analyses of our method onn common benchmarks of
disentanglement in VAEs.

(a) $\beta$-VAE

(b) $\beta$-TCVAE

(c) Control-VAE

(d) CLG-VAE

(e) G-VAE

(f) CFASL (ours)

(g) Ideal

Figure 1: Distribution of latent vectors for dimensions responsible for Shape,
X-pos, Y-pos factors in the dSprites dataset. The groupified-VAE method is
applied to $\beta$-TCVAE because this model shows a better evaluation score.
The results show disentanglement for Shape from the combination of the other
two factors by coloring three shapes (square, ellipse, heart) as red, blue,
and green color, respectively. Each 3D-plot shows the whole distribution. We
fix Scale and Orientation factor values, and plot randomly sampled 640 inputs
(20.8$\%$ of all possible observations ($32\times 32\times 3=3,072$)). We
select the dimensions responsible for the factors by selecting the largest
value of the Kullback-Leibuler divergence between the prior and the posterior.

##  2 Limits of Disentanglement Learning of VAE

By the definition of disentangled representation Bengio et al. (2013); Higgins
et al. (2018), the disentangled representations distribute on the flattened
surface as shown in Fig. 1(g) because each change of the factor only affects
to single dimension of latent vector. However, the previous methods Higgins et
al. (2017); Chen et al. (2018); Shao et al. (2020); Zhu et al. (2021); Yang et
al. (2022) show the entangled representations on their latent vector space as
shown in Fig. 1(a)-1(c). Even though the group theory-based methods improve
the disentanglement performance Zhu et al. (2021); Yang et al. (2022), these
are still struggling with the same problem as shown in Fig. 1(d) and 1(e). In
addition, symmetries are represented on the latent vector space for
disentangled representations. In current works Miyato et al. (2022); Keller
and Welling (2021a); Quessard et al. (2020), the sequential observation is
considered with unsupervised learning. However, these works need the knowledge
of sequential changes of images to set up inputs manually.

To enhance these two problems of disentanglement learning of group theory-
based methods, addressing two questions is crucial:

  1. 1.

Do the explicitly defined symmetries impact to the structuring of a
disentangled space as depicted in Fig. 1(g)?

  2. 2.

can these symmetries be represented through unsupervised learning without any
prior knowledge of factor information?

##  3 Methods

Figure 2: The overall architecture of our proposed method. The loss function
is divided into four parts: 1) commutative loss ($\mathcal{L}_{c}$), 2)
perpendicular, parallel, and sparsity loss ($\mathcal{L}_{pd}$,
$\mathcal{L}_{pl}$, and $\mathcal{L}_{s}$) in Equation 1-3, 3) factor
prediction loss ($\mathcal{L}_{p}$) in Equation 4), and 4) equivariant loss
($\mathcal{L}_{ee}$, and $\mathcal{L}_{de}$) in Equation 8). MLP is a multi-
layer perceptron, and, $tr$ is a threshold. Attention score $attn_{e}$,
$sw(\cdot)$, and $p_{s}^{i}$ are introduced in section 3.3.

###  3.1 Input: A Pair of Two Samples

To learn the symmetries between inputs with unknown factors changes, we
randomly pair the two samples as an input. During the training, samples in the
mini-batch $\displaystyle{\mathbb{X}}_{|B|}$ are divided into two parts
$\displaystyle{\mathbb{X}}_{|B|}^{1}=\\{x_{1}^{1},x_{2}^{1},\ldots,x_{\frac{|B|}{2}}^{1}\\}$,
and
$\displaystyle{\mathbb{X}}_{|B|}^{2}=\\{x_{1}^{2},x_{2}^{2},\ldots,x_{\frac{|B|}{2}}^{2}\\}$,
where $|B|$ is a mini-batch size. In the next, our model pairs the samples
$(x_{1}^{1},x_{1}^{2}),(x_{2}^{1},x_{2}^{2}),\ldots,(x_{\frac{|B|}{2}}^{1},x_{\frac{|B|}{2}}^{2})$
and is used for learning symmetries between the elements of each pair.

###  3.2 Factor-Aligned Symmetry Learning with Inductive Bias

We define the factor-aligned symmetry that represents a corresponding factor
change on the latent vector space. For factor-aligned symmetry, we compose the
symmetry codebook and inject inductive bias via Parallel loss
$\mathcal{L}_{pl}$ and Perpendicular loss $\mathcal{L}_{pd}$ that matches each
symmetry to a single factor changes. Then we add sparsity loss
$\mathcal{L}_{s}$ to the losses for disentangled representations as shown in
Fig. 3. It aligns a single factor change to an axis of latent vector space.
Also, we implement the commutative loss $\mathcal{L}_{c}$ to reduce the
computational costs for matrix exponential multiplication.

#### Explicit and Learnable Symmetry Representation for Inductive Bias
Injection

To allow the direct injection of inductive bias into symmetries, we implement
an explicit and trainable codebook for symmetry representation. we consider
the symmetry group on the latent vector space as a subgroup of the general lie
group $GL(n)$ under a matrix multiplication. The codebook
$\mathcal{G}=\\{\mathcal{G}^{1},\mathcal{G}^{2},\ldots,\mathcal{G}^{k}\\}$ is
composed of sections $\mathcal{G}^{i}$, which are affect to a different single
factor, where $k\in\\{1,2,\ldots|S|\\}$, and $|S|$ is the number of sections.
The section $\mathcal{G}^{i}$ is composed of Lie algebra
$\\{\mathfrak{g}_{1}^{i},\mathfrak{g}_{2}^{i},\ldots,\mathfrak{g}_{l}^{i}\\}$,
where $\mathfrak{g}_{j}^{i}\in\mathbb{R}^{|D|\times|D|}$,
$l\in\\{1,2,\ldots,|SS|\\}$, $|SS|$ is the number of elements in each section,
and $|D|$ is a dimension size of latent $\displaystyle{\bm{z}}$. We assume
that each Lie algebra consists of linearly independent bases
$\mathfrak{B}=\\{\mathfrak{B}_{i}|\mathfrak{B}_{i}\in\mathbb{R}^{n\times
n},\sum_{i}\alpha_{i}\mathfrak{B}_{i}\neq 0,\alpha_{i}\neq 0\\}$:
$\mathfrak{g}_{j}^{i}=\sum_{b}\alpha_{b}^{i,j}\mathfrak{B}_{b}$, where
$b\in\\{1,2,\ldots,kl\\}$. Then the dimension of the element of the codebook
is equal to $|\mathfrak{B}|$ and the dimension of the Lie group composited by
the codebook element is also $|\mathfrak{B}|$. To utilize previously studied
effective expression of symmetry for disentanglement, we set the symmetry to
be continuous Higgins et al. (2022) and invertible via matrix exponential form
Xiao and Liu (2020) as
$g_{j}^{i}=\mathbf{e}^{\mathfrak{g}_{j}^{i}}=\sum_{k=0}^{\infty}\frac{1}{k!}(\mathfrak{g}_{j}^{i})^{k}$
to construct the Lie group Hall (2015).

(a) parallel loss

(b) (a) + sparsity loss

(c) perpendicular loss

(d) (c) + sparsity loss

Figure 3: Roles of parallel, perpendicular, and sparsity loss on symmetries in
the codebook for adjusting representation change. Parallel loss is for
symmetries of the same section, and perpendicular loss is for different
sections. Each axis (x and y) only affects to a single factor.

#### Inductive Bias: Group Elements of the Same Section Impacts on the Same
Factor Changes

We add a bias that latent vector changes by two symmetries for the same factor
should be parallel
($\displaystyle{\bm{z}}-g_{j}^{i}\displaystyle{\bm{z}}\parallel\displaystyle{\bm{z}}-g_{l}^{i}\displaystyle{\bm{z}}$
for $i$th section) as shown in Fig. 3(a). We define a loss function to make
them parallel as:

$\mathcal{L}_{pl}=\sum_{i=1}^{|S|}\sum_{j,k=1}^{|SS|}\log\frac{<\displaystyle{\bm{z}}-g_{j}^{i}\displaystyle{\bm{z}},\displaystyle{\bm{z}}-g_{k}^{i}\displaystyle{\bm{z}}>}{||\displaystyle{\bm{z}}-g_{j}^{i}\displaystyle{\bm{z}}||_{2}\cdot||\displaystyle{\bm{z}}-g_{k}^{i}\displaystyle{\bm{z}}||_{2}},$
(1)

where $g_{j}^{i}=\mathbf{e}^{\mathfrak{g}_{j}^{i}}$, $<\cdot,\cdot>$ is a dot
product, and $||\cdot||_{2}$ is a L2 norm.

#### Inductive Bias: Group Elements of the Different Section Impacts on the
Different Factor Changes

Similarly to the parallel loss, we inject another bias that changes by two
symmetries for different factors should satisfy the orthogonality (
$\displaystyle{\bm{z}}-g_{j}^{i}\displaystyle{\bm{z}}\perp\displaystyle{\bm{z}}-g_{l}^{k}\displaystyle{\bm{z}}$
for different $i$th and $k$th sections) as shown in Fig. 3(c). The loss for
inducing the orthogonality is

$\mathcal{L}_{pd}=\sum_{i,k=1,i\neq
k}^{|S|}\sum_{j,l=1}^{|SS|}\frac{<\displaystyle{\bm{z}}-g_{j}^{i}\displaystyle{\bm{z}},\displaystyle{\bm{z}}-g_{l}^{k}\displaystyle{\bm{z}}>}{||\displaystyle{\bm{z}}-g_{j}^{i}\displaystyle{\bm{z}}||_{2}\cdot||\displaystyle{\bm{z}}-g_{l}^{k}\displaystyle{\bm{z}}||_{2}}.$
(2)

This loss is computationally expensive to calculate
($O(|S|^{2}\cdot|SS|^{2})$), so we randomly select a $(j,l)$ pair of
symmetries of different sections. This random selection still holds the
orthogonality, because if all elements in the same section satisfy Equation 1
and a pair of elements from a different section
($\mathcal{G}^{i},\mathcal{G}^{j}$) satisfies Equation 2, then any pair of the
element
($\mathfrak{g}^{i}\in\mathcal{G}^{i},\mathfrak{g}^{j}\in\mathcal{G}^{j}$)
satisfies the Equation 2. More details are in Appendix B.

#### Inductive Bias: Align Each Factor Changes to the Axis of Latent Space for
Disentangled Representations

Factorizing latent dimensions to represent the change of independent factors
is an attribute of disentanglement defined in Bengio et al. (2013) and derived
by ELBO term in VAE training frameworks Chen et al. (2018); Kim and Mnih
(2018). However, it is noteworthy that neither the parallel loss nor the
perpendicular loss exhibits specific constraints aimed at inducing this
characteristic as shown in Fig. 3(a), 3(c). To guide symmetries to hold the
attribute, we enforce the change
$\Delta_{j}^{i}{\bm{z}}={\bm{z}}-g_{j}^{i}{\bm{z}}$ to be a parallel shift to
a unit vector as Fig. 3(b), 3(d) via sparsity loss defined as

$\mathcal{L}_{s}=\sum_{i=1}^{|S|}\sum_{j}^{|SS|}\Big{[}\big{[}\sum_{k=1}^{|D|}(\Delta_{j}^{i}{\bm{z}}_{k})^{2}\big{]}^{2}-\max_{k}((\displaystyle\Delta_{j}^{i}{\bm{z}}_{k})^{2})^{2}\Big{]},$
(3)

where $\displaystyle\Delta_{j}^{i}{\bm{z}}_{k}$ is a $k^{th}$ dimension value.

###  3.3 Composition of Factor-Aligned Symmetries via Two-Step Attention for
Unsupervised Learning

#### First Step: Select Factor-Aligned Symmetry

In the first step, the model generate the factor-aligned symmetries of each
section through the attention score:
$g_{c}^{i}=\bm{e}^{\mathfrak{g}_{c}^{i}}$, where
$\mathfrak{g}_{c}^{i}=\sum_{j=1}^{|S.S|}attn_{j}^{i}\mathfrak{g}_{j}^{i}$,
$attn_{j}^{i}=\textit{softmax}([M;\Sigma]\displaystyle{\bm{W}}_{c}^{i}+\displaystyle{\bm{b}}_{c}^{i})$
$[M;\Sigma]=[\bm{\mu}_{1};\bm{\sigma}_{1};\bm{\mu}_{2};\bm{\sigma}_{2}]$,
$\displaystyle{\bm{W}}_{c}^{i}\in\mathbb{R}^{4|D|\times|SS|}$ and
$\displaystyle{\bm{b}}_{c}^{i}\in\mathbb{R}^{|SS|}$ are learnable parameters,
and $i\in\\{1,2,\ldots|S|\\}$.

#### Second Step: Section Selection

In the second step of our proposed model, we enforce the prediction of factors
deemed to have undergone changes. We assume that if some factor value of two
inputs is equal, then the variance of the corresponding latent vector
dimension value is smaller compared to others. Based on this assumption, we
define the target ($T$) for factor prediction: if
$\displaystyle{\bm{z}}_{1,i}-\displaystyle{\bm{z}}_{2,i}>\text{threshold}$,
then we set $T_{i}$ as 1 and 0 otherwise, where $T_{i}$ is a $i^{th}$
dimension value of $T\in\mathbb{R}^{|D|}$, $\displaystyle{\bm{z}}_{j,i}$ is an
$i^{th}$ dimension value of $\displaystyle{\bm{z}}_{j}$, and we set the
threshold as a hyper-parameter. For section prediction, we utilize the cross-
entropy loss:

$\mathcal{L}_{p}=\sum_{i=1}^{|S|}\sum_{c\in
C}\mathds{1}[T_{i}=c]\cdot\log\textit{softmax}(p_{s}^{i}),$ (4)

where
$p_{s}^{i}=[M;\Sigma]\displaystyle{\bm{W}}_{s}^{i}+\displaystyle{\bm{b}}_{s}^{i}$,
$\displaystyle{\bm{W}}_{s}^{i}\in\mathbb{R}^{4|D|\times 2}$ and
$\displaystyle{\bm{b}}_{s}^{i}\in\mathbb{R}^{2}$ are learnable parameters, and
$c\in\\{0,1\\}$.

To infer the activated section of the symmetries codebook, we utilize the
Gumbel softmax function to handle binary on-and-off scenarios, akin to a
switch operation:

$sw(G(p_{s}^{i}))=\begin{cases}G(p_{s,2}^{i})~{}~{}~{}~{}~{}~{}~{}~{}\text{if}~{}p_{s,2}^{i}\geq
0.5\\\ 1-G(p_{s,1}^{i})~{}~{}\text{if}~{}p_{s,2}^{i}<0.5\end{cases},$ (5)

where $p_{s,j}^{i}$ is a $j^{th}$ dimension value of $p_{s}^{i}$, and
$G(\cdot)$ is the Gumbel softmax with temperature as 1e-4.

#### Integration for Composite Symmetry

For the composite symmetry $g_{c}$, we compute it s the product of weighted
sums of switch function $sw(p_{s})$ and prediction distribution $attn$ as:
$g_{c}=\prod_{i=1}^{|S|}\hat{g}_{c}^{i}$, where
$\hat{g}_{c}^{i}=\mathbf{e}^{sw(G(p_{s}^{i}))\cdot\mathfrak{g}_{c}^{i}}$.

#### Commutativity Loss for Computational Efficiency

In the computation of the composite symmetry $g_{c}$, the production
$\prod_{i=1}^{|S|}\hat{g}_{c}^{i}$ is a computationally expensive Taylor
series repeated for all $(i,j)$ pairs. To reduce the cost by repetition, we
enforce all pairs of basis $\mathfrak{g}_{i}^{j}$ to be commutative to convert
the production to $\mathbf{e}^{\sum_{i}\mathfrak{g}_{c}^{i}}$ (By the matrix
exponential property:
$\mathbf{e}^{\displaystyle{\bm{A}}}\mathbf{e}^{\displaystyle{\bm{B}}}=\mathbf{e}^{\displaystyle{\bm{A}}+\displaystyle{\bm{B}}}$
as
$\displaystyle{\bm{A}}\displaystyle{\bm{B}}=\displaystyle{\bm{B}}\displaystyle{\bm{A}}$,
where $\displaystyle{\bm{A}},\displaystyle{\bm{B}}\in\mathbb{R}^{n\times n}$).
The loss for the commutativity is
$\mathcal{L}_{c}=\sum_{i,k=1}^{|S|}\sum_{j,l=1}^{|SS|}\mathfrak{g}_{j}^{i}\mathfrak{g}_{l}^{k}-\mathfrak{g}_{l}^{k}\mathfrak{g}_{j}^{i}\rightarrow\mathbf{0}$.

###  3.4 Equivariance Induction of Composite Symmetries

#### How to Induce Equivariance?

Motivated by the implementations of equivariant mapping in prior studies Yang
et al. (2022); Miyato et al. (2022) for disentanglement learning, we implement
an equivariant encoder and decoder that satisfies $q_{\phi}(\psi_{i}\ast
x)=g_{i}\circ q_{\phi}(x)$ and $p_{\theta}(g_{i}\circ z)=\psi_{i}\ast
p_{\theta}(z)$ respectively, where $q_{\phi}$ is an encoder, and $p_{\theta}$
is the decoder. In the notation, $\psi_{i}$ and $g_{i}$ are group elements of
the group $(\Psi,\ast)$ and $(\mathcal{G},\circ)$ respectively, and both
groups are isomorphic. Each group acts on the input and latent vector space
with group action $\ast$, and $\circ$, respectively. We specify the form of
symmetry $g_{i}$ and $\circ$ as an invertible matrix, and group action as
matrix multiplication on the latent vector space. Then, the encoder
equivariant function can be rewritten by multiplying the inversion of $g_{i}$
on both sides and $z$ can be replaced with the $q_{\phi}(x)$ in the decoder
equivariant function as

$\displaystyle\begin{split}&q_{\phi}(x)=g_{i}^{-1}\circ q_{\phi}(\psi_{i}\ast
x)\\\ &\Longleftrightarrow q_{\phi}(x)-g_{i}^{-1}\circ q_{\phi}(\psi_{i}\ast
x)\rightarrow 0~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}(\text{for
encoder}).\end{split}$ (6) $\displaystyle\begin{split}&p_{\theta}(g_{i}\circ
q_{\phi}(x))=\psi_{i}\ast p_{\theta}(q_{\phi}(x))\\\ &\Longleftrightarrow
p_{\theta}(g_{i}\circ q_{\phi}(x))-\psi_{i}\ast
p_{\theta}(q_{\phi}(x))\rightarrow 0~{}(\text{for decoder}),\end{split}$ (7)

where $x_{j}=\psi_{i\rightarrow j}\ast x_{i}$. For the equivariant encoder and
decoder, we differently propose the single forward process by the encoder and
decoder objective functions compared to previous work Yang et al. (2022).

#### Equivariance Loss for Encoder and Decoder

In order for the equivariant function between the input and latent vector
space, the mapping function $q_{\phi}(\cdot)$ must satisfy Equation 6.
Therefore, we directly induce an equivariant encoder between input and latent
space with MSE loss ($\mathcal{L}_{ee}$). Additionally, we induce the
equivariant decoder ($\mathcal{L}_{de}$) with MSE loss following Equation 7:

$\begin{split}\mathcal{L}_{equiv}&=\mathcal{L}_{ee}+\epsilon\mathcal{L}_{de}=\text{MSE}(q_{\phi}(x_{i}^{1}),g_{i}^{-1}\circ
q_{\phi}(x_{i}^{2}))\\\
&+\epsilon\text{MSE}(p_{\theta}(g_{i}\circ(q_{\phi}(x_{i}^{1})),\psi_{i}\ast
p_{\theta}(q_{\phi}(x_{i}^{1})),\end{split}$ (8)

where $x_{i}^{2}=\psi_{i}\ast x_{i}^{1}$. During the training, we replace the
$p_{\theta}(q_{\phi}(x_{i}^{1}))$ as a $x_{i}^{1}$ because the ELBO term
includes the reconstruction error between $p_{\theta}(q_{\phi}(x_{i}^{1}))$
and $x_{i}^{1}$ to be close to zero.

#### Objective and Base model

Our method can be plugged into existing VAE frameworks, where the objective
function is integrated additively as follows:

$\mathcal{L}(\phi,\theta;\displaystyle{\bm{x}})=\mathcal{L}_{VAE}+\mathcal{L}_{codebook}+\mathcal{L}_{equiv},$
(9)

where $\mathcal{L}_{VAE}$ is the loss function of a VAE framework (Appendix
A). The other loss
$\mathcal{L}_{codebook}=\mathcal{L}_{pl}+\mathcal{L}_{pd}+\mathcal{L}_{s}+\mathcal{L}_{c}+\mathcal{L}_{p}$
and $\mathcal{L}_{equiv}=\mathcal{L}_{ee}+\epsilon\mathcal{L}_{de}$ where
$\epsilon$ is a hyper-parameter, which are introduced in the following
subsections.

3D Car |  FVM |  beta VAE |  MIG |  SAP |  DCI |  m-FVM2 |  m-FVM3 |  m-FVM4
---|---|---|---|---|---|---|---|---
$\beta$-VAE |  91.83($\pm$4.39) |  100.00($\pm$0.00) |  11.44($\pm$1.07) |  0.63($\pm$0.24) |  27.65($\pm$2.50) |  61.28($\pm$9.40) |  - |  -
$\beta$-TCVAE |  92.32($\pm$3.38) |  100.00($\pm$0.00) |  17.19($\pm$3.06) |  1.13($\pm$0.37) |  33.63($\pm$3.27) |  59.25($\pm$5.63) |  - |  -
Factor-VAE |  93.22($\pm$2.86) |  100.00($\pm$0.00) |  10.84($\pm$0.93) |  1.35($\pm$0.48) |  24.31($\pm$2.30) |  50.43($\pm$10.65) |  - |  -
Control-VAE |  93.86($\pm$5.22) |  100.00($\pm$0.00) |  9.73($\pm$2.24) |  1.14($\pm$0.54) |  25.66($\pm$4.61) |  46.42($\pm$10.34) |  - |  -
CLG-VAE |  91.61($\pm$2.84) |  100.00($\pm$0.00) |  11.62($\pm$1.65) |  1.35($\pm$0.26) |  29.55($\pm$1.93) |  47.75($\pm$5.83) |  - |  -
CFASL |  95.70($\pm$1.90) |  100.00($\pm$0.00) |  18.58($\pm$1.24) |  1.43($\pm$0.18) |  34.81($\pm$3.85) |  62.43($\pm$8.08) |  - |  -

smallNORB |  FVM |  beta VAE |  MIG |  SAP |  DCI |  m-FVM2 |  m-FVM3 |  m-FVM4
---|---|---|---|---|---|---|---|---
$\beta$-VAE |  60.71($\pm$2.47) |  59.40($\pm$7.72) |  21.60($\pm$0.59) |  11.02($\pm$0.18) |  25.43($\pm$0.48) |  24.41($\pm$3.34) |  15.13($\pm$2.76) |  -
$\beta$-TCVAE |  59.30($\pm$2.52) |  60.40($\pm$5.48) |  21.64($\pm$0.51) |  11.11($\pm$0.27) |  25.74($\pm$0.29) |  25.71($\pm$3.51) |  15.66($\pm$3.74) |  -
Factor-VAE |  61.93($\pm$1.90) |  56.40($\pm$1.67) |  22.97($\pm$0.49) |  11.21($\pm$0.49) |  24.84($\pm$0.72) |  26.43($\pm$3.47) |  17.25($\pm$3.50) |  -
Control-VAE |  60.63($\pm$2.67) |  61.40($\pm$4.33) |  21.55($\pm$0.53) |  11.18($\pm$0.48) |  25.97($\pm$0.43) |  24.11($\pm$3.41) |  16.12($\pm$2.53) |  -
CLG-VAE |  62.27($\pm$1.71) |  62.60($\pm$5.17) |  21.39($\pm$0.67) |  10.71($\pm$0.33) |  22.95($\pm$0.62) |  27.71($\pm$3.45) |  17.16($\pm$3.12) |  -
CFASL |  62.73($\pm$3.96) |  63.20($\pm$4.13) |  22.23($\pm$0.48) |  11.42($\pm$0.48) |  24.58($\pm$0.51) |  27.96($\pm$3.00) |  17.37($\pm$2.33) |  -

dSprites |  FVM |  beta VAE |  MIG |  SAP |  DCI |  m-FVM2 |  m-FVM3 |  m-FVM4
---|---|---|---|---|---|---|---|---
$\beta$-VAE |  73.54($\pm$6.47) |  83.20($\pm$7.07) |  13.19($\pm$4.48) |  5.69($\pm$1.98) |  21.49($\pm$6.30) |  53.80($\pm$10.29) |  50.13($\pm$11.98) |  48.02($\pm$8.98)
$\beta$-TCVAE |  79.19($\pm$5.87) |  89.20($\pm$4.73) |  23.95($\pm$10.13) |  7.20($\pm$0.66) |  35.33($\pm$9.07) |  61.75($\pm$6.71) |  57.82($\pm$5.39) |  63.81($\pm$9.45)
Factor-VAE |  78.10($\pm$4.45) |  84.40($\pm$5.55) |  25.74($\pm$10.58) |  6.37($\pm$1.82) |  32.30($\pm$9.47) |  58.39($\pm$5.18) |  51.63($\pm$2.88) |  53.71($\pm$4.22)
Control-VAE |  69.64($\pm$7.67) |  82.80($\pm$7.79) |  5.93($\pm$2.78) |  3.89($\pm$1.89) |  12.42($\pm$4.95) |  38.99($\pm$9.31) |  29.00($\pm$10.75) |  19.33($\pm$5.98)
CLG-VAE |  82.33($\pm$5.59) |  86.80($\pm$3.43) |  23.96($\pm$6.08) |  7.07($\pm$0.86) |  31.23($\pm$5.32) |  63.21($\pm$8.13) |  48.68($\pm$9.59) |  51.00($\pm$8.13)
CFASL |  82.30($\pm$5.64) |  90.20($\pm$5.53) |  33.62($\pm$8.18) |  7.28($\pm$0.63) |  46.52($\pm$6.18) |  68.32($\pm$0.13) |  66.25($\pm$7.36) |  71.35($\pm$12.08)

3D Shapes |  FVM |  beta VAE |  MIG |  SAP |  DCI |  m-FVM2 |  m-FVM3 |  m-FVM4
---|---|---|---|---|---|---|---|---
$\beta$-VAE |  84.33($\pm$10.65) |  91.20($\pm$4.92) |  45.80($\pm$21.20) |  8.66($\pm$3.80) |  66.05($\pm$7.44) |  70.26($\pm$6.27) |  61.52($\pm$8.62) |  60.17($\pm$8.48)
$\beta$-TCVAE |  86.03($\pm$3.49) |  87.80($\pm$3.49) |  60.02($\pm$10.05) |  5.88($\pm$0.79) |  70.38($\pm$4.63) |  70.20($\pm$4.08) |  63.79($\pm$5.66) |  63.61($\pm$5.90)
Factor-VAE |  79.54($\pm$10.72) |  95.33($\pm$5.01) |  52.68($\pm$22.87) |  6.20($\pm$2.15) |  61.37($\pm$12.46) |  66.93($\pm$17.49) |  63.55($\pm$18.02) |  57.00($\pm$21.36)
Control-VAE |  81.03($\pm$11.95) |  95.00($\pm$5.60) |  19.61($\pm$12.53) |  4.76($\pm$2.79) |  55.93($\pm$13.11) |  62.22($\pm$11.35) |  55.83($\pm$13.61) |  51.66($\pm$12.08)
CLG-VAE |  83.16($\pm$8.09) |  89.20($\pm$4.92) |  49.72($\pm$16.75) |  6.36($\pm$1.68) |  63.62($\pm$3.80) |  65.13($\pm$5.26) |  58.94($\pm$6.59) |  60.51($\pm$7.62)
CFASL |  89.70($\pm$9.65) |  96.20($\pm$4.85) |  62.12($\pm$13.38) |  9.28($\pm$1.92) |  75.49($\pm$8.29) |  74.26($\pm$2.82) |  67.68($\pm$2.67) |  63.48($\pm$4.12)

Table 1: Disentanglement scores for single factor change (left 5 metrics) and multi-factor change (m-FVMs) with 10 random seeds. |  3D Car |  smallNORB |  dSprites |  3D Shapes |  Avg.
---|---|---|---|---|---
$\beta$-VAE |  3.33 |  4.86 |  4.88 |  3.38 |  4.11
$\beta$-TCVAE |  2.50 |  4.29 |  2.50 |  2.88 |  3.04
Factor-VAE |  3.17 |  2.71 |  3.38 |  4.00 |  3.31
Control-VAE |  3.67 |  3.86 |  6.00 |  5.50 |  4.76
CLG-VAE |  3.00 |  3.86 |  3.13 |  4.13 |  3.53
CFASL |  1.00 |  1.43 |  1.13 |  1.13 |  1.17
Table 2: Disentanglement performance rank. Each dataset rank is an average of
evaluation metrics, and Avg. is an average of all datasets.

###  3.5 Extended Evaluation Metric: m-FVM Metric for Disentanglement in
Multi-Factor Change

We define the multi-factor change condition as simultaneously altering more
than two factors in the transformation between two samples or representations.
To the best of our knowledge, there is no evaluation metric for
disentanglement in multi-factor change, so we propose the extended version of
the Factor-VAE metric (FVM) score called as multi-FVM score (m-FVMk), where
$k\in\\{2,3,\dots,|F|-1\\}$, and $|F|$ is a number of factors. Similar to FVM,
1) we randomly choose the $k$ fixed factors ($F_{i},F_{j},\ldots$). 2) We
sample each factor’s value ($f_{i},f_{j},\ldots$) and fix the corresponding
factor dimension value in the mini-batch, where
$f_{i}\in\\{1,2,\ldots|F_{i}|\\},f_{j}\in\\{1,2,\ldots|F_{j}|\\},\ldots$,
$|F_{i}|$ and $|F_{j}|$ is a maximum value of each factor label. 3)
Subsequently, we estimate the standard deviation (std.) of each dimension to
find the number of $k$ lowest std. dimension ($z_{l1},z_{l2},\ldots$) in one
epoch. 4) We then count each pair of selected dimensions by std. values (the
number of ($z_{l1},z_{l2},\ldots$), which are corresponded to fixed factors).
5) In the last, we add the maximum value of the number of
($z_{l1},z_{l2},\ldots$) on all fixed factor cases, and divide with epoch.

##  4 Related Work

#### Disentanglement Learning

Diverse works for unsupervised disentanglement learning have elaborated in the
machine learning field. The VAE based approaches have factorized latent vector
dimensions with weighted hyper-parameters or controllable weighted values to
penalize Kullback-Leibler divergence (KL divergence) Higgins et al. (2017);
Shao et al. (2020, 2022). Extended works penalize total correlation for
factorizing latent vector dimensions with divided KL divergence Chen et al.
(2018) and discriminator Kim and Mnih (2018). Differently, we induce
disentanglement learning with group equivariant VAE for inductive bias.

#### Group Theory-Based Approaches for Disentangled Representation

In recent periods, various unsupervised disentanglement learning research
proposes different approaches with another definition of disentanglement,
which is based on the group theory Higgins et al. (2018). To learn the
equivariant function, Topographic VAE Keller and Welling (2021b) proposes the
sequentially permuted activations on the latent vector space called shifting
temporal coherence, and Groupified VAE Yang et al. (2022) method proposes that
inputs pass the encoder and decoder two times to implement permutation group
equivariant VAE models. Also, Commutative Lie Group VAE (CLG-VAE) Zhu et al.
(2021); Mercatali et al. (2022) maps latent vectors into Lie algebra with one-
parameter subgroup decomposition for inductive bias to learn the group
structure from abstract canonical point to inputs. Differently, we propose the
trainable symmetries that are extracted between two samples directly on the
latent space while maintaining the equivariance function between input and
latent vector space.

| $\mathcal{L}_{p}$ | $\mathcal{L}_{c}$ | $\mathcal{L}_{e.}$ | $\mathcal{L}_{pl}$ | $\mathcal{L}_{pd}$ | $\mathcal{L}_{s}$ | FVM | MIG | SAP | DCI |  m-FVM2
---|---|---|---|---|---|---|---|---|---|---|---
$\beta$-VAE | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |  88.19($\pm$4.60) |  6.82($\pm$2.93) |  0.63($\pm$0.33) |  20.45($\pm$3.93) |  42.36($\pm$7.16)
✗ | ✓ | ✓ | ✓ | ✓ | ✓ |  88.57($\pm$6.68) |  7.18($\pm$2.52) |  1.85($\pm$1.04) |  18.39($\pm$4.80) |  48.23($\pm$5.51)
✓ | ✓ | ✗ | ✓ | ✓ | ✓ |  88.56($\pm$7.78) |  7.27($\pm$4.16) |  1.31($\pm$0.70) |  19.58($\pm$4.45) |  42.63($\pm$4.21)
✓ | ✓ | ✓ | ✗ | ✓ | ✓ |  86.95($\pm$5.96) |  7.11($\pm$3.49) |  1.09($\pm$0.40) |  18.35($\pm$3.32) |  41.90($\pm$7.80)
✓ | ✓ | ✓ | ✓ | ✗ | ✓ |  85.42($\pm$7.89) |  7.30($\pm$3.73) |  1.15($\pm$0.70) |  21.69($\pm$4.70) |  41.90($\pm$6.07)
✓ | ✓ | ✓ | ✗ | ✗ | ✗ |  89.34($\pm$5.18) |  9.44($\pm$2.91) |  1.26($\pm$0.40) |  23.14($\pm$5.51) |  51.37($\pm$9.29)
✓ | ✓ | ✓ | ✓ | ✓ | ✗ |  90.71($\pm$5.75) |  9.29($\pm$3.74) |  1.07($\pm$0.65) |  22.74($\pm$5.06) |  45.84($\pm$7.71)
✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  91.91($\pm$3.45) |  9.51($\pm$2.74) |  1.42($\pm$0.52) |  20.72($\pm$3.65) |  55.47($\pm$10.09)
Table 3: Ablation study for loss functions on 3D-Cars and $\beta$-VAE with 10
random seeds.

#### Symmetry Learning with Equivariant Model

Lie group equivariant CNN Dehmamy et al. (2021) and Finzi et al. (2020)
construct the In the other literature, several works extract symmetries, which
consist of matrices, between two inputs or objects. Miyato et al. (2022)
extracts the symmetries between sequential or sequentially augmented inputs by
penalizing the transformation of difference of the same time interval. Other
work extracts the symmetries by comparing two inputs, in which the
differentiated factor is a rotation or translation, and implements symmetries
with block diagonal matrices Bouchacourt et al. (2021). Furthermore, Marchetti
et al. (2023) decomposes the class and pose factor simultaneously by invariant
and equivariant loss function with weakly supervised learning. The
unsupervised learning work Winter et al. (2022a) achieves class invariant and
group equivariant function in less constraint conditions. Differently, we
generally extend the a class invariant and group equivariant model in the more
complex disparity condition without any knowledge of the factors of datasets.

##  5 Experiments

###  5.1 Settings

We implement $\beta$-VAE Higgins et al. (2017), $\beta$-TCVAE Chen et al.
(2018), Factor-VAE Kim and Mnih (2018), control-VAE Shao et al. (2020), and
Commutative Lie Group VAE (CLG-VAE) Zhu et al. (2021) for baseline. For common
settings to baselines, we set batch size 64, learning rate 1e-4, and random
seed from $\\{1,2,\ldots,10\\}$ without weight decay. We train for $3\times
10^{5}$ iterations on dSprites smallNORB and 3D Cars, and $5\times 10^{5}$
iterations on 3D Shapes. Also, each dataset guarantees the commutativity of
transformation. More details for experimental settings are in Appendix C.

###  5.2 Quantitative Analysis Results and Discussion

#### Disentanglement Performance in Single and Multi-Factor Change

We evaluate four common disentanglement metrics: FVM Kim and Mnih (2018), MIG
Chen et al. (2018), SAP Kumar et al. (2018), and DCI Eastwood and Williams
(2018), and more details of evaluation settings are in Appendix C. As shown in
Table 1, our method gradually improves the disentanglement learning in
dSprites, 3D Cars, 3D Shapes, and smallNORB datasets in most metrics. This
result also shows that our method positively affects single factor change
conditions. More details are in Appendix D.1.

To show the quantitative score of the disentanglement in multi-factor change,
we evaluate the m-FVMk, where max($k$) is 2, 3, and 4 in 3D Cars, smallNORB,
and dSprites datasets respectively. As shown in Table 2, the proposed method
shows a statistically significant improvement, as indicated by the higher
average rank of dataset metrics compared to other approaches. It implies that
our method has the benefit of disentanglement learning in the multi-factor
change condition. We provide additional results in Appendix D.1.

#### Ablation Study

Table 3 shows the ablation study to evaluate the impact of each component of
our method for disentanglement learning. To compare factor-aligned losses (w/o
$\mathcal{L}_{pl}$, w/o $\mathcal{L}_{pd}$, w/o $\mathcal{L}_{s}$, and w/o
$\mathcal{L}_{pl}+\mathcal{L}_{pd}+\mathcal{L}_{s}$), the best of among four
cases is the w/o $\mathcal{L}_{pl}+\mathcal{L}_{pd}+\mathcal{L}_{s}$ and it
implies that these losses are interrelated. In the case of w/o
$\mathcal{L}_{pl}$, the extraction of the composite symmetry $g_{c}$ becomes
challenging due to the lack of unified roles among individual sections.
Composite symmetry $g_{c}$ is affected by the second section selection method,
which is whether to use the section or not (0 or 1). Therefore, composite that
has a different role on elements in the same section struggles with
constructing adequate composite symmetry $g_{c}$. With the similar perspective
referred to w/o $\mathcal{L}_{pl}$ case, the coverage of code w/o
$\mathcal{L}_{pd}$ is limited due to the absence of assurance that each
section aligns with distinct factors. In the case of w/o $\mathcal{L}_{s}$,
each section assigns a different role and the elements of each section align
on the same factor, w/o $\mathcal{L}_{s}$ case is better than w/o
$\mathcal{L}_{pl}$ and w/o $\mathcal{L}_{pd}$. Also, constructing the
symmetries without the equivariant model is meaningless because the model does
not satisfy Equation 6\- 8. The w/o $\mathcal{L}_{equiv}$ naturally shows the
lowest results compared to other cases except w/o $\mathcal{L}_{pd}$ and
$\mathcal{L}_{pl}$. Moreover, the w/o $\mathcal{L}_{p}$ case shows the impact
of the second section selection for unsupervised learning. Above all, each
group exhibits a positive influence on disentanglement when compared to the
base model ($\beta$-VAE). When combining all loss functions, our method
consistently outperforms the others across the majority of evaluation metrics.
The inductive bias for symmetry changes ($\mathcal{L}_{pl}+\mathcal{L}_{pd}$)
is less effective than that for composition because the bias is only for
symmetry change control without latent dimension matching to a factor. The
inclusion of a sparsity loss effectively addresses this concern and yields the
best improvement. More details are in the Appendix D.2.

Figure 4: Heatmaps of Eigenvectors for latent vector representations.

(a) Generated images by composite symmetry and its factor-aligned symmetries.
Image ① and ② are inputs, and image ③ is an output from image ①
($p_{\theta}(z_{1})$). Image ⑤ is a output of group element $g_{c}$ acted on
$z_{1}$ ($p(g_{c}z_{1})$). Images ④ are outputs of decomposed composite
symmetry $g_{c}$ acted on $z_{1}$ sequentially.

(b) Generated images by dimension change. Red and blue colored squares
represent the value of latent vector dimensions of $z_{1}$ and $z_{2}$. The
images of baseline and CAFSL are the generated images from each latent vector.

(c) Generalization over unseen pairs of images. We set pairs
$\\{(x_{i-1},x_{i})|1\leq i\leq|\displaystyle|{\mathbb{X}}||-1\\}$ then
extract the symmetries between elements of each pair
$g_{p}=\\{g_{(1,2)},g_{(2,3)},\ldots g_{(k-1,k)}\\}$ in inference step, where
$g_{(k-1,k)}$ is a symmetry between $z_{k-1}$ and $z_{k}$. The first row
images are inputs (targets) and the second row images are the generated images
by symmetry codebook.

Figure 5: Qualitative Analysis to Generate Images from Latent Vectors in
Various Conditions. More details are in the Appendix D.3.

###  5.3 Qualitative Analysis Results and Discussion

#### Is Latent Vector Space Close to Disentangled Space?

The previous result as shown in Fig. 1 is a clear example of of whether the
latent vector space closely approximates a disentangled space. The latent
vector space of previous works (Fig. 1(a)-1(e)) are far from disentangled
space (Fig. 1(g)) but CFASL shows the closest disentangled space compare to
other methods.

#### Alignment of Latent Dimensions to Factor-Aligned Symmetry

In the principal component analysis of latent vectors shown in Fig. 4, the
eigenvectors
$\displaystyle{\bm{V}}=[\displaystyle{\bm{v}}_{1},\displaystyle{\bm{v}}_{2},\ldots,\displaystyle{\bm{v}}_{|D|}]$
are close to one-hot vectors compared to the baseline, and the dominating
dimension of the one-hot vectors are all different. This result implies that
the representation (factor) changes are aligned to latent dimensions.

#### Factor Aligned Symmetries

To verify the representation of learnable codebook over composite symmetries
and factor-aligned symmetries, we randomly select a sample pair as shown in
Fig. 5(a). The results imply that $g_{c}$ generated from the codebook
represents the composite symmetries between two images (① and ②) because the
image ② and the generated image ⑤ by symmetry $g_{c}$ are similar
($p(z_{2})\approx p(g_{c}z_{1})$). Also, each factor-aligned symmetry
($g_{i}$) generated from codebook section affects a single factor changes as
shown in images ④. in Fig. 5(a).

#### Factor Aligned Latent Dimension

To analyze each factor changes aligned to each dimension of latent vector
space, we set the qualitative analysis as shown in 5(b). We select two random
samples ($x_{1}$, $x_{2}$), generate latent vectors $z_{1}$ and $z_{2}$, and
select the largest Kullback-Leibler divergence (KLD) value dimension from
their posterior. Then, replacing the dimension value of $z_{1}$ to the value
of $z_{2}$ one by one sequentially. As a result, the shape and color factors
are changed when a single dimension value is replaced within the baseline.
However, our method results show no overlapped factor changes compared to
baseline results. It implies that each latent vector dimension of the proposed
method contains a single factor of information.

#### Unseen Change Prediction in Sequential Data

The sequential observation as Miyato et al. (2022) is rarely observed in our
methods, because of the random pairing during training (less 1 pair of
observation). But their generated images via trained symmetries of our method
are similar to target images as shown in Fig. 5(c). This result implies that
our method is strongly regularized for unseen change.

##  6 Conclusion

This work tackles the difficulty of disentanglement learning of VAEs in
unknown factors change conditions. We propose a novel framework to learn
composite symmetries from explicit factor-aligned symmetries by codebook to
directly represent the multi-factor change of a pair of samples in
unsupervised learning. The framework enhances disentanglement by learning an
explicit symmetry codebook, injecting three inductive biases on the symmetries
aligned to unknown factors, and inducing a group equivariant VAE model. We
quantitatively evaluate disentanglement in the condition by a novel metric
(m-FVMk) extended from a common metric for a single factor change condition.
This method significantly improved in the multi-factor change and gradually
improved in the single factor change condition compared to state-of-the-art
disentanglement methods of VAEs. Also, training process does not need the
knowledge of factor information of datasets. This work can be easily plugged
into VAEs and extends disentanglement to more general factor conditions of
complex datasets.

## References

  * Bengio et al. [2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: a review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1798—1828, August 2013.
  * Bouchacourt et al. [2021] Diane Bouchacourt, Mark Ibrahim, and Stéphane Deny.  Addressing the topological defects of disentanglement via distributed operators, 2021.
  * Burgess and Kim [2018] Chris Burgess and Hyunjik Kim.  3d shapes dataset.  https://github.com/deepmind/3dshapes-dataset/, 2018.
  * Cao et al. [2022] Jinkun Cao, Ruiqian Nai, Qing Yang, Jialei Huang, and Yang Gao.  An empirical study on disentanglement of negative-free contrastive learning.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * Chen et al. [2018] Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud.  Isolating sources of disentanglement in variational autoencoders.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * Dehmamy et al. [2021] Nima Dehmamy, Robin Walters, Yanchen Liu, Dashun Wang, and Rose Yu.  Automatic symmetry discovery with lie algebra convolutional network.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * Eastwood and Williams [2018] Cian Eastwood and Christopher K. I. Williams.  A framework for the quantitative evaluation of disentangled representations.  In International Conference on Learning Representations, 2018.
  * Finzi et al. [2020] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson.  Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data.  arXiv preprint arXiv:2002.12880, 2020.
  * Hall [2015] B. Hall.  Lie Groups, Lie Algebras, and Representations: An Elementary Introduction.  Graduate Texts in Mathematics. Springer International Publishing, 2015.
  * Higgins et al. [2017] Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  In ICLR, 2017.
  * Higgins et al. [2018] Irina Higgins, David Amos, David Pfau, Sébastien Racanière, Loïc Matthey, Danilo J. Rezende, and Alexander Lerchner.  Towards a definition of disentangled representations.  CoRR, abs/1812.02230, 2018.
  * Higgins et al. [2022] Irina Higgins, Sébastien Racanière, and Danilo Rezende.  Symmetry-based representations for artificial and biological general intelligence, 2022.
  * Jeong and Song [2019] Yeonwoo Jeong and Hyun Oh Song.  Learning discrete and continuous factors of data via alternating disentanglement.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3091–3099. PMLR, 09–15 Jun 2019.
  * Keller and Welling [2021a] T. Anderson Keller and Max Welling.  Topographic VAEs learn equivariant capsules.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * Keller and Welling [2021b] T. Anderson Keller and Max Welling.  Topographic vaes learn equivariant capsules.  CoRR, abs/2109.01394, 2021.
  * Kim and Mnih [2018] Hyunjik Kim and Andriy Mnih.  Disentangling by factorising.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2649–2658. PMLR, 10–15 Jul 2018.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes, 2013.
  * Kumar et al. [2018] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan.  VARIATIONAL INFERENCE OF DISENTANGLED LATENT CONCEPTS FROM UNLABELED OBSERVATIONS.  In International Conference on Learning Representations, 2018.
  * LeCun et al. [2004] Yann LeCun, Fu Jie Huang, and Léon Bottou.  Learning methods for generic object recognition with invariance to pose and lighting.  In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR’04, page 97–104, USA, 2004. IEEE Computer Society.
  * Liu et al. [2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  2015 IEEE International Conference on Computer Vision (ICCV), pages 3730–3738, 2015.
  * Locatello et al. [2019] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem.  Challenging common assumptions in the unsupervised learning of disentangled representations.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4114–4124. PMLR, 09–15 Jun 2019.
  * Marchetti et al. [2023] Giovanni Luca Marchetti, Gustaf Tegnér, Anastasiia Varava, and Danica Kragic.  Equivariant representation learning via class-pose decomposition.  In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 4745–4756. PMLR, 25–27 Apr 2023.
  * Matthey et al. [2017] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner.  dsprites: Disentanglement testing sprites dataset.  https://github.com/deepmind/dsprites-dataset/, 2017.
  * Mercatali et al. [2022] Giangiacomo Mercatali, Andre Freitas, and Vikas Garg.  Symmetry-induced disentanglement on graphs.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 31497–31511. Curran Associates, Inc., 2022.
  * Michlo [2021] Nathan Juraj Michlo.  Disent - a modular disentangled representation learning framework for pytorch.  Github, 2021.
  * Miyato et al. [2022] Takeru Miyato, Masanori Koyama, and Kenji Fukumizu.  Unsupervised learning of equivariant structure from sequences.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * Nasiri and Bepler [2022] Alireza Nasiri and Tristan Bepler.  Unsupervised object representation learning using translation and rotation group equivariant VAE.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * Quessard et al. [2020] Robin Quessard, Thomas Barrett, and William Clements.  Learning disentangled representations and group structure of dynamical environments.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 19727–19737. Curran Associates, Inc., 2020.
  * Reed et al. [2015] Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee.  Deep visual analogy-making.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * Shakerinava et al. [2022] Mehran Shakerinava, Arnab Kumar Mondal, and Siamak Ravanbakhsh.  Structuring representations using group invariants.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * Shao et al. [2020] Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher.  ControlVAE: Controllable variational autoencoder.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8655–8664. PMLR, 13–18 Jul 2020.
  * Shao et al. [2022] Huajie Shao, Yifei Yang, Haohong Lin, Longzhong Lin, Yizhuo Chen, Qinmin Yang, and Han Zhao.  Rethinking controllable variational autoencoders.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19250–19259, June 2022.
  * Winter et al. [2022a] Robin Winter, Marco Bertolini, Tuan Le, Frank Noe, and Djork-Arné Clevert.  Unsupervised learning of group invariant and equivariant representations.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * Winter et al. [2022b] Robin Winter, Marco Bertolini, Tuan Le, Frank Noe, and Djork-Arné Clevert.  Unsupervised learning of group invariant and equivariant representations.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * Xiao and Liu [2020] Changyi Xiao and Ligang Liu.  Generative flows with matrix exponential.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10452–10461. PMLR, 13–18 Jul 2020.
  * Yang et al. [2022] Tao Yang, Xuanchi Ren, Yuwang Wang, Wenjun Zeng, and Nanning Zheng.  Towards building a group-based unsupervised representation disentanglement framework.  In International Conference on Learning Representations, 2022.
  * Zhu et al. [2021] Xinqi Zhu, Chang Xu, and Dacheng Tao.  Commutative lie group VAE for disentanglement learning.  CoRR, abs/2106.03375, 2021.

##  Appendix A Loss Function of Baseline

As shown in Table 4, we train the baselines with each objective function.

VAEs | $\mathcal{L}_{VAE}$
---|---
$\beta$-VAE | $\mathbb{E}_{q_{\phi}(z|x)}\log p_{\theta}(x|z)-\beta\mathcal{D}_{KL}(q_{\phi}(z|x)||p(z))$
$\beta$-TCVAE | $\mathbb{E}_{q_{\phi}(z|x)}\log p_{\theta}(x|z)-\alpha\mathcal{D}_{KL}(q(z,n)||q(z)p(n))$
$-\beta\mathcal{D}_{KL}(q(z)||\prod_{j}a(z_{j}))-\gamma\sum_{j}\mathcal{D}_{KL}(q(z_{j})||p(z_{j}))$
Factor-VAE | $\frac{1}{N}\sum_{i}^{N}[\mathbb{E}_{q(z|x^{i})}[\log p(x^{i}|z)]-\mathcal{D}_{KL}(q(z|x^{i})||p(z))]$
$-\gamma\mathcal{D}_{KL}(q(z)||\prod_{j}(z_{j}))$
Control-VAE | $\mathbb{E}_{q_{\phi}(z|x)}\log p_{\theta}(x|z)-\beta(t)\mathcal{D}_{KL}(q_{\phi}(z|x)||p(z))$
CLG-VAE | $\mathbb{E}_{a(z|x)q(t|z)}\log p(x|z)p(z|t)$
$-\mathbb{E}_{q(z|x)}\mathcal{D}_{KL}(q(t|z)||p(t))-\mathbb{E}_{q(z|x)}\log
q(z|x)$
Table 4: Objective Function of the VAEs.

##  Appendix B Perpendicular and Parallel Loss Relationship

We define parallel loss $\mathcal{L}_{p}$ to set two vectors in the same
section of the symmetries codebook to be parallel: $z-g_{j}^{i}\parallel
z-g_{j^{\prime}}^{i}z$ then,

$\displaystyle\displaystyle{\bm{z}}-g_{j}^{i}\displaystyle{\bm{z}}=c(\displaystyle{\bm{z}}-g_{j^{\prime}}^{i}\displaystyle{\bm{z}})$
(10) $\displaystyle\Rightarrow~{}$
$\displaystyle(1-c)\displaystyle{\bm{z}}=(g_{j}^{i}-cg_{j^{\prime}}^{i})\displaystyle{\bm{z}}$
(11) $\displaystyle\Rightarrow~{}$
$\displaystyle(1-c)\displaystyle{\bm{I}}=g_{j}^{i}-cg_{j^{\prime}}^{i}~{}\text{or}~{}[(1-c)I+cg_{j^{\prime}}^{i}-g_{j}^{i}]\displaystyle{\bm{z}}=0,$
(12)

where $\displaystyle{\bm{I}}$ is an identity matrix and constant
$c\in\mathbb{R}$. However, all latent $z$ is not eigenvector of
$[(1-c)\displaystyle{\bm{I}}+cg_{j^{\prime}}^{i}-g_{j}^{i}]$. Then, we
generally define symmetry as:

$g_{j^{\prime}}^{i}=\frac{1}{c}g_{j}^{i}+\frac{c-1}{c}\displaystyle{\bm{I}},$
(13)

where $i,j,$ and $j^{\prime}$ are natural number $1\leq i\leq|S|$, $1\leq
j,j^{\prime}\leq|SS|$, and $k\neq j$. Therefore, all symmetries in the same
section are parallel then, any symmetry in the same section is defined by a
specific symmetry in the same section.

We define orthogonal loss $\mathcal{L}_{o}$ between two vectors, which are in
different sections, to be orthogonal:
$\displaystyle{\bm{z}}-g_{j}^{i}\displaystyle{\bm{z}}\perp\displaystyle{\bm{z}}-g_{l}^{k}\displaystyle{\bm{z}}$,
where $i\neq k$, $1\leq i,k\leq|S|$, and $1\leq j,l\leq|SS|$. By the Equation
13,

$\displaystyle\displaystyle{\bm{z}}-g_{j}^{i}\displaystyle{\bm{z}}\perp\displaystyle{\bm{z}}-g_{l}^{k}\displaystyle{\bm{z}}$
(14) $\displaystyle\Rightarrow~{}$
$\displaystyle(\frac{1}{c_{a}}g_{a}^{i}+\frac{c_{a}-1}{c_{a}}\displaystyle{\bm{I}})\displaystyle{\bm{z}}-\displaystyle{\bm{z}}\perp(\frac{1}{c_{b}}g_{b}^{k}+\frac{c_{b}-1}{c_{b}}\displaystyle{\bm{I}})\displaystyle{\bm{z}}-\displaystyle{\bm{z}}$
(15) $\displaystyle\Rightarrow~{}$
$\displaystyle\frac{1}{c_{a}}(g^{i}_{a}\displaystyle{\bm{z}}-\displaystyle{\bm{z}})\perp\frac{1}{c_{b}}(g^{k}_{b}\displaystyle{\bm{z}}-\displaystyle{\bm{z}}),$
(16)

where $c_{a}$ and $c_{b}$ are any natural number, and $1\leq a,b\leq|SS|$.
Therefore, if two vectors from different sections are orthogonal and satisfied
with Equation 13, then any pair of vectors from different sections is always
orthogonal.

##  Appendix C Experiment Details

#### Device

We set the below settings for all experiments in a single Galaxy 2080Ti GPU
for 3D Cars and smallNORB, and a single Galaxy 3090 for dSprites 3D Shapes and
CelebA. More details are in README.md file.

#### Datasets

1) The dSprites dataset consists of 737,280 binary $64\times 64$ images with
five independent ground truth factors(number of values), $i.e.$
x-position(32), y-position(32), orientation(40), shape(3), and scale(6),
Matthey et al. [2017]. Any composite transformation of x- and y-position,
orientation (2D rotation), scale, and shape is commutative. 2) The 3D Cars
dataset consists of 17,568 RGB $64\times 64\times 3$ images with three
independent ground truth factors: elevations(4), azimuth directions(24), and
car models(183) Reed et al. [2015]. Any composite transformation of
elevations(x-axis 3D rotation), azimuth directions (y-axis 3D rotation), and
models are commutative. 3) The smallNORB LeCun et al. [2004] dataset consists
of total $96\times 96$ 24,300 grayscale images with four factors, which are
category(10), elevation(9), azimuth(18), light(6) and we resize the input as
$64\times 64$. Any composite transformation of elevations(x-axis 3D rotation),
azimuth (y-axis 3D rotation), light, and category is commutative. 4) The 3D
Shapes dataset consists of 480,000 RGB $64\times 64\times 3$ images with six
independent ground truth factors: orientation(15), shape(4), floor color(10),
scale(8), object color(10), and wall color(10) Burgess and Kim [2018]. 5) The
CelebA dataset Liu et al. [2015] consists of 202,599 images, and we crop the
center 128 $\times$ 128 area and then, resize to 64 $\times$ 64 images.

#### Evaluation Settings

We set prune_dims.threshold as 0.06, 100 samples to evaluate global empirical
variance in each dimension, and run it a total of 800 times to estimate the
FVM score introduced in Kim and Mnih [2018]. For the other metrics, we follow
default values introduced in Michlo [2021], training and evaluation 10,000 and
5,000 times with 64 mini-batches, respectively Cao et al. [2022].

#### Model Hyper-parameter Tuning

We implement $\beta$-VAE Higgins et al. [2017], $\beta$-TCVAE Chen et al.
[2018], control-VAE Shao et al. [2020], Commutative Lie Group VAE (CLG-VAE)
Zhu et al. [2021], and Groupified-VAE (G-VAE) Yang et al. [2022] for baseline.
For common settings to baselines, we set batch size 64, learning rate 1e-4,
and random seed from $\\{1,2,\ldots,10\\}$ without weight decay. We train for
$3\times 10^{5}$ iterations on dSprites smallNORB and 3D Cars, $6\times
10^{5}$ iterations on 3D Shapes, and $10^{6}$ iterations on CelebA. We set
hyper-parameter $\beta\in\\{1.0,2.0,4.0,6.0\\}$ for $\beta$-VAE and
$\beta$-TCVAE, fix the $\alpha,\gamma$ for $\beta$-TCVAE as 1 Chen et al.
[2018]. We follow the ControlVAE settings Shao et al. [2020], the desired
value $C\in\\{10.0,12.0,14.0,16.0\\}$, and fix the $K_{p}=0.01$,
$K_{i}=0.001$. For CLG-VAE, we also follow the settings Zhu et al. [2021] as
$\lambda_{hessian}=40.0$, $\lambda_{decomp}=20.0$, $p=0.2$, and balancing
parameter of $loss_{\text{rec group}}\in\\{0.1,0.2,0.5,0.7\\}$. For G-VAE, we
follow the official settings Yang et al. [2022] with $\beta$-TCVAE
($\beta\in\\{10,20,30\\}$), because applying this method to the $\beta$-TCVAE
model usually shows higher performance than other models Yang et al. [2022].
Then we select the best case of models. We run the proposed model on the
$\beta$-VAE and $\beta$-TCVAE because these methods have no inductive bias to
symmetries. We set the same hyper-parameters of baselines with
$\epsilon\in\\{0.1,0.01\\}$, threshold $\in\\{0.2,0.5\\}$, $|S|=|SS|=|D|$,
where $|D|$ is a latent vector dimension. More details for experimental
settings.

###  C.1 Best Models for Quantitative Analysis

In this section, we show how we pick the best model among hyper-parameter
tuning results. As shown in Table 5-7, we choose the best model on each
datasets.

$\beta$ | beta VAE | FVM | MIG | SAP | DCI
---|---|---|---|---|---
1.0 |  78.80($\pm$6.61) |  65.13($\pm$12.78) |  4.62($\pm$3.21) |  2.67($\pm$1.52) |  9.22($\pm$3.05)
2.0 |  81.00($\pm$7.62) |  64.78($\pm$10.02) |  6.34($\pm$3.66) |  3.37($\pm$1.70) |  10.95($\pm$4.42)
4.0 |  82.67($\pm$7.28) |  73.54($\pm$6.47) |  13.19($\pm$4.48) |  5.69($\pm$1.98) |  21.49($\pm$6.30)
6.0 |  74.80($\pm$10.46) |  63.20($\pm$6.76) |  8.35($\pm$2.95) |  2.43($\pm$1.27) |  13.45($\pm$5.07)
(a) $\beta$-VAE

$\beta$ | beta VAE | FVM | MIG | SAP | DCI
---|---|---|---|---|---
1.0 |  77.20($\pm$8.01) |  65.46($\pm$8.79) |  4.32($\pm$1.46) |  2.41($\pm$1.30) |  9.34($\pm$1.23)
2.0 |  78.20($\pm$9.59) |  70.68($\pm$11.16) |  11.74($\pm$8.51) |  3.84($\pm$2.83) |  16.80($\pm$11.20)
4.0 |  87.40($\pm$4.72) |  78.18($\pm$7.31) |  19.47($\pm$6.61) |  6.32($\pm$1.70) |  30.05($\pm$8.57)
6.0 |  89.20($\pm$4.73) |  79.19($\pm$5.87) |  23.95($\pm$10.13) |  7.20($\pm$0.66) |  35.33($\pm$9.07)
(b) $\beta$-TCVAE

$C$ | beta VAE | FVM | MIG | SAP | DCI
---|---|---|---|---|---
10.0 |  82.80($\pm$ 7.79) |  69.64($\pm$7.67) |  5.93($\pm$2.78) |  3.89($\pm$1.89) |  12.42($\pm$4.95)
12.0 |  75.20($\pm$5.43) |  68.00($\pm$8.67) |  5.10($\pm$2.24) |  2.49($\pm$1.50) |  9.82($\pm$3.69)
14.0 |  73.60($\pm$9.03) |  61.58($\pm$7.87) |  4.53($\pm$2.60) |  2.11($\pm$1.67) |  9.30($\pm$1.89)
16.0 |  76.20($\pm$8.14) |  63.28($\pm$7.98) |  4.09($\pm$2.00) |  2.08($\pm$1.37) |  8.91($\pm$1.88)
(c) Control-VAE

$loss_{\text{rec group}}$ | beta VAE | FVM | MIG | SAP | DCI
---|---|---|---|---|---
0.1 |  86.80($\pm$3.43) |  82.33($\pm$5.59) |  23.96($\pm$6.08) |  7.07($\pm$0.86) |  31.23($\pm$5.32)
0.2 |  88.20($\pm$4.57) |  82.88($\pm$3.55) |  20.39($\pm$6.31) |  6.82($\pm$1.80) |  28.28($\pm$7.09)
0.5 |  88.20($\pm$5.53) |  81.05($\pm$7.51) |  20.63($\pm$6.64) |  6.49($\pm$1.98) |  27.45($\pm$6.07)
0.7 |  88.00($\pm$4.81) |  79.93($\pm$8.16) |  18.95($\pm$6.86) |  6.94($\pm$1.19) |  27.27($\pm$6.76)
(d) Commutative Lie Group VAE

Table 5: Baselines hyper-parameter tuning results on dSprites dataset with 10
random seeds.

$\beta$ | FVM | MIG | SAP | DCI |  m-fvm2
---|---|---|---|---|---
1.0 |  88.19($\pm$4.60) |  6.82($\pm$2.93) |  0.63($\pm$0.33) |  20.45($\pm$3.93) |  42.36($\pm$7.16)
2.0 |  88.51($\pm$5.44) |  10.00($\pm$3.84) |  0.79($\pm$0.38) |  28.78($\pm$7.28) |  50.98($\pm$8.33)
4.0 |  90.95($\pm$4.01) |  12.76($\pm$1.19) |  0.61($\pm$0.36) |  30.70($\pm$3.06) |  55.76($\pm$9.97)
6.0 |  91.83($\pm$4.39) |  11.44($\pm$1.07) |  0.63($\pm$0.24) |  27.65($\pm$2.50) |  61.28($\pm$9.40)
(a) $\beta$-VAE

$\beta$ | FVM | MIG | SAP | DCI |  m-fvm2
---|---|---|---|---|---
1.0 |  89.85($\pm$7.17) |  7.27($\pm$3.94) |  0.71($\pm$0.40) |  21.21($\pm$6.26) |  41.99($\pm$3.09)
2.0 |  91.29($\pm$3.95) |  11.62($\pm$3.70) |  0.79($\pm$0.27) |  30.60($\pm$5.23) |  54.87($\pm$3.20)
4.0 |  92.70($\pm$3.41) |  17.31($\pm$2.91) |  1.07($\pm$0.36) |  33.19($\pm$3.38) |  59.12($\pm$2.20)
6.0 |  92.32($\pm$3.38) |  17.20($\pm$3.06) |  1.13($\pm$0.36) |  33.63($\pm$3.27) |  59.25($\pm$5.63)
(b) $\beta$-TCVAE

$C$ | FVM | MIG | SAP | DCI |  m-fvm2
---|---|---|---|---|---
10.0 |  93.86($\pm$5.12) |  9.73($\pm$2.24) |  1.14($\pm$0.54) |  25.66($\pm$4.61) |  46.42($\pm$10.34)
12.0 |  91.43($\pm$5.32) |  8.65($\pm$3.59) |  11.28($\pm$0.70) |  21.05($\pm$3.93) |  46.06($\pm$11.20)
14.0 |  88.09($\pm$5.46) |  6.11($\pm$3.46) |  1.05($\pm$0.57) |  22.09($\pm$4.34) |  45.87($\pm$10.84)
16.0 |  89.65($\pm$6.87) |  8.12($\pm$3.71) |  0.71($\pm$0.36) |  20.89($\pm$6.30) |  45.77($\pm$10.23)
(c) Control-VAE

$loss_{\text{rec group}}$ | FVM | MIG | SAP | DCI |  m-fvm2
---|---|---|---|---|---
0.1 |  91.64($\pm$3.91) |  10.68($\pm$3.18) |  1.22($\pm$0.47) |  31.24($\pm$5.42) |  45.74($\pm$8.68)
0.2 |  91.18($\pm$3.18) |  11.45($\pm$1.12) |  1.06($\pm$0.25) |  31.09($\pm$4.15) |  48.12($\pm$6.04)
0.5 |  90.19($\pm$3.46) |  10.90($\pm$1.53) |  1.51($\pm$0.30) |  30.68($\pm$3.01) |  49.53($\pm$8.44)
0.7 |  91.61($\pm$2.84) |  11.62($\pm$1.65) |  1.35($\pm$2.61) |  29.55($\pm$1.93) |  47.75($\pm$5.83)
(d) Commutative Lie Group VAE

Table 6: Baselines hyper-parameter tuning results on 3D Cars dataset with 10
random seeds.

$\beta$ | beta VAE | FVM | MIG | SAP | DCI
---|---|---|---|---|---
1.0 |  59.40($\pm$7.72) |  60.71($\pm$2.47) |  21.61($\pm$0.59) |  11.02($\pm$0.18) |  25.43($\pm$0.48)
2.0 |  56.80($\pm$7.90) |  54.69($\pm$2.96) |  19.97($\pm$0.31) |  10.45($\pm$0.24) |  21.15($\pm$0.47)
4.0 |  52.40($\pm$7.65) |  55.19($\pm$1.73) |  19.14($\pm$0.49) |  9.67($\pm$0.24) |  20.54($\pm$0.41)
6.0 |  52.67($\pm$7.28) |  53.42($\pm$1.54) |  18.05($\pm$0.27) |  10.10($\pm$0.28) |  21.03($\pm$0.27)
(a) $\beta$-VAE

$\beta$ | beta VAE | FVM | MIG | SAP | DCI
---|---|---|---|---|---
1.0 |  60.40($\pm$5.48) |  59.30($\pm$2.52) |  21.64($\pm$0.51) |  11.11($\pm$0.27) |  25.74($\pm$0.29)
2.0 |  56.60($\pm$9.24) |  59.48($\pm$2.14) |  21.72($\pm$0.44) |  11.08($\pm$0.35) |  23.74($\pm$0.33)
4.0 |  58.00($\pm$6.86) |  56.40($\pm$1.55) |  21.50($\pm$0.62) |  10.98($\pm$0.35) |  22.29($\pm$0.73)
6.0 |  56.00($\pm$8.17) |  55.46($\pm$1.42) |  21.49($\pm$0.52) |  10.50($\pm$0.25) |  20.24($\pm$0.51)
(b) $\beta$-TCVAE

$C$ | beta VAE | FVM | MIG | SAP | DCI
---|---|---|---|---|---
10.0 |  59.80($\pm$5.77) |  60.34($\pm$2.58) |  21.53($\pm$0.33) |  10.91($\pm$0.37) |  25.55($\pm$0.49)
12.0 |  60.20($\pm$10.60) |  61.00($\pm$1.86) |  21.39($\pm$0.41) |  11.25($\pm$0.32) |  25.71($\pm$0.37)
14.0 |  61.40($\pm$4.33) |  60.63($\pm$2.67) |  21.55($\pm$0.53) |  11.18($\pm$0.48) |  25.97($\pm$0.43)
16.0 |  60.20($\pm$7.69) |  60.50($\pm$2.89) |  21.72($\pm$0.31) |  11.30($\pm$0.41) |  25.60($\pm$0.33)
(c) Control-VAE

$loss_{\text{rec group}}$ | beta VAE | FVM | MIG | SAP | DCI
---|---|---|---|---|---
0.1 |  59.20($\pm$5.75) |  59.54($\pm$1.64) |  20.61($\pm$0.41) |  10.93($\pm$0.36) |  23.77($\pm$0.60)
0.2 |  63.40($\pm$9.14) |  59.74($\pm$1.60) |  20.87($\pm$0.36) |  10.80($\pm$0.47) |  23.59($\pm$0.63)
0.5 |  64.20($\pm$8.24) |  61.28($\pm$1.68) |  21.20($\pm$0.53) |  10.58($\pm$0.36) |  22.88($\pm$0.52)
0.7 |  62.60($\pm$5.17) |  62.26($\pm$1.71) |  21.39($\pm$0.67) |  10.71($\pm$0.33) |  22.95($\pm$0.62)
(d) Commutative Lie Group VAE

Table 7: Baselines hyper-parameter tuning results on smallNORB dataset with 10
random seeds.

p-value | FVM | MIG | SAP | DCI
---|---|---|---|---
dSprites | 0.011 | 0.005 | 0.016 | 0.001
3D Cars | 0.006 | 0.000 | 0.97 | 0.003
smallNORB | 0.000 | 0.002 | 0.000 | 1.000
Table 8: p-value estimation on each datasets. 3D Shapes |  $\beta$-VAE |  $\beta$-TCVAE |  Factor-VAE |  Control-VAE |  CLG-VAE |  OURS
---|---|---|---|---|---|---
m-FVM5 |  80.26($\pm$3.78) |  79.21($\pm$5.87) |  76.69($\pm$5.08) |  73.31($\pm$6.54) |  73.61($\pm$4.22) |  83.03($\pm$2.73)
Table 9: m-FVMs results.

Figure 6: Disentanglement Scores with box plots.

$\epsilon$ |  FVM |  beta VAE |  MIG |  SAP |  DCI
---|---|---|---|---|---
0.01 |  76.98($\pm$8.63) |  87.33($\pm$7.87) |  29.68($\pm$11.38) |  6.96($\pm$1.16) |  41.28($\pm$11.93)
0.1 |  82.21($\pm$1.34) |  90.33($\pm$5.85) |  34.79($\pm$3.26) |  7.45($\pm$0.61) |  48.07($\pm$5.62)
1.0 |  76.77($\pm$7.05) |  78.33($\pm$13.88) |  22.42($\pm$11.14) |  6.02($\pm$0.48) |  38.87($\pm$7.83)
(a) Hyper-parameter tuning with 6 random seeds.

3D Cars |  $|\mathcal{G}|$=100 |  $|\mathcal{G}|$=10
---|---|---
FVM |  95.70($\pm$1.90) |  48.63($\pm$24.55)
MIG |  18.58($\pm$1.24) |  2.99($\pm$6.04)
SAP |  1.43($\pm$0.18) |  0.29($\pm$0.34)
DCI |  34.81($\pm$3.85) |  6.12($\pm$10.44)
FVM2 |  62.43($\pm$8.08) |  37.94($\pm$10.01)
(b) Codebook size impact

Table 10: Table

##  Appendix D Additional Experiment

###  D.1 Disentanglement Performance

#### Statistically Significant Improvements

As shown in Figure 6, our model significantly improves disentanglement
learning.

#### 3D Shapes

As shown in Table 9, CFASL also shows an advantage on multi-factor change.

###  D.2 Ablation Studies

#### How Commutative Lie Group Improves Disetanglement Learning?

The Lie group is not commutative, however most factors of the used datasets
are commutative. For example, 3D Shapes dataset factors consist of the azimuth
(x-axis), yaw (z-axis), coloring, scale, and shape. Their 3D rotations are all
commutative. Also, other composite symmetries as coloring and scale are
commutative. Even though we restrict the Lie group to be commutative, our
model shows better results than baselines as shown in Table 1.

#### Impact of Hyper-Parameter tuning

We operate a grid search of the hyper-parameter $\epsilon$. As shown in Figure
7(a), the Kullback-Leibler divergence convergences to the highest value, when
$\epsilon$ is large ($\epsilon=1.0$) and it shows less stable results. It
implies that the CFASL with larger $\epsilon$ struggles with disentanglement
learning, and is shown in Tabel 10(a). Also, the $\mathcal{L}_{ee}$ in Figure
7(b) is larger than other cases, which implies that the model struggles with
extracting adequate composite symmetry because its encoder is far from the
equivariant model and it is also shown in Table 10(a). Even though
$\epsilon=0.01$ case shows the lowest value in the most loss,
$\mathcal{L}_{de}$ in Figure 7(e) is higher than others and it also implies
the model struggles with learning symmetries, as shown in Table 10(a) because
the model does not close to the equivariant model compare to $\epsilon=0.1$
case.

#### Impact of Factor-Aligned Symmetry Size

We set the codebook size as 100, and 10 to validate the robustness of our
method. In Table 10(b), the larger size shows better results than the smaller
one, and is more stable by showing a low standard deviation.

3D Cars | $\mathcal{L}_{c}$ |  without $\mathcal{L}_{c}$
---|---|---
|  x4.63 | x1.00
Table 11: Complexity.

#### Impact of Commutative Loss on Computational Complexity

As shown in Table 11, our methods reduce the composite symmetries computation.
Matrix exponential is based on the Taylor series and it needs high computation
cost though its approximation is lighter than the Taylor series. We need one
matrix exponential computation for composite symmetries with commutative loss,
in contrast, the other case needs the number of symmetry codebook elements
$|S|\cdot|SS|$ for the matrix exponential and also $|S|\cdot|SS|-1$ time
matrix multiplication.

(a) kld (hyper)

(b) encoder (hyper)

(c) kld (w/o)

(d) encoder (w/o)

(e) decoder (hyper)

(f) perpendicular (hyper)

(g) decoder (w/o)

(h) perpendicular (w/o)

(i) parallel (hyper)

(j) sparsity (hyper)

(k) parallel (w/o)

(l) sparsity (w/o)

Figure 7: Loss curves: 1) HT: hyper-parameter tuning ($\epsilon\in\\{0.01,0.1,1.0\\}$) with $\beta$-TCVAE based CFASL. 2) AB: ablation study with $\beta$-VAE based CFASL. Datasets | FVM | MIG | SAP | DCI
---|---|---|---|---
G-VAE |  CFASL |  G-VAE |  CFASL |  G-VAE |  CFASL |  G-VAE |  CFASL
dSprites |  69.75($\pm$13.66) |  82.30($\pm$5.64) |  21.09($\pm$9.20) |  33.62($\pm$8.18) |  5.45($\pm$2.25) |  7.28($\pm$0.63) |  31.08($\pm$10.87) |  46.52($\pm$6.18)
3D Car |  92.34($\pm$2.96) |  95.70($\pm$1.90) |  11.95($\pm$2.16) |  18.58($\pm$1.24) |  2.10($\pm$0.96) |  1.43($\pm$0.18) |  26.91($\pm$6.24) |  34.81($\pm$3.85)
smallNROB |  46.64($\pm$1.45) |  61.15($\pm$4.23) |  20.66($\pm$1.22) |  22.23($\pm$0.48) |  10.37($\pm$0.51) |  11.12($\pm$0.48) |  27.77($\pm$0.68) |  24.59($\pm$0.51)
Table 12: Comparison of disentanglement scores of plug-in methods in single
factor change.

#### Comparison of Plug-in Methods

To compare plug-in methods, we evaluate common disentanglement metrics on
G-VAE Shakerinava et al. [2022] and apply both methods to $\beta$-TCVAE. As
shown in Table 12, our method shows statistically significant improvements in
disentanglement learning although $\beta$ hyper-parameter of CFASL is smaller
than G-VAE. As shown in Table 8, we estimate the p-value over common
disentanglement metrics on each dataset. Most values show that improvements in
disentanglement learning are statistically significant.

###  D.3 Additional Qualitative Analysis (Baseline vs. CFASL)

Fig. 9-9 show the qualitative results on 3D Cars introduced in Fig. 5. Fig.
11-11, and Fig. 14 show the dSprites and smallNORB dataset results
respectively. Additionally, we describe Fig. 13-13results over 3D Shapes
datasets respectively. We randomly sample the images in all cases.

#### 3D Cars

As shown in Figure 8(c), CFASL shows better results than the baseline. In the
$1^{st}$ and $2^{nd}$ rows, the baseline changes shape and color factor when a
single dimension value is changed, but ours clearly disentangle the
representations. Also in the $3^{rd}$ row, the baseline struggles with
separating color and azimuth but CFASL successfully separates the color and
azimuth factors.

  * •

$1^{st}$ row: our model disentangles the shape and color factors when the
$2^{nd}$ dimension value is changed.

  * •

$2^{nd}$ row: ours disentangles shape and color factors when the $1^{st}$
dimension value is changed.

  * •

$4^{th}$ row: ours disentangles the color, and azimuth factors when the
$2^{nd}$ dimension value is changed.

#### dSprites

As shown in Figure 10(c), the CFASL shows better results than the baseline.
The CFASL significantly improves the disentanglement learning as shown in the
$4^{th}$ and $5^{th}$ rows. The baseline shows the multi-factor changes during
a single dimension value is changed, while ours disentangles all factors.

  * •

$1^{st}$ row: ours disentangles the x- and y-pos factor when the $2^{nd}$
dimension value is changed.

  * •

$2^{nd}$ row: ours disentangles the rotation and scale factor when the
$2^{nd}$ dimension value is changed.

  * •

$3^{rd}$ row: ours disentangles the x- and y-pos, and rotation factor when the
$1^{st}$ and $2^{nd}$ dimension values are changed.

  * •

$4^{th}$ row: ours disentangles the all factors when the $1^{st}$ and $2^{nd}$
dimension values are changed.

#### 3D Shapes

As shown in Figure 12(c), the CFASL shows better results than the baseline. In
the $1^{st}$, $3^{rd}$, and $5^{th}$ rows, our model clearly disentangles the
factors while the baseline struggles with disentangling multi-factors. Even
though our model does not clearly disentangle the factors, compared to the
baseline, which is too poor for disentanglement learning, ours improves the
performance.

  * •

$1^{st}$ row: our model disentangles the object color and floor color factor
when the $2^{nd}$ and $3^{rd}$ dimension values are changed.

  * •

$2^{nd}$ row: ours disentangles shape factor in $1^{st}$ dimension, and object
color and floor color factors at the $4^{th}$ dimension value are changed.

  * •

$3^{rd}$ row: ours disentangles the object color and floor color factor when
the $3^{rd}$ dimension value is changed.

  * •

$4^{th}$ row: ours disentangles the scale, object color, wall color, and floor
color factor when the $2^{nd}$ and $3^{rd}$ dimension values are changed.

  * •

$5^{th}$ row: ours disentangles the shape, object color, and floor color
factor when the $1^{st}$ and $2^{nd}$ dimension values are changed.

#### smallNORB

Even though our model does not clearly disentangle the multi-factor changes,
ours shows better results than the baseline as shown in Figure 14(c).

  * •

$1^{st}$ row: our model disentangles the category and light factor when the
$2^{nd}$ dimension value is changed.

  * •

$3^{rd}$ row: ours disentangles category factor and azimuth factors when the
$5^{th}$ dimension value is changed.

(a) Generated images by composite symmetry on 3D Cars dataset. The images in
the red box are inputs. The images in the blue box at odd column are same as ③
and even column are same as ⑤ in Fig. 5(a).

(b) Generated images by its factor-aligned symmetries on 3D Cars datset. The
images are same as ④ in Fig 5(a).

(c) Generated images by dimension change on 3D Cars dataset.

Figure 8: Generalization over unseen pairs of images on 3D Cars dataset.
Figure 9: Fig. 8(a) shows the generation quality of composite symmetries
results, Fig. 8(b) shows the disentanglement of symmetries by factors results,
and Fig. 8(c) shows the disentanglement of latent dimensions by factors
results.

(a) Generated images by composite symmetry on dSprites dataset. The images in
the red box are inputs. The images in the blue box at odd column are same as ③
and even column are same as ⑤ in Fig. 5(a).

(b) Generated images by its factor-aligned symmetries on dSprites datset. The
images are same as ④ in Fig 5(a).

(c) Generated images by dimension change on dSprites dataset.

Figure 10: Generalization over unseen pairs of images on dSprites dataset.
Figure 11: Fig. 10(a) shows the generation quality of composite symmetries
results, Fig. 10(b) shows the disentanglement of symmetries by factors
results, and Fig. 10(c) shows the disentanglement of latent dimensions by
factors results.

(a) Generated images by composite symmetry on 3DShapes dataset. The images in
the red box are inputs. The images in the blue box at odd column are same as ③
and even column are same as ⑤ in Fig. 5(a).

(b) Generated images by its factor-aligned symmetries on 3DShapes datset. The
images are same as ④ in Fig 5(a)

(c) Generated images by dimension change on 3DShapes dataset.

Figure 12: Generalization over unseen pairs of images on 3DShapes dataset.
Figure 13: Fig. 12(a) shows the generation quality of composite symmetries
results, Fig. 12(b) shows the disentanglement of symmetries by factors
results, and Fig. 12(c) shows the disentanglement of latent dimensions by
factors results.

(a) Generated images by composite symmetry on smallNORB dataset. The images in
the red box are inputs. The images in the blue box at odd column are same as ③
and even column are same as ⑤ in Fig. 5(a).

(b) Generated images by its factor-aligned symmetries on smallNORB datset. The
images are same as ④ in Fig 5(a)

(c) Generated images by dimension change on smallNORB dataset.

Figure 14: Fig. 14(a) shows the generation quality of composite symmetries
results, Fig. 14(b) shows the disentanglement of symmetries by factors
results, and Fig. 14(c) shows the disentanglement of latent dimensions by
factors results.