Learning Disentangled Semantic Spaces of Explanations via Invertible
  Neural Networks
Disentangled latent spaces usually have better semantic separability and
geometrical properties, which leads to better interpretability and more
controllable data generation. While this has been well investigated in Computer
Vision, in tasks such as image disentanglement, in the NLP domain sentence
disentanglement is still comparatively under-investigated. Most previous work
have concentrated on disentangling task-specific generative factors, such as
sentiment, within the context of style transfer. In this work, we focus on a
more general form of sentence disentanglement, targeting the localised
modification and control of more general sentence semantic features. To achieve
this, we contribute to a novel notion of sentence semantic disentanglement and
introduce a flow-based invertible neural network (INN) mechanism integrated
with a transformer-based language Autoencoder (AE) in order to deliver latent
spaces with better separability properties. Experimental results demonstrate
that the model can conform the distributed latent space into a better
semantically disentangled sentence space, leading to improved language
interpretability and controlled generation when compared to the recent
state-of-the-art language VAE models.
Yingji Zhang, Danilo S. Carvalho, André Freitas
# Learning Disentangled Semantic Spaces of Explanations

###### Abstract

Disentangling sentence representations over continuous spaces can be a critical process in improving interpretability and semantic control by localising explicit generative factors. Such process confers to neural-based language models some of the advantages that are characteristic of symbolic models, while keeping their flexibility. This work presents a methodology for disentangling the hidden space of a BERT-GPT2 autoencoder by transforming it into a more separable semantic space with the support of a flow-based invertible neural network (INN). Experimental results indicate that the INN can transform the distributed hidden space into a better semantically disentangled latent space, resulting in better interpretability and controllability, when compared to recent state-of-the-art models.

## 1 Introduction

Disentangled representations, in which each learned features of data refer to a semantically meaningful and independent concept (Bengio et al., 2012), are widely investigated and explored in the field of Computer Vision because of their interpretability and controllability (Higgins et al., 2017; Kim and Mnih, 2018). These works reveal that the semantic features of imaging data, such as mouth and eyes in face images, can be mapped to specific latent dimensions. However, the use of disentanglement for the representation of textual data is comparatively less-explored.

Recent work has started articulating how the disentangled factors of generative models can be used to support the representation of natural language definitions (Carvalho et al., 2022), and science explanations (Zhang et al., 2022), i.e. sentences which relate and compose scientific concepts. These works have the motivation of understanding whether the properties introduced by disentangled generative models can support a consistent organisation of the latent space, where syntactic and semantic transformations can be localised, interpolated and controlled. This category of models has fundamental practical implications. Firstly, similarly to the computer vision models, where image objects can be meaningfully interpolated and transformed, it can provide a framework for transforming and combining sentences which communicate complex concepts, definitions and explanations. Secondly, the localisation of latent factors allows sentence representation models to be more consistent and interpretable. For example, one recent work (Zhang et al., 2022) illustrates that the predicate-argument semantic structure of explanatory sentences from the WorldTree corpus (Jansen et al., 2018) could be partially disentangled through a Variational AutoEncoder based model (Optimus) (Li et al., 2020). For example, a simple explanatory fact such as "_animals require oxygen for survival_" can be projected into a latent space where each pair role-content, such as ARG0-animal or VERB-require, is described by a hypersolid over the latent space. In this case, the generation of explanations can be semantically controlled/manipulated in a coherent manner. For ex

Figure 1: Use of an invertible neural network to support better semantic disentanglement and separation of explanatory sentences.
ample, we can control the generation of sentences by manipulating the movement of latent vectors between different role-content regions (e.g., moving the representation of the sentence "_animals require oxygen for survival_" to an ARG1-warmth region would produce the sentence "_animals require warmth for survival_"). Such ability is inherently valuable to downstream tasks such as natural language inference (for argument alignment, substitution or abstraction on inference chains) or improving the consistency of neural search (by representing more consistent queries and sentences).

In this work, we build upon the seminal work of [22], proposing a better way to separate the predicate-argument structure of explanatory sentences in the latent space. Inspired by the work of [10], we apply an invertible neural network (INN), a control component, to learn the bijective transformation between the hidden space of the autoencoder and the smooth latent space of INN due to its low computational overhead and theoretically low information loss on bijective mapping.

More importantly, the transformation modelled by the proposed approach approximates the VAE-defined latent space, both being multivariate Gaussian. Thus, there is potential to learn a latent space with improved geometric properties. That is to say, the same semantic roles and associated content clusters can be better separated over the latent space modelled by the INN (an illustration can be found in Figure 1). In this case, we can improve control over the decoding process due to the reduction of overlapping (ambiguous) regions.

In summary, this work aims to explore the utilisation of flow-based INNs as a control component for controlling the generation of a typical neural language modelling autoencoder setting (e.g. BERT-GPT2). This addition can transform the latent space of an autoencoder (BERT-GPT2) into a constrained multivariate Gaussian space via INN in a supervised approach. In this latent space, different role-content regions can be better separated. This smoother and better separated space can be later operated over in order to improve the control of the generation of the autoencoder using geometric operators, such as traversal [17], interpolation [14], and vector arithmetic [16]. The following are our contributions:

**1.** We find that adding a flow-based INN is an effective mechanism for transforming the hidden space of the autoencoder into a smooth multivariate Gaussian latent space for representing sentences. It can be applied to arbitrary existing large-scale autoencoders without any further training. **2.** We put forward a supervised training strategy for INNs to learn a controllable semantic space with higher disentanglement than previous work. **3.** We introduce the use of this representation to support semantically coherent data augmentation (generating sentences). Our algorithm can increase the diversity of the data while keeping the distribution of the original data unchanged.

## 2 Related work

Sentence DisentanglementMercatali and Freitas (2021) pioneered the work on the use of disentanglement to control syntactic-level generative factors in sentence representations. More specialised architectures such as the Attention-Driven Variational Autoencoder (ADVAE) [12] were later introduced for learning disentangled syntactic latent spaces. Recent contributions moved in the direction of exploring disentanglement for encoding sentence semantics, where Carvalho et al. (2022) proposed a supervised training strategy for learning a disentangled representation of definitions by injecting the semantic role labelling inductive biases into the latent space, with the support of a conditional VAE. Comparatively, this work builds-upon on the recent advances of encoding semantic generative factors in latent spaces, focusing on the representation of explanatory statements, and proposing flow-based INN autoencoders as a mechanism to achieve improved separation and control.

INN in NLPThe properties of INN-based representations have recently started being investigated in language. [20] concentrate on modelling morphological inflection and lemmatization tasks, utilizing INN to learn a bijective transformation between the word surface and its morphemes. [11] focused on sentence-level representation learning, transforming sentences from a BERT sentence embedding space to standard Gaussian space, which improves sentence embeddings on a variety of semantic textual similarity tasks. Comparatively, this work is the first to explore the bijective mapping between the distributed sentence space of an autoencoder and a multivariate Gaussian space to improve the se
mantic separability and control over the distributed representation of sentences. Moreover, this is the first to explore this mechanism to support semantically coherent data augmentation.

## 3 Background

Disentangled semantic spacesZhang et al. (2022) demonstrates that semantic role supervision of explanations can induce disentanglement of semantic factors in a latent space of sentences modeled using the Optimus autoencoder configuration Li et al. (2020). Each _semantic role - content concept_ combination is described by a hypersolid (region) in the latent space. More information about the semantic roles can be found in Appendix B. Figure 1 illustrates examples of role-content/concept semantic clusters, such as _ARG1-shelter_ and _ARG1-warmth_ ("shelter" or "warmth" as direct objects) around the cluster _ARG0-animal_ ("animal" as the subject). However, the qualitative and quantitative analysis of Zhang et al. (2022) shows that role-content clusters are still substantially entangled. In order to support highly controlled operations over the latent space, this work addresses this limitation, demonstrating that the bijective mapping induced by INNs can provide a measurably better disentanglement and cluster separation.

Invertible Neural NetworksFlow-based INNs Dinh et al. (2014, 2016); Kingma and Dhariwal (2018) is a class of neural networks that models the bijective mapping between observation distribution \(p(x)\) and latent distribution \(p(z)\). In this case, we use \(T\) and \(T^{\prime}\) to represent forward mapping (from \(p(x)\) to \(p(z)\)) and backward mapping (from \(p(z)\) to \(p(x)\)), respectively. Unlike VAEs that approximate the posterior distribution to multivariate Gaussian distributions, INNs use multivariate Gaussian directly. The forward mapping can be learned by the following objective function:

\[\mathcal{L}=-\mathbb{E}_{x\sim p(x)}\Big{[}T(x)\Big{]}^{2}-\log\big{|}T^{ \prime}(x)\big{|} \tag{1}\]

where \(T(x)\) learns the transformation from \(x\) to \(z\sim N(0,1)\). \(|T^{\prime}(x)|\) is the determinant of the Jacobian, which indicates how much the transformation locally expands or contracts the space to ensure the integration of the probability density function is one.

## 4 Proposed Approach

Starting from the Optimus-based conditional VAE architecture proposed by Zhang et al. (2022), we encode each sentence \(x\) with an autoencoder and consider their sentence-level latent representation as the input of INNs, which is described as \(E(x)\). Next, we put forward two training strategies to map the hidden representations into a better semantically disentangled space.

### Training Strategy

Unsupervised INNsFirstly, we train the INN in an unsupervised fashion so that it minimizes the negative log-likelihood of the marginal distribution of latent representation \(z=E(x)\):

\[\begin{split}\mathcal{L}_{\text{unsup}}=&-\mathbb{ E}_{x\sim p(x)}\Big{[}T(E(x))\Big{]}^{2}\\ &-\log\big{|}T^{\prime}(E(x))\big{|}\end{split} \tag{2}\]

As this leads to a bijective mapping between distributed representation and the disentangled latent representation (multivariate Gaussian space), it allows us to explore the geometric clustering property of its latent space by traversal, interpolation, and latent space arithmetic.

Cluster-supervised INNAccording to the findings of Zhang et al. (2022) that the content of semantic role can be disentangled over the latent space approximated to multivariate Gaussian learned using the Optimus autoencoder setting, expanded with a conditional VAE term, we next train the INN to learn the embeddings, by minimizing the distance (cosine) between points in the same role-content regions and maximizing the distance between points in different regions, based on the explanation embeddings and their corresponding central point from the Optimus model. For example, given a sentence _"animal require food for survival"_ and its central vector of _ARG1-animal_, the training moves the sentence representation closer to the _ARG1-animal_ region center in the latent space of INN.

More specifically, during the calculation of the posterior, we replace the mean and variance of standard Gaussian distribution by the center point of its cluster and a hyper-parameter which should be less than one, respectively. In this case, each role-content cluster in the latent space will be mapped to a space where each cluster will have its embeddings more densely and regularly distributed around its
center. The objective function can be described as follows:

\[\begin{split}\mathcal{L}_{\text{sup}}=&-\mathbb{E}_{x \sim p_{cluster}(x)}\frac{\left[T(E(x))-\mu_{cluster}\right]^{2}}{1-\sigma^{2} }\\ &-\log\left|T^{\prime}(E(x))\right|\end{split} \tag{3}\]

where \(T(E(x))\) learns the transformation from \(x\) to \(z\sim N(\mu_{cluster},1-\sigma^{2})\). More training and architecture details are provided in Appendix A.

### Data Augmentation

To better capture the different features between distinct role-content clusters, more training sentences are needed in those clusters for training INNs. Therefore, we consider vector arithmetic and traversal as a systematic mechanism to support data augmentation. This is done as described in Equations 4. More details are provided in Appendix A.

\[\begin{split} vec&=average(E(s_{i}),E(s_{j}))\\ vec_{i}&=N(0,1)\quad\forall i\in\{0,..,size(vec)\} \\ s&=D(vec)\end{split} \tag{4}\]

where \(s_{k}\in S\) (explanation sentence corpus), \(E(s):S\rightarrow\mathbb{R}^{n}\) is the encoder (embedding) function, and \(D(e):\mathbb{R}^{n}\to S\) is the decoder function. The term \(vec_{i}=N(0,1)\) is done to resample each dimension and the last term generates a new sentence. Table 1 lists some randomly selected examples from augmented explanations.

## 5 Experiments

During the experiment, we consider both WorldTree (Jansen et al., 2018) and EntailmentBank (Dalvi et al., 2021) as datasets. The statistic information of datasets can be found in Appendix A.

### Disentanglement Analysis

In this section, we analyze the disentanglement of the latent space from the INN model independently of the cluster-based supervision. We empirically evaluate the properties of the latent space from two different perspectives: (i) quantitative: with the measurement of the disentanglement metrics, and (ii) qualitative: with the support of the geometrical operations (traversal, interpolation, and vector arithmetic) of the space which allows for eliciting the semantic-geometric behaviour of the space. We also qualitatively evaluate its reconstruction performance. Reconstruction examples have been included in the Appendix D.

Disentanglement metricsFirstly, we probe the ability of the model to disentangle the predicate-argument structure, such as ARG0 and PRED. Then, we compare its performance with the reference model (Optimus) under two training frameworks (Zhang et al., 2022) using six quantitative metrics for disentanglement. A description of the metrics is provided in Appendix C.

Table 2 summarises the results. The INN-based autoencoder can outperform the baseline model (Optimus) under three metrics: 5.1% in z-min-var, 42.3% in MIG, and 8.6% in Modularity. Those three metrics are plain statistical and do not depend on a trainable machine learning classifier, avoiding classifier fitting biases. We refer to (Carbonneau et al., 2022) for an in-depth discussion on a critical analysis of the strengths and limitations of

\begin{table}
\begin{tabular}{l c c c} \hline \hline \multicolumn{4}{c}{Optimus vs INN} \\ \hline model & z-min-var \(\downarrow\) & MIG & Modularity \\ \hline O(U) &.451 &.027 &.758 \\ O(S) &.453 &.067 &.753 \\ OC &.401 &.039 &.751 \\ INN & **.350** & **.491** & **.844** \\ \hline model & Disentanglement & Completeness & Informativeness \(\downarrow\) \\ \hline O(U) & **.307** &.493 & **.451** \\ O(S) &.302 &.491 &.466 \\ OC &.306 & **.493** &.474 \\ INN &.186 &.270 &.503 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Disentanglement metrics. O(U), O(S), O(C) are three different training strategies for Optimus.

\begin{table}
\begin{tabular}{l c c c} \hline \hline \multicolumn{4}{c}{
\begin{tabular}{l} Role-content & Augmented sentences \\ \end{tabular} } \\ \hline \multirow{4}{*}{ARG0-animal} & an animal requires energy to move \\  & animals produce offspring \\  & some adult animals lay eggs \\  & an animal requires shelter \\  & an animal can use its body to breathe \\ \hline \multirow{4}{*}{ARG0-human} & humans travel sometimes \\  & humans usually use gasoline \\  & humans sometimes endanger themselves \\  & humans use coal to make food \\  & humans depend on pollinators for survival \\ \hline \multirow{4}{*}{PRED-are} & wheels are a part of a car \\  & lenses are a part of eyeglasses \\ \cline{1-1}  & toxic chemicals are poisonous \\ \cline{1-1}  & green plants are a source of food for animals \\ \cline{1-1}  & copper and zinc are two metals \\ \hline \multirow{4}{*}{PRED-mean} & summit mean the top of the mountain \\ \cline{1-1}  & colder mean a decrease in heat energy \\ \cline{1-1}  & helping mean something can be done better \\ \cline{1-1}  & cleaner mean (less ; lower ) in pollutants \\ \cline{1-1}  & friction mean the product of a physical change \\ \hline \hline \end{tabular}
\end{table}
Table 1: Example of augmented explanations.
disentanglement metrics. Following the current methodological norm, we decided to include most used metrics. After quantitative evaluation, we next qualitatively assess its disentanglement.

TraversalThe traversal of a latent factor is obtained as the decoding of the vectors corresponding to the latent variables, where the evaluated factor is changed within a fixed interval, while all others are kept fixed. A disentangled representation should cause the decoded sentences to only change with respect to a single latent factor when that factor is traversed. In this experiment, the traversal is set up from a starting point given by a "seed" sentence. As illustrated in Table 3 we can observe that the generated sentences can hold concepts at different argument positions unchanged, localizing that specific semantic component at the sentences at a locus of the latent space. For example, the sentence traversed in the low dimension can hold the same semantic role-concept pairing _ARG0-animals_. During the traversal, the sentences present close variations (realizations in this case) of the semantic concept given by _animal_, such as _mammal_ and _predator_, tied to the same semantic role (ARG0).

InterpolationNext, we demonstrate the ability of INNs to provide smooth transitions between latent space representations of sentences [20]. In practice, the interpolation mechanism encodes two sentences \(x_{1}\) and \(x_{2}\) as \(z_{1}\) and \(z_{2}\), respectively. It interpolates a path \(z_{t}=z_{1}\cdot(1-t)+z_{2}\cdot t\) with \(t\) increased from \(0\) to \(1\) by a step size of \(0.1\). As a result, \(9\) sentences are generated on each interpolation step. If the latent space is semantically disentangled, the intermediate sentences should present discrete changes, with semantic roles changing between the endpoints \(x_{1}\) and \(x_{2}\) in each step. In Table 4, we provide qualitative results with latent space interpolations on explanation sentences. We can observe that the intermediate explanations could transition smoothly (i.e., no unrelated content between steps) from source to target. e.g., predicate changed from _eat_ to _must eat_ to _must hunt_, and ARG0s are changed from _humans_ to _some animals_.

Latent space arithmeticIn this part, we analyse whether averaging two input vectors with the same role-content preserve this property. If the averaged vector holds the same semantic concept as the input sentences, the latent space is better disentangled with regard to the induced semantic representation [21]. Table 5 shows examples of output sentences after vector averaging. We can observe that the lower dimensions can hold the same semantic information as input. However, this information is lost during the traversal of higher dimensions, which indicates that the latent space of INNs stores explanatory information differently from the Optimus baseline model. Therefore, we examine next whether our supervision method could better enforce separation and disentanglement.

\begin{table}
\begin{tabular}{|l|} \hline \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{} \\ \end{tabular}
\end{table}
Table 4: Interpolation examples where top and bottom sentences are source and target, respectively.
### Cluster-supervised INN model

After analyzing the disentanglement of the latent space of unsupervised INN, next, we examine could cluster-supervised lead to more separable latent space than Optimus. Reconstructed examples are provided in Appendix E.

Disentanglement between _ARG0_ clustersIn this case, we consider four ARG0 clusters, including _human_, _animal_, _plant_, and _something_, and evaluate model performance from two sides, including forward mapping and backward mapping. For forward mapping, we assess the disentanglement of the latent space of the INN model from two aspects (visualization and classification metrics). Figure 2 displays the distributions of four role-content clusters over the latent space. As we can observe, after the cluster-supervised training strategy, the embeddings are more concentrated on the center of their cluster, and there is a clear boundary between clusters, indicating better disentanglement. This visualization indicates that our supervised approach can help the INN-based architecture to learn a better separated semantic space when compared to the baseline models (Optimus, unsupervised INNs). Additionally, the unsupervised INN latent space (shown in the middle) does not display good separation compared with Optimus (left), which supports the result from latent space arithmetic 5.1, in that unsupervised INN stores explanations in a different way (not separating by role-content).

It is also observable that there are low-density embeddings at the transition between two clusters, which leads to the connection between clusters. We decode the middle datapoints between _animal_ and _human_ clusters and list them in Table 6. From those examples, we can observe that such explanations are related to both _animal_ and _human_ (e.g., _animals eat humans_). This result implies that the explanations may be geometrically represented in a similar way as they were originally designed in the WorldTree corpus (maximising lexical overlaps pred-arg alignments within an explanation chain), in which each explanation is typically linked with another through a subject or object abstraction/realisation, for supporting multi-hop inference tasks.

Next, we quantitatively evaluate the disentanglement of ARG0-content clusters (not like Table 2 that only evaluates the disentanglement by semantic roles). We consider classification task metrics (_accuracy_, _precision_, _recall_, _f1_) as proxies for evaluating region separability, effectively testing cluster membership across different clusters. The performance of those classifiers represents disentanglement performance. As shown in table 7, all classifiers trained over supervised latent representation outperform both unsupervised INN and Optimus, which indicates that the cluster-supervised approach leads to better disentanglement.

As for the evaluation of backward mapping, we calculate the ratio of generated sentences that hold the same role-content as the inputs (henceforth called invertibility ratio). We randomly selected 100 em

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multicolumn{5}{c}{ARG0: disentanglement proxy metrics} \\ \hline classifier & train & accuracy & precision & recall & f1 score \\ \hline \multirow{3}{*}{KNN} & O & 0.983 & 0.983 & 0.983 & 0.983 \\  & U & 0.972 & 0.972 & 0.972 & 0.972 \\  & C & **0.986** & **0.986** & **0.986** & **0.986** \\ \hline \multirow{3}{*}{NB} & O & 0.936 & 0.936 & 0.936 & 0.936 \\  & U & 0.961 & 0.961 & 0.961 & 0.961 \\  & C & **0.979** & **0.979** & **0.979** & **0.979** \\ \hline \multirow{3}{*}{SVM} & O & 0.979 & 0.979 & 0.979 & 0.979 \\  & U & 0.975 & 0.975 & 0.975 & 0.975 \\ \cline{1-1}  & C & **0.981** & **0.981** & **0.981** & **0.981** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Middle explanations between _ARG0-animal_ and _ARG0-human_.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multicolumn{5}{c}{ARG0: disentanglement proxy metrics} \\ \hline classifier & train & accuracy & precision & recall & f1 score \\ \hline \multirow{3}{*}{KNN} & O & 0.983 & 0.983 & 0.983 & 0.983 \\  & U & 0.972 & 0.972 & 0.972 & 0.972 \\  & C & **0.986** & **0.986** & **0.986** & **0.986** \\ \hline \multirow{3}{*}{NB} & O & 0.936 & 0.936 & 0.936 & 0.936 \\  & U & 0.961 & 0.961 & 0.961 & 0.961 \\ \cline{1-1}  & C & **0.979** & **0.979** & **0.979** & **0.979** \\ \hline \multirow{3}{*}{SVM} & O & 0.979 & 0.979 & 0.979 & 0.979 \\ \cline{1-1}  & U & 0.975 & 0.975 & 0.975 & 0.975 \\ \cline{1-1}  & C & **0.981** & **0.981** & **0.981** & **0.981** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Disentanglement of ARG0 between Optimus (O), unsupervised INN (U), and cluster-supervised INN (C) where KNN: k-neighbours, NB: naive bayes, SVM: support vector machine. The abbreviations are the same for remaining.

Figure 2: ARG0: t-SNE plot, different color represents different content regions (blue: animal, green: human, red: plant, purple: something) (left: Optimus, middle: unsupervised, right: cluster supervised).
beddings as inputs and show the corresponding ratios in Table 8. We can observe that both unsupervised and supervised cases can achieve high invertibility ratios, such as 0.99 of ARG0-plant in both of them, indicating that the INN component is performing its inverse mapping function, which means that INN provides us the means to control the sentence decoding step more precisely by operating the vector over its transformed latent space. If we compare the ratio between unsupervised and supervised, however, there is no significant difference between them, confirming the low information loss from the transformation.

Finally, we follow [22] on using decision trees to guide the movement of latent vectors over different clusters for controlling the explanation generation of the autoencoder. Table 9 shows the generation step following the path from _animal_ to _something_. We can observe from the results that under the guidance of the decision tree, the ARG0 content of the generated explanations gradually changes from _animals_ to _something_, and these generated explanations can maintain the semantics related to _animals_ even though the content of target, _something_, is not related to _animal_, which indicates that the generation of sentence can be localised control via supervised INN.

#### 4.2.2 Disentanglement between _Argi_ clusters

Next, we consider four ARG1 clusters, including _ARG1-food_, _ARG1-oxygen_, _ARG1-sun_, _ARG1-water_, and evaluate model performance following the same procedure. Figure 3 first displays the distributions of four role-content clusters over the latent space. With similar observations as before, the INN-supervised training strategy can learn the better disentanglement between ARG1 clusters. Additionally, when compared with the ARG0 cluster, the Optimus model does not show observable disentanglement.

Table 10 shows the disentanglement metrics (top) and invertibility ratio (bottom). With similar observations as the previous experiment, all classifiers trained over supervised latent representation outperform both the unsupervised INN model and Optimus, and both unsupervised and supervised cases can achieve higher ratio (at least 0.95). We also evaluate the results for _PRED_ clusters. Same observation as both _ARG0_ and _ARG1_. More information can be found in Appendix F due to the page limitation.

#### 4.2.3 Disentanglement between _Animal_ clusters

Before that, we investigated the separation between the same semantic roles but different content clusters. Next, we explore separating different semantic

\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{5}{c}{ARG1: disentanglement proxy metrics (forward: \(T\))} \\ \hline classifier & train & accuracy & precision & recall & f1 score \\ \hline \multirow{3}{*}{KNN} & O & 0.958 & 0.958 & 0.958 \\  & U & 0.951 & 0.951 & 0.951 \\  & C & **0.969** & **0.969** & **0.969** \\ \hline \multirow{3}{*}{NB} & O & 0.907 & 0.907 & 0.907 \\  & U & 0.926 & 0.926 & 0.926 & 0.926 \\  & C & **0.956** & **0.956** & **0.956** \\ \hline \multirow{3}{*}{SVM} & O & 0.956 & 0.956 & 0.956 & 0.956 \\  & U & 0.953 & 0.953 & 0.953 & 0.953 \\ \cline{1-1}  & C & **0.958** & **0.958** & **0.958** & **0.958** \\ \hline \multicolumn{5}{c}{ARG1: invertibility ratio (backward: \(T^{\prime}\))} \\ \hline train & food & oxygen & sun & water \\ \hline U & **0.99** & **0.98** & 0.95 & 1.00 \\ C & 0.96 & 0.95 & **0.96** & **1.00** \\ \hline \hline \end{tabular}
\end{table}
Table 10: Forward and backward evaluation for ARG1.

Figure 3: ARG1: t-SNE plot (blue: _food_, green: _oxygen_, red: _sun_, purple: _water_) (left: Optimus, middle: unsupervised INN, right: cluster supervised INN).

\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{5}{c}{ARG0: invertibility ratio (backward: \(T^{\prime}\))} \\ \hline train & human & animal & plant & something \\ \hline U & 0.98 & **0.89** & 0.99 & **1.00** \\ C & **1.00** & 0.86 & **0.99** & 0.95 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Invertibility test for ARG0.
roles with the same content. We thus focus on the _animal_ cluster, and investigate the disentanglement between _ARGO-animal_, _ARG1-animal_, and _ARG2-animal_. As illustrated in Figure 4, the animal clusters with different semantic roles can be separated after cluster-supervised training, which indicates that the INN model can capture the difference between the same contents with different semantic roles in the case of similar topic. That is to say, the INN-based approach could jointly learn separable embeddings w.r.t. role-content and content alone.

Table 11 shows the disentanglement metrics and the invertibility ratio. Similarly to the previous experiment, the supervised case outperforms both the unsupervised and the Optimus models. We can also observe that both unsupervised and supervised cases can achieve good invertibility (at least 90%).

Table 12 shows the decoded explanations traversed around the central point of each cluster in the latent space of cluster-supervised INN. From it, we can observe that the INN-based model can generate explanations that hold the same role as its cluster, indicating that INN can separate the information of different semantic roles in similar contextual information.

## 6 Conclusions

In this work, we first analyze the disentanglement of the latent space of INN. The experimental results indicate that it can transform the distributed hidden space from a BERT-GPT2 autoencoder into a smooth latent space where syntactic and semantic transformations can be localised, interpolated and controlled. Secondly, we propose a supervised training strategy for INNs, which leads to an improved disentangled and separated space. This property can help us control the autoencoder generation by manipulating the movement of latent vectors. Thirdly, we utilize these geometric properties and semantic controls to support a semantically coherent and controlled data augmentation strategy.

## 7 Limitations

This work explores how flow-based INN autoencoders can support better semantic disentanglement and separation for sentence representations over continuous sentence spaces. While this work is motivated by providing more localised distributed representations, which can impact the safety and coherence of generative models, the specific safety guarantees of these models are not fully established.
