# Unsupervised Semantic Variation Prediction
using the Distribution of Sibling Embeddings

Taichi Aida
Tokyo Metropolitan University
<EMAIL_ADDRESS>&Danushka Bollegala
Amazon, University of Liverpool
<EMAIL_ADDRESS>
###### Abstract

Languages are dynamic entities, where the meanings associated with words
constantly change with time. Detecting the semantic variation of words is an
important task for various NLP applications that must make time-sensitive
predictions. Existing work on semantic variation prediction have predominantly
focused on comparing some form of an averaged contextualised representation of
a target word computed from a given corpus. However, some of the previously
associated meanings of a target word can become obsolete over time (e.g.
meaning of _gay_ as _happy_), while novel usages of existing words are
observed (e.g. meaning of _cell_ as a mobile phone). We argue that mean
representations alone cannot accurately capture such semantic variations and
propose a method that uses the entire cohort of the contextualised embeddings
of the target word, which we refer to as the _sibling distribution_.
Experimental results on SemEval-2020 Task 1 benchmark dataset for semantic
variation prediction show that our method outperforms prior work that consider
only the mean embeddings, and is comparable to the current state-of-the-art.
Moreover, a qualitative analysis shows that our method detects important
semantic changes in words that are not captured by the existing methods.
111Source code is available at https://github.com/a1da4/svp-gauss .

##  § 1 Introduction

(a) gay

(b) cell

Figure 1: t-SNE projections of BERT token vectors (dotted) in two time periods
and the average vector (starred) for each period. (a) the word gay has lost
its original meaning related to happy and is now used to mean homosexual,
resulting in a significant shift in its distribution. (b) the word cell is now
also used to mean cell phone, while retaining the meaning of prison or
biology, widening the distribution but not significantly changing the mean
vector.

The meaning of words evolves over time, and even in everyday life,
technological innovations and cultural aspects can cause a word to have a
different meaning than in the past. For example, the meaning of the word gay
has completely changed from happy to homosexual (Figure 1), and cell has added
cell phone to its previous meanings of prison and biology (Figure 1). In the
semantic change detection task, the goal is to detect the words whose meanings
have changed across time-specific corpora Kutuzov et al. (2018); Tahmasebi et
al. (2021).

As illustrated in Figure 1, we can identify two types of semantic changes
associated with words – (a) the word gay obtains a new meaning by replacing
its past meaning (Figure 1), whereas (b) the word cell obtains a new meaning,
while preserving its past meanings (Figure 1). On the other hand, much prior
work have resort to a scheme where they first individually represent the
meaning of a target word in a given time-specific corpora using a single
embedding, such as the mean of the non-contextualised Kim et al. (2014);
Kulkarni et al. (2015); Hamilton et al. (2016); Yao et al. (2018); Dubossarsky
et al. (2019); Aida et al. (2021) or contextualised Martinc et al. (2020);
Beck (2020); Kutuzov and Giulianelli (2020); Rosin et al. (2022); Rosin and
Radinsky (2022) embeddings of the target word taken over all of its occurring
contexts in the corpus. Next, various distance measures are used to compare
those embeddings to quantify the semantic variation of the target word across
corpora. However, as seen from Figure 1, using the mean embedding of a target
word alone for predicting semantic variations of words can be misleading
especially when the variance of the embedding distribution is large.

To address the above-mentioned limitations, we use the distribution of
contextualised embeddings of a target word $w$ in all of its occurrence
contexts $\mathcal{S}(w)$ in a given corpus, which we refer to as the _sibling
distribution_ Zhou et al. (2022) of $w$. We then approximate the sibling
distribution of a word using a multivariate Gaussian, which has shown to
accurately capture the uncertainty in word embedding spaces Vilnis and
McCallum (2015); Iwamoto and Yukawa (2020); Yüksel et al. (2021). We can then
use a broad range of distance and divergence measures defined over Gaussian
distributions to quantify the semantic variation of a target word across
multiple time-specific corpora.

Experimental results on SemEval-2020 Task 1 benchmark dataset show that our
proposed method outperforms several prior methods, and achieves comparable
performance to the current state-of-the-art (SoTA) Rosin and Radinsky (2022).
More importantly, our proposal to model both the mean and variance of sibling
embeddings consistently outperforms methods that use only the mean
contextualised embedding from the same Masked Language Model (MLM) Rosin and
Radinsky (2022). Moreover, for computational convenience, prior work had
assumed the covariance matrix of sibling embeddings to be diagonal Iwamoto and
Yukawa (2020); Yüksel et al. (2021), but we show that further performance
improvements can be obtained by using the full covariance matrix.

##  § 2 Related Work

Historically, the diachronic semantic changes of words have been studied by
linguists Tahmasebi et al. (2021), which has also received much attention
lately within the NLP community. Automatic detection of words whose meanings
change over time has provided important insights for diverse fields such as
linguistics, lexicology, sociology, and information retrieval (IR) Traugott
and Dasher (2001); Cook and Stevenson (2010); Michel et al. (2011); Kutuzov et
al. (2018). For example, in IR one must know the seasonal association of
keywords used in user queries to provide relevant results pertaining to a
particular time period. Moreover, it has been shown that the performance of
publicly available pretrained foundation models Bommasani et al. (2021)
declines over time when applied to emerging data Loureiro et al. (2022);
Lazaridou et al. (2021) because they are trained using a static snapshot. Su
et al. (2022) showed that the temporal generalisation of foundation models is
closely related to their ability to detect semantic variations of words.

Semantic change detection is modelled in the literature as an unsupervised
task of detecting words whose meanings change between two given time-specific
corpora Kutuzov et al. (2018); Tahmasebi et al. (2021). In recent years,
several shared tasks have been held Schlechtweg et al. (2020); Basile et al.
(2020); Kutuzov and Pivovarova (2021), where participants are required to
predict the degree or presence of semantic changes for a given target word
between two given corpora sampled from different time periods. For this
purpose, much prior work have used non-contextualised or contextualised word
embeddings to represent the meaning of the target word in each corpus. Unlike
non-contextualised word embeddings, which represent a word by the same vector
in all of its contexts, contextualised word embeddings represent the same
target word with different vectors in different contexts. Various methods have
been proposed to map vector spaces from different time periods, such as
initialisation Kim et al. (2014), alignment Kulkarni et al. (2015); Hamilton
et al. (2016), and joint learning Yao et al. (2018); Dubossarsky et al.
(2019); Aida et al. (2021).

The existing methods that have been proposed for the semantic variation
detection of words can be broadly categorised into two groups: (a) methods
that compare word/context clusters Hu et al. (2019); Giulianelli et al.
(2020); Montariol et al. (2021), and (b) methods that compare embeddings of
the target words computed from different corpora sampled at different time
periods Martinc et al. (2020); Beck (2020); Kutuzov and Giulianelli (2020);
Rosin et al. (2022). Recently, it has been reported that adding time-specific
attention mechanisms Rosin and Radinsky (2022) achieves SoTA performance.
However, this model requires additional training of the entire MLM including
the time-specific mechanisms, which is computationally costly for large-scale
MLMs.

Despite the recent success of using word embeddings for the semantic change
detection task, many of these methods struggle to detect meaning changes of
words which have a wide range of usages because they use only the mean
embedding to represent a target word Kutuzov et al. (2022). Although methods
that use point estimates in the embedding space, such as using non-
contextualised word embeddings or comparing the average of contextualised word
embeddings, are able to detect semantic variations that result in a loss of a
prior meaning (e.g. gay in Figure 1), they are inadequate when detecting
semantic variations due to novel usages of words, while preserving their
former meanings (e.g. cell in Figure 1).

To alleviate this problem, some studies have used Gaussian Embeddings Vilnis
and McCallum (2015) for semantic change detection Iwamoto and Yukawa (2020);
Yüksel et al. (2021). They used the mean and the diagonal approximation of the
covariance matrix computed using non-contextualised word embeddings. However,
as argued previously, contextualised embeddings provide useful clues regarding
the meaning of a word as used in a context. Therefore, in our proposed method,
we consider the entire cohort of contextualised word embeddings of a target
word taken across all of its occurring contexts (i.e. siblings) obtained from
an MLM. As confirmed later by the evaluations presented in § 4.4, our proposed
method consistently outperforms the methods proposed by Iwamoto and Yukawa
(2020) and Yüksel et al. (2021) that use non-contextualised embeddings.

##  § 3 Semantic Variation Prediction

Let us consider a target word $w$ that occurs in two given corpora $C_{1}$ and
$C_{2}$. For example, $C_{1}$ and $C_{2}$ could have been sampled at two
distinct time slots, respectively $T_{1}$ and $T_{2}$, reflecting any
_temporal_ semantic variations of words, or alternatively sampled at similar
periods in time but from distinct domains (e.g. _biology_ vs. _law_)
expressing semantic variations of words due to the differences in the
_domains_. Our goal in this paper is to propose a method that can accurately
predict whether $w$ is used in the same meaning in both $C_{1}$ and $C_{2}$
(i.e. $w$ is semantically invariant across the two corpora) or otherwise (i.e.
its meaning is different in the two corpora). Although we consider two corpora
in the subsequent description for simplicity of the disposition, our proposed
method can be easily extended to measure the semantic variation of a word over
multiple corpora.

According to the distributional hypothesis Firth (1957), the context in which
a word occurs provides useful clues regarding its meaning. Contextualised word
embeddings such as the ones produced by MLMs have shown to concisely and
accurately encode contextual information related to a target word in a given
context. For example, Zhou and Bollegala (2021) showed that contextualised
word embeddings can be used to induce word-sense embeddings that represent the
distinct senses of an ambiguous word with different vectors. Inspired by such
prior work using contextualised word embeddings as a proxy for accessing
contextual information related to a target word, we propose a method to detect
the semantic variations of a target word using its multiple occurrences in a
corpus.

To describe our proposed method in detail, let us denote the set of contexts
containing $w$ in corpus $C_{i}$ by $\mathcal{S}(w,C_{i})$. The scope of the
context of $w$ could be limited to a predefined fixed token window or extended
to the entire sentence containing $w$ as we do in our experiments. Let us
denote the contextualised (token) embedding of $w$ in a context
$s\in\mathcal{S}(w,C_{i})$ produced by an MLM $M$ by
$\boldsymbol{f}_{M}(w,s)\in\mathbb{R}^{d}$, where $d$ is the dimensionality of
the token embeddings produced by $M$. Following the terminology introduced by
Zhou et al. (2022), we refer to type embedding $\boldsymbol{f}_{M}(w,s)$ as
the _sibling_ embeddings of $w$ in context $s$. The number of siblings of $w$
in $C_{i}$ is denoted by $N^{w}_{i}=|\mathcal{S}(w,C_{i})|$. Moreover, let the
set of sibling embeddings of $w$ created from its occurrences in $C_{i}$ to be
$\mathcal{D}(w,C_{i})=\\{\boldsymbol{f}_{M}(w,s)|s\in\mathcal{S}(w,C_{i})\\}$.
As we later see, the distribution of sibling embeddings of a word $w$ encodes
information about the usage of $w$ in a corpus, which is useful for predicting
any semantic variations of $w$ across different corpora.

We can obtain a context-independent embedding,
$\boldsymbol{\mu}^{w}_{i}\in\mathbb{R}^{d}$ for $w$ by averaging all of its
sibling embeddings over the contexts as given by (1).

$\displaystyle\boldsymbol{\mu}^{w}_{i}=\frac{1}{N^{w}_{i}}\sum_{s\in\mathcal{S}(w,C_{i})}\boldsymbol{f}_{M}(w,s)$
(1)

Although much prior work has used $\boldsymbol{\mu}^{w}_{i}$ as a proxy for
the usage of $w$ in $C_{i}$ for numerous tasks such as studying the properties
of contextualised embeddings Ethayarajh (2019) and predicting semantic
variation of words Martinc et al. (2020); Beck (2020); Kutuzov and Giulianelli
(2020); Rosin et al. (2022); Rosin and Radinsky (2022), the mean of the
sibling embedding distribution is insensitive to the rare yet important usages
of the target word. In particular, when the sibling embedding distribution is
not uniformly distributed around its mean, the mean embedding can be
misleading as a representation of the distribution. To overcome this
limitation, in addition to $\boldsymbol{\mu}^{w}_{i}$, we also use the co-
variance matrix $\text{\bf V}^{w}_{i}\in\mathbb{R}^{d\times d}$ computed from
the sibling embedding distribution of $w$ as defined by (2).

$\displaystyle\text{\bf
V}^{w}_{i}=\frac{1}{N^{w}_{i}(N^{w}_{i}-1)}\\!\\!\sum_{s\in\mathcal{S}(w,C_{i})}\\!\\!\\!\\!\boldsymbol{f}_{M}(w,s)\boldsymbol{f}_{M}(w,s){}^{\top}$
(2)

We approximate the distribution of sibling embeddings of $w$ using a Gaussian,
$\mathcal{N}(\boldsymbol{\mu}^{w}_{i},\text{\bf V}^{w}_{i})$ with mean and
variance given respectively by (1) and (2). Gaussian distribution is the
maximum entropy distribution over the real values given a finite mean and
covariance and no further information Jaynes (2003). Moreover, by
approximating the sibling distribution as a Gaussian, we can use a broad range
of distance and divergence measures for quantifying the semantic variation of
$w$ across corpora. In the field of information theory, MLMs have been shown
to store the information of a given sentence in a vector Pimentel et al.
(2020). There is a strong correlation between the word frequency $N^{w}_{i}$
and the rank of its covariance matrix $\text{\bf V}^{w}_{i}$ (Figure 2 in
Appendix A), which indicates that covariance matrix also retains important
information regarding sibling embedding distribution. This observation further
supports our proposal to represent target words by $\boldsymbol{\mu}^{w}_{i}$
and $\text{\bf V}^{w}_{i}$.

###  § 3.1 Quantifying Semantic Variations

Given a target word $w$, following the method described above, we represent
$w$ in $C_{1}$ and $C_{2}$ respectively by the two Gaussian distributions
$\mathcal{N}(\boldsymbol{\mu}^{w}_{1},\text{\bf V}^{w}_{1})$ and
$\mathcal{N}(\boldsymbol{\mu}^{w}_{2},\text{\bf V}^{w}_{2})$. We can then
compute a _semantic variation score_ for $w$ that indicates how likely the
meaning of $w$ has changed from $C_{1}$ to $C_{2}$ by using different distance
(or divergence) measures to quantify the differences between two Gaussians.
For this purpose, we use two types of measures.

Divergence measures quantify the divergence between two distributions. We use
two divergence measures in our experiments: Kullback-Liebler (KL) divergence
and Jeffrey’s divergence. Given that we approximate sibling distribution of
$w$ in a corpus by a Gaussian, we can analytically compute both KL and
Jeffery’s divergence measures using
$\boldsymbol{\mu}^{w}_{1},\boldsymbol{\mu}^{w}_{2},\text{\bf V}^{w}_{1}$ and
$\text{\bf V}^{w}_{2}$ in closed-form formulas (Appendix B).

Distance measures are defined between two points in the sibling embedding
space. We use the seven distance measures: Bray-Curtis, Canberra, Chebyshev,
City Block, Correlation, Cosine, and Euclidean. The definitions of the
distance measures used in this paper are provided in Appendix C. Given a
distance measure $\psi(\boldsymbol{w}_{1},\boldsymbol{w}_{2})$ that takes two
$d$-dimensional sibling embeddings of $w$, each computed from contexts
selected respectively from $C_{1}$ and $C_{2}$ and returns a nonzero real
number indicating the distance between $\boldsymbol{w}_{1}$ and
$\boldsymbol{w}_{2}$, we compute the semantic variation score, ${\rm
score}(w)$, of $w$ between $C_{1}$ and $C_{2}$ as the average distance over
all pairwise comparisons between the sibling embeddings as given by (3).

$\displaystyle{\rm
score}(w)=\frac{1}{N^{w}_{1}N^{w}_{2}}\sum_{\begin{subarray}{c}\boldsymbol{w}_{1}\in\mathcal{D}(w,C_{1})\\\
\boldsymbol{w}_{2}\in\mathcal{D}(w,C_{2})\end{subarray}}\psi(\boldsymbol{w}_{1},\boldsymbol{w}_{2})$
(3)

The number of occurrences of some target words $w$ can be significantly
different between $C_{1}$ and $C_{2}$, which can make the computation of (3)
biased towards the corpus with more contexts for $w$. To overcome this issue,
instead of using sibling embeddings of $w$ computed from actual occurrence
contexts of $w$, we sample equal numbers of sibling embeddings from
$\mathcal{N}(\boldsymbol{\mu}^{w}_{1},\text{\bf V}^{w}_{1})$ and
$\mathcal{N}(\boldsymbol{\mu}^{w}_{2},\text{\bf V}^{w}_{2})$. Samples can be
drawn efficiently from a multidimensional Gaussian by first drawing samples
from a standard normal distribution (i.e. with zero mean and unit variance)
and subsequently applying a affine transformation parametrised by the
$\boldsymbol{\mu}^{w}_{i}$ and $\text{\bf V}^{w}_{i}$ of the associated
sibling distribution.

##  § 4 Experiments

###  § 4.1 Data and Metric

We use the SemEval-2020 Task 1 English dataset222It is licensed under a
Creative Commons Attribution 4.0 International License. Schlechtweg et al.
(2020) to evaluate the performance in detecting words whose meanings change
between time periods. This task includes two subtasks, classification and
ranking. In the classification task, the words in the evaluation set must be
classified as to whether they have semantically changed over time or
otherwise. Classification accuracy is used as the evaluation metric for this
task. On the other hand, in the ranking task, the words in the evaluation set
must be sorted according to the degree of semantic change. Spearman’s rank
correlation coefficient between the human-rated gold scores and the induced
ranking scores is used as the evaluation metric for this task. In this study,
the evaluation is conducted on the ranking task using English data. We do not
perform the classification task because no validation set is available for
tuning a classification threshold.

Statistics of the data used in our experiments are in Table 1. This data
includes two corpora from different centuries extracted from CCOHA Alatrash et
al. (2020). Let us denote the early 1800s and late 1900s to early 2000s
corpora respectively by $C_{1}$ and $C_{2}$. The test set has 37 target words
that are selected for indicating whether they have undergone a semantic change
between the two time periods. These words are annotated indicating whether
their meaning has changed over time and the degree of their semantic change.

Time Period | #Sentences | #Tokens | #Types
---|---|---|---
1810s–1860s | 254k | 6.5M | 87k
1960s–2010s | 354k | 6.7M | 150k
Table 1: Statistics of the SemEval-2020 Task 1 English dataset Schlechtweg et
al. (2020).

###  § 4.2 Setup

We use two types of BERT-base models as the MLM in our experiments: a publicly
available pretrained model333https://huggingface.co/bert-base-uncased
(MLM${}_{\textit{pre}}$) and a fine-tuned model (MLM${}_{\textit{temp}}$) from
MLM${}_{\textit{pre}}$ Rosin et al. (2022). The base model consists of 12
layers, which we use in two different configurations: (a) we use the last
layer (MLM${}_{\textit{pre}|\textit{temp},\textrm{last}}$), and (b) the mean-
pool over the last four layers
(MLM${}_{\textit{pre}|\textit{temp},\textrm{four}}$), which has shown good
performance across languages following Laicher et al. (2021). Rosin and
Radinsky (2022) recommend using the mean pooling over all (12) hidden layers.
However, we found no statistically significant differences between the mean-
pool over all layers vs. the last four layers in our preliminary experiments.

In the prediction of the degree of semantic change for a given word, the set
of sibling embeddings for each time period $\mathcal{D}(w,C_{1})$ and
$\mathcal{D}(w,C_{2})$ is acquired from all occurrences in each corpus using
the MLM described above, and the distributions across time periods
$\mathcal{N}(\boldsymbol{\mu}^{w}_{1},\text{\bf V}^{w}_{1})$ and
$\mathcal{N}(\boldsymbol{\mu}^{w}_{2},\text{\bf V}^{w}_{2})$ are compared. For
calculating the seven distance measures, we sample 1,000 sibling embeddings
from each sibling distribution. We use the covariance matrix of the sibling
embedding, which defines the distribution, only for the diagonal components
(diag(cov)) in the divergence measures,444In the above two divergences, it is
necessary to calculate the inverse of the covariance matrix, but in the case
of full components, it is often impossible to calculate the inverse matrix
because it is not regular. and both diagonal and full components (full(cov))
in the distance measures. Previous studies assume that the covariance matrix
is diagonal (diag(cov)) Iwamoto and Yukawa (2020); Yüksel et al. (2021). This
assumption increases computational efficiency compared to full(cov), at the
expense of loosing information on the non-diagonal elements. In our settings,
representation of a sibling distribution
$\mathcal{N}(\boldsymbol{\mu}^{w}_{i},\text{\bf V}^{w}_{i})$ in diag(cov) or
full(cov) requires $2d$ or $d(1+d)$ parameters, respectively.

| Model
---|---
Measure | MLM${}_{\textit{pre},\textrm{last}}$ | MLM${}_{\textit{pre},\textrm{four}}$ | MLM${}_{\textit{temp},\textrm{last}}$ | MLM${}_{\textit{temp},\textrm{four}}$
KL($C_{1}||C_{2}$) | 0.075 | 0.130 | 0.414 | 0.431
KL($C_{2}||C_{1}$) | 0.100 | 0.117 | 0.361 | 0.411
Jeff($C_{1}||C_{2}$) | 0.090 | 0.129 | 0.391 | 0.409
Bray-Curtis | 0.217 | 0.241 | 0.464 | 0.480
Canberra | 0.192 | 0.251 | 0.455 | 0.517
Chebyshev | 0.154 | 0.166 | 0.517 | 0.478
City Block | 0.198 | 0.140 | 0.461 | 0.459
Correlation | 0.191 | 0.266 | 0.480 | 0.463
Cosine | 0.190 | 0.270 | 0.478 | 0.480
Euclidean | 0.198 | 0.249 | 0.473 | 0.474
Table 2: Results of two divergences and seven distance functions under various MLM conditions with the proposed method using diag(cov). The best performance in each MLM condition is shown in bold. $C_{1}$ and $C_{2}$ refer to the early 1800s and late 1900s to early 2000s corpora, respectively. We report two types of KL divergence because of its asymmetric nature. Unlike KL divergence, Jeffrey’s divergence is symmetric, and we report just one result. | Model
---|---
Measure | MLM${}_{\textit{pre},\textrm{last}}$ | MLM${}_{\textit{pre},\textrm{four}}$ | MLM${}_{\textit{temp},\textrm{last}}$ | MLM${}_{\textit{temp},\textrm{four}}$
Bray-Curtis | 0.219 | 0.263 | 0.460 | 0.467
Canberra | 0.195 | 0.246 | 0.502 | 0.489
Chebyshev | 0.145 | 0.132 | 0.529 | 0.451
City Block | 0.192 | 0.248 | 0.414 | 0.452
Correlation | 0.181 | 0.286 | 0.481 | 0.468
Cosine | 0.189 | 0.272 | 0.479 | 0.454
Euclidean | 0.204 | 0.231 | 0.454 | 0.457
Table 3: Results of two divergences and seven distance functions under various
MLM conditions with the proposed method using full(cov). The best performance
in each MLM condition is shown in bold.

###  § 4.3 Result

We show the results of the proposed method under various conditions in Table 2
and Table 3. As reported in previous studies Rosin et al. (2022); Rosin and
Radinsky (2022), we find that the fine-tuned model (MLM${}_{\textit{temp}}$)
achieves high performance in all settings. Moreover, for the hidden layers, we
have confirmed that our method, by using the last four layers
(MLM${}_{\textit{pre}|\textit{temp},\textrm{four}}$), yields even higher
correlations than using only the last layer
(MLM${}_{\textit{pre}|\textit{temp},\textrm{last}}$).

#### Prediction measures.

Our method allows us to try a variety of measures. In the diag(cov) setting,
we try two divergences and seven distance measures. Comparing within
divergence measures, Table 2 shows that $\textrm{KL}(C_{1}||C_{2})$ achieves
high performance in all MLM conditions. This result means that many existing
words acquire novel meanings. On the other hand, comparing the distance
measures, we find that Canberra and Chebyshev outperform the commonly used
cosine distance in MLM${}_{\textit{temp}}$ (Table 2 and Table 3). Since the
cosine distance makes underestimations in MLMs Zhou et al. (2022), this result
suggests that it is better to calculate the absolute distance per dimension as
in Canberra and Chebyshev.

#### Components of the covariance matrices.

When applying the distance measures, the vectors can be extracted from the
full or diagonal covariance matrix. From Table 3 we see that using all
components of the covariance matrix (full(cov)) further improves performance
obtaining a correlation coefficient of $0.529$
(MLM${}_{\textit{temp},\textrm{last}}$, full(cov), Chebyshev). Previous
studies had assumed that the covariance matrix is diagonal for computational
convenience Iwamoto and Yukawa (2020); Yüksel et al. (2021). However, as our
results show, further performance improvements can be obtained by considering
all components of the covariance matrix. Here onwards, we will refer to the
best setting (i.e. MLM${}_{\textit{temp},\textrm{last}}$, full(cov),
Chebyshev) as the Proposed method.

###  § 4.4 Comparisons against Prior work

In this section, we compare our proposed method against related prior work. We
do not re-implement or re-run those methods, but instead compare using the
published results from the original papers.

Word2Gauss${}_{\textrm{light}}$ Iwamoto and Yukawa (2020): They apply Gaussian
Embeddings Vilnis and McCallum (2015) based architecture in each time period.
For each word, they define a computationally lightweight Gaussian embedding as
follows: the mean vector is the vector of the word2vec learned by the
initialization method Kim et al. (2014), and the covariance matrix is the
diagonal matrix, uniformly weighted by frequency. They calculate the KL
divergence of the Gaussian embeddings for the semantic variation prediction.

Word2Gauss Yüksel et al. (2021): They apply pure Gaussian Embeddings Vilnis
and McCallum (2015). For a given word, the mean vector and the covariance
matrix of the Gaussian Embedding are trained using the inner-product with the
positive examples and the KL divergence with the negative examples. For
computational convenience and to reduce the number of parameters, they use a
diagonal covariance matrix. After training separate word embedding models for
each time period, the mean vectors are aligned between time periods using a
rotation matrix Hamilton et al. (2016), and predictions are made using cosine
distance or Jeffrey’s divergence. They have reported the cosine distance as
the best metric.

MLM${}_{\textit{temp}}$ Rosin et al. (2022): They fine-tuned the published
BERT model to specific time periods. To adapt to specific time periods, they
insert a special token indicating the time period at the beginning of the
sentence in the target corpus, and fine-tuned on the corpora available for
each time period. They use two measures for prediction: (a) the distance
between the predicted probability of the target word in the sentence at each
time period, and (b) the cosine distance of the average token vector at each
time period. Their results report that the cosine distance is the best metric
(MLM${}_{\textit{temp}}$, Cosine). However, Kutuzov and Giulianelli (2020)
have shown that the average pairwise cosine distance (3) is better than the
cosine distance between average sibling embeddings. Based on this result, we
only run this setting that MLM${}_{\textit{temp}}$ model with the average
pairwise cosine distance (MLM${}_{\textit{temp}}$, APD).

MLM${}_{\textit{pre}}$ w/ Temp. Att. Rosin and Radinsky (2022): They propose a
time-specific attention mechanism to adapt MLMs to specific time periods. They
add time-specific vectors and an attention weight matrix to the published BERT
as trainable parameters and perform additional training on the target corpora.
During prediction, they use the cosine distance following Rosin et al. (2022).

MLM${}_{\textit{temp}}$ w/ Temp. Att. Rosin and Radinsky (2022): It is the
combination of the above two methods (MLM${}_{\textit{temp}}$ and
MLM${}_{\textit{pre}}$ w/ Temp. Att.), which is considered as the current SoTA
model for semantic variation prediction. They add time-specific special tokens
to the beginning of each sentence in the target corpus, and conduct additional
training on the publicly available BERT model with the time-specific attention
mechanism. They also use the cosine distance as used by Rosin et al. (2022).

Model | Spearman
---|---
Word2Gauss${}_{\textrm{light}}$ | 0.358
Word2Gauss | 0.399
MLM${}_{\textit{temp}}$, Cosine | 0.467
MLM${}_{\textit{temp}}$, APD | 0.479
MLM${}_{\textit{pre}}$ w/ Temp. Att. | 0.520
MLM${}_{\textit{temp}}$ w/ Temp. Att. | 0.548
Proposed | 0.529
Table 4: Comparison against prior work including SoTA. In our method, we
report the top three results and all of the cosine distance results. The best
performance is shown in bold, and the second best is shown in underlined.

Experimental results are summarised in Table 4. This result shows that our
proposed method achieves the second best performance compared to prior work.
We can see that the contextualised mean embeddings based method
(MLM${}_{\textit{temp}}$) outperforms the non-contextualised distribution
based methods (Word2Gauss${}_{\textrm{light}}$ and Word2Gauss), and further
improvement can be obtained by adding the time-specific attention mechanisms
(MLM${}_{\textit{pre}}$ w/ Temp. Att. and MLM${}_{\textit{temp}}$ w/ Temp.
Att.). Moreover, the contextualised distribution based approach (Proposed) can
yield performance improvement similar to adding time-specific attention
mechanisms. We will discuss the detailed analyses as follows.

#### Comparison within the base model (MLM${}_{\textit{temp}}$).

Since our method is based on MLM${}_{\textit{temp}}$, we compare performance
within MLM${}_{\textit{temp}}$. As in the previous work Rosin et al. (2022),
we discuss the results when using the cosine distance. Table 4 shows that the
average pairwise cosine distance (MLM${}_{\textit{temp}}$, APD) outperforms
the cosine distance between average sibling embeddings
(MLM${}_{\textit{temp}}$, Cosine). Moreover, from Table 2 and Table 3, we can
see that our distribution based method outperforms the previous method using
only the mean embeddings (0.467 in Table 4) in most settings (0.478 by
MLM${}_{\textit{temp},\textrm{last}}$, diag(cov), 0.480 by
MLM${}_{\textit{temp},\textrm{four}}$, diag(cov), and 0.479 by
MLM${}_{\textit{temp},\textrm{last}}$, full(cov)). This result indicates the
importance of considering not only the mean but also the variance of the
sibling embeddings.

#### Comparison against SoTA.

Although our proposed method and the SoTA MLM${}_{\textit{temp}}$ w/ Temp.
Att. are based on the same model MLM${}_{\textit{temp}}$, their configurations
are significantly different. Specifically, MLM${}_{\textit{temp}}$ w/ Temp.
Att. adds a time-specific attention mechanism to the model and learns its
parameters with additional training, whereas our proposed method uses only
MLM${}_{\textit{temp}}$ and thus does _not_ require additional parameters or
training. Although according to Table 4, MLM${}_{\textit{temp}}$ w/ Temp. Att.
reports a correlation of 0.548 and marginally outperforms the Proposed method,
which obtains a correlation of 0.529, we find no statistically significant
difference between those two methods.555To measure the statistical
significance, we use the Fisher transformation Fisher (1992).

###  § 4.5 Ablation Study

We conduct an ablation study to understand the importance of (i) predicting
semantic variation with sibling distributions
$\mathcal{N}(\boldsymbol{\mu}^{w}_{i},\text{\bf V}^{w}_{i})$, and (ii)
constructing sibling distributions from the mean $\boldsymbol{\mu}^{w}_{i}$
and covariance $\text{\bf V}^{w}_{i}$ of sibling embeddings. Based on our best
setting Proposed (MLM${}_{\textit{temp},\textrm{last}}$, full(cov),
Chebyshev), we define two variants: (i) predicting semantic variation score
using mean vectors $\boldsymbol{\mu}^{w}_{1}$ and $\boldsymbol{\mu}^{w}_{2}$
only as previous studies, and (ii) constructing a sibling distribution with
the identity matrix $\mathcal{N}(\boldsymbol{\mu}^{w}_{i},\text{\bf I})$
instead of the covariance matrix $\text{\bf V}^{w}_{i}$. In the SemEval-2020
Task 1 English evaluation set, the existence of a semantic change (binary
judgement) and its degree (continuous judgement) are provided. Therefore, due
to the limited space, we analyse the top eight semantically changed words with
the highest degrees of semantic changes and the bottom eight semantically
stable words with the lowest degrees of semantic change.

From Table 5, we see that our distribution-based variants ($\text{\bf
V}^{w}_{i}=\text{\bf I}$ and Proposed) eliminate overestimation or
underestimation problems in using mean vectors only (w/o $\text{\bf
V}^{w}_{i}$). The variant w/o $\text{\bf V}^{w}_{i}$ correctly detects words
plane and graft that have changed meaning significantly between time periods.
However, this variant also reports underestimation (stab and bit) and
overestimation (contemplation and chairman) in other words, whose meanings are
changed/stable but the mean vectors are changed little/significantly. This is
because it makes predictions based only on the mean of sibling embeddings. On
the other side, the distribution-based variants ($\text{\bf
V}^{w}_{i}=\text{\bf I}$ and Proposed) can appropriately rank semantically
changed words ($\Delta=\textrm{\checkmark}$) that have small changes in mean
vectors (stab and bit), and stable words ($\Delta=\textrm{\char 55}$) that
have large changes in mean vectors (contemplation and chairman).666The
distribution-based methods fail to detect highly ambiguous words with distinct
word senses (plane and graft). However, the proposed method approximates the
distribution of embeddings for a word using a “single” Gaussian. We believe by
using a mixture of Gaussian this issue can be resolved. Moreover, we find that
even with the distribution-based variants, using covariance matrices
$\text{\bf V}^{w}_{i}$ computed from sibling embeddings yields even better
performance than identity matrices ($\text{\bf V}^{w}_{i}=\text{\bf I}$). This
result further verifies our hypothesis that considering the mean and the
variance of the sibling embeddings is important for semantic change detection
tasks.

Word | Gold |  w/o $\text{\bf V}_{i}^{w}$ |  $\text{\bf V}_{i}^{w}=\text{\bf I}$ |  Proposed
---|---|---|---|---
rank |  $\Delta$ | rank | rank | rank
plane |  1 |  ✓ |  3 |  18 |  15
tip |  2 |  ✓ |  7 |  9 |  7
prop |  3 |  ✓ |  16 |  1 |  4
graft |  4 |  ✓ |  2 |  36 |  36
record |  5 |  ✓ |  15 |  12 |  14
stab |  7 |  ✓ |  31 |  10 |  11
bit |  9 |  ✓ |  27 |  11 |  9
head |  10 |  ✓ |  23 |  28 |  28
multitude |  30 |  ✗ |  24 |  35 |  35
savage |  31 |  ✗ |  20 |  26 |  26
contemplation |  32 |  ✗ |  1 |  37 |  37
tree |  33 |  ✗ |  33 |  31 |  30
relationship |  34 |  ✗ |  26 |  34 |  34
fiction |  35 |  ✗ |  21 |  29 |  29
chairman |  36 |  ✗ |  5 |  32 |  33
risk |  37 |  ✗ |  10 |  19 |  21
Spearman | 1.000 |  0.070 |  0.503 |  0.529
Table 5: Ablation study on the top-8 semantically changed ($\Delta=$ ✓) words
with the highest degree of semantic change and the bottom-8 stable words
($\Delta=$ ✗) with the lowest degree of semantic change. w/o $\text{\bf
V}_{i}^{w}$ predicts using mean vectors $\boldsymbol{\mu}_{1}^{w}$ and
$\boldsymbol{\mu}_{2}^{w}$ directly, whereas $\text{\bf V}_{i}^{w}=\text{\bf
I}$ samples sibling embeddings from a Gaussian with the identity variance
(i.e. $\mathcal{N}(\boldsymbol{\mu}_{i}^{w},\text{\bf I})$) instead of
$\mathcal{N}(\boldsymbol{\mu}_{i}^{w},\text{\bf V}_{i}^{w})$.

##  § 5 Conclusion

We proposed a method to detect semantic variations of words using sibling
embeddings. Experimental results on SemEval-2020 Task 1 English dataset show
that the proposed method consistently outperforms methods that use only the
mean embedding vectors, and reports results comparable to the current SoTA.
Furthermore, a qualitative analysis shows that the proposed method correctly
detects semantic variation of words, which are either over/underestimated by
the existing methods.

##  § 6 Limitations

#### Language-related limitations.

For the ease of the analysis, we conducted experiments using only the English
dataset in this study. Although our proposed method can be applied to any
language, its performance must be evaluated on languages other than English.
For example, the SemEval-2020 Task 1 dataset includes Latin, German, and
Swedish language datasets, in addition to English, and can be used for this
purpose. In particular, our proposed method requires only pretrained MLMs and
does not require additional training data for the target languages, which
makes it easily scalable to many languages.

#### Availability of MLMs for the target language.

Experimental results show that the quality of the MLM is an important factor
determining the performance of the proposed method. For example, the proposed
method reports good performance with vanilla BERT model in Table 2 but further
gains in performance can be obtained with the fine-tuned BERT model on masked
time stamps. However, since our method assumes the availability of pretrained
MLMs, a problem arises when trying to adapt our method to minor languages
where no pretrained MLMs are available. This limitation could be mitigated to
an extent by using multilingual MLMs. For example, Arefyev and Zhikov (2020)
demonstrated that satisfactory levels of accuracies can be obtained for
semantic change detection by using multilingual MLMs. Our proposed method can
further benefit from the fact that new and larger MLMs are being publicly
released for many languages in the NLP community.

##  § 7 Ethical Considerations

In this paper, we proposed a distribution based method using publicly
available MLMs, and evaluated with the SemEval-2020 Task 1 English data.
Although we have not published any datasets or models, Basta et al. (2019)
shows that pretrained MLMs encode and even amplify unfair social biases such
as gender or racial biases. Given that we obtain sibling distributions from
such potentially socially biased MLMs, we must further evaluate the
sensitivity of our method for such undesirable social biases.

## Acknowledgements

This work was supported by JST, the establishment of university fellowships
towards the creation of science technology innovation, Grant Number
JPMJFS2139. Danushka Bollegala holds concurrent appointments as a Professor at
University of Liverpool and as an Amazon Scholar. This paper describes work
performed at the University of Liverpool and is not associated with Amazon.

## References

  * Aida et al. (2021) Taichi Aida, Mamoru Komachi, Toshinobu Ogiso, Hiroya Takamura, and Daichi Mochihashi. 2021.  A comprehensive analysis of PMI-based models for measuring semantic differences.  In _Proceedings of the 35th Pacific Asia Conference on Language, Information and Computation_ , pages 21–31, Shanghai, China. Association for Computational Lingustics.
  * Alatrash et al. (2020) Reem Alatrash, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2020\.  CCOHA: Clean corpus of historical American English.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 6958–6966, Marseille, France. European Language Resources Association.
  * Arefyev and Zhikov (2020) Nikolay Arefyev and Vasily Zhikov. 2020.  BOS at SemEval-2020 task 1: Word sense induction via lexical substitution for lexical semantic change detection.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 171–179, Barcelona (online). International Committee for Computational Linguistics.
  * Basile et al. (2020) Pierpaolo Basile, Annalina Caputo, Tommaso Caselli, Pierluigi Cassotti, and Rossella Varvara. 2020.  Diacr-ita @ evalita2020: Overview of the evalita2020 diachronic lexical semantics (diacr-ita) task.  In _Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020)_. CEUR Workshop Proceedings (CEUR-WS.org).  Evaluation Campaign of Natural Language Processing and Speech Tools for Italian, EVALITA 2020 ; Conference date: 17-12-2020.
  * Basta et al. (2019) Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019.  Evaluating the underlying gender bias in contextualized word embeddings.  In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , pages 33–39, Florence, Italy. Association for Computational Linguistics.
  * Beck (2020) Christin Beck. 2020.  DiaSense at SemEval-2020 task 1: Modeling sense change via pre-trained BERT embeddings.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 50–58, Barcelona (online). International Committee for Computational Linguistics.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramér, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021.  On the Opportunities and Risks of Foundation Models.
  * Cook and Stevenson (2010) Paul Cook and Suzanne Stevenson. 2010.  Automatically identifying changes in the semantic orientation of words.  In _Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)_ , Valletta, Malta. European Language Resources Association (ELRA).
  * Dubossarsky et al. (2019) Haim Dubossarsky, Simon Hengchen, Nina Tahmasebi, and Dominik Schlechtweg. 2019\.  Time-out: Temporal referencing for robust modeling of lexical semantic change.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 457–470, Florence, Italy. Association for Computational Linguistics.
  * Ethayarajh (2019) Kawin Ethayarajh. 2019.  How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 55–65, Hong Kong, China. Association for Computational Linguistics.
  * Firth (1957) John R. Firth. 1957.  A synopsis of linguistic theory 1930-55.  _Studies in Linguistic Analysis_ , pages 1 – 32.
  * Fisher (1992) R. A. Fisher. 1992.  _Statistical Methods for Research Workers_, pages 66–70. Springer New York, New York, NY.
  * Giulianelli et al. (2020) Mario Giulianelli, Marco Del Tredici, and Raquel Fernández. 2020.  Analysing lexical semantic change with contextualised word representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3960–3973, Online. Association for Computational Linguistics.
  * Hamilton et al. (2016) William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016.  Diachronic word embeddings reveal statistical laws of semantic change.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1489–1501, Berlin, Germany. Association for Computational Linguistics.
  * Hu et al. (2019) Renfen Hu, Shen Li, and Shichen Liang. 2019.  Diachronic sense modeling with deep contextualized word embeddings: An ecological view.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3899–3908, Florence, Italy. Association for Computational Linguistics.
  * Iwamoto and Yukawa (2020) Ran Iwamoto and Masahiro Yukawa. 2020.  RIJP at SemEval-2020 task 1: Gaussian-based embeddings for semantic change detection.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 98–104, Barcelona (online). International Committee for Computational Linguistics.
  * Jaynes (2003) E. T. Jaynes. 2003.  _Probability Theory_.  Cambridge University Press.
  * Kim et al. (2014) Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014.  Temporal analysis of language through neural language models.  In _Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science_ , pages 61–65, Baltimore, MD, USA. Association for Computational Linguistics.
  * Kulkarni et al. (2015) Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015.  Statistically significant detection of linguistic change.  In _WWW 2015_ , pages 625–635.
  * Kutuzov et al. (2022) Andrei Kutuzov, Erik Velldal, and Lilja Ovrelid. 2022.  Contextualized embeddings for semantic change detection: Lessons learned.  _Northern European Journal of Language Technology_ , 8.
  * Kutuzov and Giulianelli (2020) Andrey Kutuzov and Mario Giulianelli. 2020.  UiO-UvA at SemEval-2020 task 1: Contextualised embeddings for lexical semantic change detection.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 126–134, Barcelona (online). International Committee for Computational Linguistics.
  * Kutuzov et al. (2018) Andrey Kutuzov, Lilja Ovrelid, Terrence Szymanski, and Erik Velldal. 2018.  Diachronic word embeddings and semantic shifts: a survey.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1384–1397, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Kutuzov and Pivovarova (2021) Andrey Kutuzov and Lidia Pivovarova. 2021.  RuShiftEval: a shared task on semantic shift detection for Russian.  In _Computational linguistics and intellectual technologies: Papers from the annual conference Dialogue_.
  * Laicher et al. (2021) Severin Laicher, Sinan Kurtyigit, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021.  Explaining and improving BERT performance on lexical semantic change detection.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop_ , pages 192–202, Online. Association for Computational Linguistics.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomáš Kočiský, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021.  Mind the gap: Assessing temporal generalization in neural language models.  In _Advances in Neural Information Processing Systems_.
  * Loureiro et al. (2022) Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-collados. 2022.  TimeLMs: Diachronic language models from Twitter.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 251–260, Dublin, Ireland. Association for Computational Linguistics.
  * Martinc et al. (2020) Matej Martinc, Petra Kralj Novak, and Senja Pollak. 2020.  Leveraging contextual embeddings for detecting diachronic semantic shift.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4811–4819, Marseille, France. European Language Resources Association.
  * Michel et al. (2011) Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K. Gray, null null, Joseph P. Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, Steven Pinker, Martin A. Nowak, and Erez Lieberman Aiden. 2011.  Quantitative analysis of culture using millions of digitized books.  _Science_ , 331(6014):176–182.
  * Montariol et al. (2021) Syrielle Montariol, Matej Martinc, and Lidia Pivovarova. 2021.  Scalable and interpretable semantic change detection.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4642–4652, Online. Association for Computational Linguistics.
  * Pimentel et al. (2020) Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020.  Information-theoretic probing for linguistic structure.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4609–4622, Online. Association for Computational Linguistics.
  * Rosin et al. (2022) Guy D. Rosin, Ido Guy, and Kira Radinsky. 2022.  Time masking for temporal language models.  In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ , WSDM ’22, pages 833–841, New York, NY, USA. Association for Computing Machinery.
  * Rosin and Radinsky (2022) Guy D. Rosin and Kira Radinsky. 2022.  Temporal attention for language models.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1498–1508, Seattle, United States. Association for Computational Linguistics.
  * Schlechtweg et al. (2020) Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020.  SemEval-2020 task 1: Unsupervised lexical semantic change detection.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 1–23, Barcelona (online). International Committee for Computational Linguistics.
  * Su et al. (2022) Zhaochen Su, Zecheng Tang, Xinyan Guan, Lijun Wu, Min Zhang, and Juntao Li. 2022\.  Improving temporal generalization of pre-trained language models with lexical semantic change.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6380–6393, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Tahmasebi et al. (2021) Nina Tahmasebi, Lars Borina, and Adam Jatowtb. 2021.  Survey of computational approaches to lexical semantic change detection.  _Computational approaches to semantic change_ , 6:1.
  * Traugott and Dasher (2001) Elizabeth Closs Traugott and Richard B. Dasher. 2001.  _Prior and current work on semantic change_, Cambridge Studies in Linguistics, page 51–104. Cambridge University Press.
  * Vilnis and McCallum (2015) Luke Vilnis and Andrew McCallum. 2015.  Word representations via gaussian embedding.  In _Proceedings of the 3rd International Conference on Learning Representations_ , San Diego, CA, USA.
  * Yao et al. (2018) Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, and Hui Xiong. 2018.  Dynamic word embeddings for evolving semantic discovery.  In _WSDM 2018_ , page 673–681.
  * Yüksel et al. (2021) Arda Yüksel, Berke Uğurlu, and Aykut Koç. 2021.  Semantic change detection with gaussian word embeddings.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3349–3361.
  * Zhou et al. (2022) Kaitlyn Zhou, Kawin Ethayarajh, Dallas Card, and Dan Jurafsky. 2022.  Problems with cosine as a measure of embedding similarity for high frequency words.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 401–423, Dublin, Ireland. Association for Computational Linguistics.
  * Zhou and Bollegala (2021) Yi Zhou and Danushka Bollegala. 2021.  Learning sense-specific static embeddings using contextualised word embeddings as a proxy.  In _Proceedings of the 35th Pacific Asia Conference on Language, Information and Computation_ , pages 493–502, Shanghai, China. Association for Computational Lingustics.
  * Zhou and Bollegala (2022) Yi Zhou and Danushka Bollegala. 2022.  On the curious case of l2 norm of sense embeddings.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 2593–2602, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

##  Appendix A Information of Sibling Distribution

In the semantic variation prediction, prior work have applied the mean
embeddings $\boldsymbol{\mu}^{w}_{i}$ of sibling distribution
$\mathcal{D}(w,C_{i})$ for each word $w$. However, since these methods
compress multiple vectors of $\mathcal{D}(w,C_{i})$ into a single vector
$\boldsymbol{\mu}^{w}_{i}$, there is a risk of loosing the information
contained in each vector Pimentel et al. (2020). To discuss the amount of
information a sibling distribution holds, we analyse the relationship between
the size of a sibling distribution $\mathcal{D}(w,C_{i})$ (word frequency
$N^{w}_{i}$) and the rank of a covariance matrix $\text{\bf V}^{w}_{i}$
calculated from $\mathcal{D}(w,C_{i})$.

Figure 2 shows the relationship between the frequency of randomly sampled
1,000 words and the rank of their covariance matrices. For each word, we
construct a covariance matrix from sibling embeddings as in (2). These
matrices have $d\times d$ dimensions (BERT base models have $d=768$ hidden
size), and we use their full components (full(cov)) for computing their ranks.
We see that there is a strong correlation between the frequency and the rank
of the covariance matrix, and when the frequency exceeds the dimension size,
the rank remains constant at the dimensionality of the contextualised
embedding space. This result implies that, upto the dimensionality of the
contextualised embedding space, the covariance matrix computed from the
sibling distribution $\mathcal{D}(w,C_{i})$, retains information about the
individual occurrences of a word. Given that contextualised embeddings are
often high dimensional (e.g. 768, 1024 etc.) the covariance matrix $\text{\bf
V}^{w}_{i}$ computed from the sibling distribution $\mathcal{D}(w,C_{i})$
preserves sufficient information about $w$ for semantic variations related to
$w$.

In this analysis, we show that an interesting trend of the word frequency and
the rank of covariance matrix. We speculate that this result may be related to
the trend of the sense frequency and the length of sense representation
reported in the previous study Zhou and Bollegala (2022). However, we leave
the investigation of this interesting trend to future research.

Figure 2: The relationship between the frequency and the rank of the
covariance matrix of randomly sampled 1,000 words.

##  Appendix B List of Divergence Measures

We describe the divergence measures as detailed next. For simplicity, we
denote two Gaussian distributions
$\mathcal{N}(\boldsymbol{\mu}^{w}_{1},\text{\bf V}^{w}_{1})$ and
$\mathcal{N}(\boldsymbol{\mu}^{w}_{2},\text{\bf V}^{w}_{2})$ as
$\mathcal{N}^{w}_{1}$ and $\mathcal{N}^{w}_{2}$, respectively.

Kullback-Liebler

$\begin{split}&\mathrm{KL}(\mathcal{N}^{w}_{1}||\mathcal{N}^{w}_{2})\\\
=&\frac{1}{2}\Bigl{(}\mathrm{tr}({\text{\bf V}^{w}_{2}}^{-1}\text{\bf
V}^{w}_{1})-d-\log\frac{\mathrm{det}(\text{\bf
V}^{w}_{1})}{\mathrm{det}(\text{\bf V}^{w}_{2})}\\\ \ \ \ \
&+(\boldsymbol{\mu}^{w}_{2}-\boldsymbol{\mu}^{w}_{1}){}^{\top}{\text{\bf
V}^{w}_{2}}^{-1}(\boldsymbol{\mu}^{w}_{2}-\boldsymbol{\mu}^{w}_{1})\Bigr{)}\end{split}$
(4)

Jeffrey’s

$\begin{split}&\mathrm{Jeff}(\mathcal{N}^{w}_{1}||\mathcal{N}^{w}_{2})\\\
=&\frac{1}{2}\mathrm{KL}(\mathcal{N}^{w}_{1}||\mathcal{N}^{w}_{2})+\frac{1}{2}\mathrm{KL}(\mathcal{N}^{w}_{2}||\mathcal{N}^{w}_{1})\\\
=&\frac{1}{4}\Bigl{(}\mathrm{tr}({\text{\bf V}^{w}_{2}}^{-1}\text{\bf
V}^{w}_{1})+\mathrm{tr}({\text{\bf V}^{w}_{1}}^{-1}\text{\bf V}^{w}_{2})-2d\\\
\ \ \ \
&+(\boldsymbol{\mu}^{w}_{2}-\boldsymbol{\mu}^{w}_{1}){}^{\top}{\text{\bf
V}^{w}_{2}}^{-1}(\boldsymbol{\mu}^{w}_{2}-\boldsymbol{\mu}^{w}_{1})\\\ \ \ \ \
&+(\boldsymbol{\mu}^{w}_{1}-\boldsymbol{\mu}^{w}_{2}){}^{\top}{\text{\bf
V}^{w}_{1}}^{-1}(\boldsymbol{\mu}^{w}_{1}-\boldsymbol{\mu}^{w}_{2})\Bigr{)}\end{split}$
(5)

##  Appendix C List of Distance Measures

We describe the distance measures as detailed next. $\boldsymbol{w}(i)$
denotes the $i$-th value of a word vector $\boldsymbol{w}$ and
$\overline{\boldsymbol{w}}$ denotes a subtracted vector from the average of
all dimension values.

Bray-Curtis

$\psi(\boldsymbol{w}_{1},\boldsymbol{w}_{2})=\frac{\sum_{i\in
d}|\boldsymbol{w}_{1}(i)-\boldsymbol{w}_{2}(i)|}{\sum_{i\in
d}|\boldsymbol{w}_{1}(i)+\boldsymbol{w}_{2}(i)|}$ (6)

Canberra

$\psi(\boldsymbol{w}_{1},\boldsymbol{w}_{2})=\sum_{i\in
d}\frac{|\boldsymbol{w}_{1}(i)-\boldsymbol{w}_{2}(i)|}{|\boldsymbol{w}_{1}(i)|+|\boldsymbol{w}_{2}(i)|}$
(7)

Chebyshev

$\psi(\boldsymbol{w}_{1},\boldsymbol{w}_{2})=\max_{i}|\boldsymbol{w}_{1}(i)-\boldsymbol{w}_{2}(i)|$
(8)

City Block

$\psi(\boldsymbol{w}_{1},\boldsymbol{w}_{2})=\sum_{i\in
d}|\boldsymbol{w}_{1}(i)-\boldsymbol{w}_{2}(i)|$ (9)

Correlation

$\psi(\boldsymbol{w}_{1},\boldsymbol{w}_{2})=1-\frac{\overline{\boldsymbol{w}}_{1}\cdot\overline{\boldsymbol{w}}_{2}}{||\overline{\boldsymbol{w}}_{1}||_{2}\
||\overline{\boldsymbol{w}}_{2}||_{2}}$ (10)

Cosine

$\psi(\boldsymbol{w}_{1},\boldsymbol{w}_{2})=1-\frac{\boldsymbol{w}_{1}\cdot\boldsymbol{w}_{2}}{||\boldsymbol{w}_{1}||_{2}\
||\boldsymbol{w}_{2}||_{2}}$ (11)

Euclidean

$\psi(\boldsymbol{w}_{1},\boldsymbol{w}_{2})=||\boldsymbol{w}_{1}-\boldsymbol{w}_{2}||_{2}$
(12)


# Discovering Novel Customer Features with Recurrent Neural Networks for
Personality Based Financial Services

Charl Maree & Christian W. Omlin
Center for Artificial Intelligence Research
University of Agder
Grimstad, Norway
<EMAIL_ADDRESS>
Second affiliation: Strategy, Insight, and Development, Sparebank 1 SR-Bank,
Stavanger, Norway.

###### Abstract

The micro-segmentation of customers in the finance sector is a non-trivial
task and has been an atypical omission from recent scientific literature.
Where traditional segmentation classifies customers based on coarse features
such as demographics, micro-segmentation depicts more nuanced differences
between individuals, bringing forth several advantages including the potential
for improved personalization in financial services. AI and representation
learning offer a unique opportunity to solve the problem of micro-
segmentation. Although ubiquitous in many industries, the proliferation of AI
in sensitive industries such as finance has become contingent on the
imperatives of responsible AI. We had previously solved the micro-segmentation
problem by extracting temporal features from the state space of a recurrent
neural network (RNN). However, due to the inherent opacity of RNNs our
solution lacked an explanation - one of the imperatives of responsible AI. In
this study, we address this issue by extracting an explanation for and
providing an interpretation of our temporal features. We investigate the state
space of our RNN and through a linear regression model reconstruct the
trajectories in the state space with high fidelity. We show that our linear
regression coefficients have not only learned the rules used to create the
RNN’s output data but have also learned the relationships that were not
directly evident in the raw data.

##  1 Introduction

Customer segmentation is an important field in banking and with customer bases
growing banks are having to employ ever more advanced methods to maintain, if
not improve, levels of personalization [1]. Customer segmentation has
typically been achieved using demographics such as age, gender, location, etc.
[2]. However, these features not only produce coarse segments, but also
introduce the potential for discrimination, e.g., when using postal codes for
credit rating [3]. In contrast, micro-segmentation provides a more
sophisticated, fine-grained classification that depicts nuanced differences
between individuals, improves personalization, and promotes fairness. Despite
these advantages and the fact that the need for such fine-grained segmentation
has been highlighted [4], the scientific community has been surprisingly quiet
on the topic with only a few recent publications from, e.g., the health sector
[5, 6] and none from the finance sector.

Artificial intelligence is fast becoming ubiquitous across multiple industries
with representation learning an auspicious method for customer micro-
segmentation [7]. Sensitive industries such as finance face legal and moral
obligations towards the responsible implementation of AI [8]. The European
Commission has published several guidelines surrounding responsible AI and
scientific fundamentals have been consolidated in recent surveys on the topic
[9, 10]. Explainability and interpretability are key elements in responsible
AI [11], which are generally not yet adequately addressed in applications of
AI in finance [12]. Our perspective on explainability in AI refers to a
symbolic representation of a model, whereas interpretability refers to a human
understanding of and reasoning about the functionality of the model.
Explainability therefore neither guarantees nor implies interpretability. In
this study, we address both the issues of explainability and interpretability.
Our aim is to extract and facilitate the use of salient features in future
financial services; we have already shown the potential in predicting default
rate and customer liquidity indices [7]. Our ultimate goal is the development
of personalized financial services in which responsible customer micro-
segmentation is key.

##  2 Related Work

###  2.1 Representation Learning using Recurrent Neural Networks

In [13], the authors developed a model for predicting spending personality
from aggregated financial transactions with the intent to investigate the
causality between personality-aligned spending and happiness. They rated each
of 59 spending categories according to its association with the Big-Five
personality traits - extraversion, neuroticism, openness, conscientiousness,
and agreeableness - which resulted in a set of $59\times 5$ linear
coefficients. We used these coefficients in a previous study to train a
recurrent neural network (RNN) to predict customers’ personality traits from
their aggregated transactions [7]. We showed that the temporal features in the
state space of the RNN had interesting properties: they formed smooth
trajectories which formed hierarchical clusters along successive levels of
dominance of the personality traits. The dominant personalty trait is the one
with the largest coefficient in the Big-Five model of personality traits [7].
We also showed that similarly salient features could not be extracted from the
raw data otherwise. Spending patterns over time are either more consistent
than transactions aggregated over a short time period, they may fluctuate, or
they may change based on life circumstances. Modelling spending over time
reinforces spending patterns and thus may lead to better features.
Fluctuations or changes are also better represented by time series. The
hierarchical clustering of the extracted features provided a means of micro-
segmenting customers based on their financial behaviour. However, the
responsible employment of this model demands an explanation and
interpretation, which is what we address in this study. Other studies have
employed representation learning using RNNs to encode spatial and temporal
information contained in the two-dimensional trajectories of physical objects
[14], as well as for a wide variety of applications in speech and video data
[15].

###  2.2 Explaining Recurrent Neural Networks

Finding symbolic representations of AI models is a key challenge in
responsible AI. In [16] the authors developed a symbolic regression algorithm
that successfully extracted physics equations from neural networks. They
managed to extract all 100 of the equations from the well known Feynman
Lectures on Physics and 90% of more complicated equations, an improvement from
15% using state of the art software. This was an important study because it
not only proved that deep neural networks are capable of learning complicated
equations and coefficients, but that it is possible to extract symbolic
knowledge from such networks. The authors in [17] presented a visual method to
explain RNNs used in natural language processing problems. They clustered the
activations in the state space and used word clouds to visualize correlations
between node activations and words in the input sentences. Similarly, the
authors in [18] applied clustering in the state space of RNNs, but here the
authors showed that _symbolic_ representations could be extracted as opposed
to visual explanations. Studies such as these prove that deep neural networks
are indeed not inexplicable black box systems, but could be a means of
discovering symbolic representations of complex relationships in data.

##  3 Experimental Methodology

Our input data for training our RNN were aggregated financial transactions of
approximately 26,000 customers, while the output data were their personality
traits [13]. The transactions were aggregated annually across 97 transaction
classes, such as groceries, transport, leisure, etc., over a period of six
years. The personality traits were based on the Big-Five personality model and
were calculated using a set of $59\times 5$ published coefficients linking
transaction classes to personality traits [19, 13]. Our RNN consisted of three
long short-term memory (LSTM) nodes [20]. After training and during
prediction, we inspected the activations of the three hidden nodes for each of
the six time-steps; each customer was represented by a trajectory with six
data-points in a three-dimensional space, where each dimension represented the
activation of one LSTM node. These trajectories were our extracted features
which may be used for micro-segmentation of customers. To provide an
explanation for the model, we trained a linear regression model - an
inherently transparent class of models [10] \- to replicate the trajectories
from each customer’s aggregated spending distribution. Further inspection of
the trajectories’ behaviour in relation to the associated customers’ fuzzy
grades of membership in the personality traits provided an interpretation of
the extracted features. The results are discussed in the following section.

##  4 Results

In Figure 1, we show the features that we extracted from our RNN. Figure 1(a)
illustrates the clustering behaviour of the trajectories in the state space.
Our empirical observations led us to hypothesise the existence of attractors
for each of the five personality traits. Figure 1(b) shows two trajectories
for the same customer where the inputs to the RNN were aggregated over two
different time periods: one year and six years. The fact that there is little
difference between these two trajectories is significant; it demonstrates that
the duration of the time window does not affect customer classification. This
was not the case when clustering the raw personality data, where customers
frequently moved between different clusters for different time periods due to
variations in spending with changing life circumstances. Although we did
observe significant course changes for some customers’ trajectories, the vast
majority of customers remained in their assigned clusters for the six-year
period. This stability in customer micro-segmentation is key for personalized
financial services, as financial advice has to be consistent.

(a)

(b)

Figure 1:  Trajectories in the 3-dimensional state space of a recurrent neural
network trained to predict personality from aggregated transactions. While (a)
shows the clustering of the trajectories of many customers according to the
most dominant personality trait, (b) shows two trajectories for the same
customer being identically classified over two different time periods: one
year vs. six years.

To explain our model, we fit a linear regression model to reproduce the
trajectories from the RNN’s input data: the 97 aggregated transaction classes,
or spending patterns. From our observations in Figure 1, we hypothesized that
the lengths of the trajectories were not as important as their directions. We
therefore simplified the trajectory vectors and represented them by the two
angles which fully describe their directions in three-dimensional space. These
angles were the outputs of our linear regression model, which fit the data
with a coefficient of determination ($R^{2}$) of 0.78 for an unseen test set,
while a more complicated polynomial regression model managed an only slightly
better 0.79. Other methods such as ridge regression and decision tree
regression could not perform better. Our 97 transaction classes mostly
overlapped with those of the $59\times 5$ published coefficients and due to
aggregations such as ”health and fitness” being expanded to ”health” and
”fitness”, there were $61\times 5$ non-zero coefficients for calculating our
customers’ personality traits. The linear regression model had $69\times 2$
non-zero 111Non-zero here refers to coefficients with values that are not
insignificantly small compared to the mean value of all the coefficients.
coefficients with a strong correlation to the original non-zero coefficients.
Due to this and the relatively high coefficient of determination, we conclude
that the linear regression model matched the RNN with high fidelity.

Within each of the clusters in Figure 1, we observed hierarchical sub-clusters
along the second, third, and fourth most dominant personality traits. This
hierarchical sub-clustering is important because it provides a means of micro-
segmenting customers which was not present in the raw data and could neither
be replicated using feed-forward neural networks nor auto-encoders. Using our
linear regression model, we created a two-dimensional plot of trajectory
angles (Figure 2). In this figure, we illustrate the hierarchical clustering
behaviour that we observed for the trajectories from the RNN, where (a) shows
the clustering along the customers’ most dominant personality trait and (b)
through (d) show the hierarchy of sub-clusters within the parent clusters.
These clusters, like the trajectory clusters, were consistent in time, i.e.,
the new features retained the desirable properties of the features from the
state space of our RNN.

(a)

(b)

(c)

(d)

Figure 2:  Hierarchical clustering for trajectory angles in 2-dimensional
space. Each axis represents an angle (in radians) which describes the
direction of the trajectories in 3-dimensional space and each data point
represents a trajectory. We show all the levels of hierarchical clustering:
(a) shows the highest level, while (b) through (d) show sub-clustering within
each of the subsequent parent clusters.

The interpretation of the features in Figure 2 is that they mark the locations
where the trajectories would penetrate the inside of a sphere in the state
space of the RNN. We observed that the directions of the trajectories were
consistent with the grades of the customers’ fuzzy memberships in each of the
five personality traits, i.e., the output data of the RNN. The greater a
customer’s fuzzy membership in the dominant personality trait, the quicker the
trajectories converged towards the corresponding hypothesised attractor. The
five attractors acted not only on the dominant personality trait, but also on
succeedingly lesser personality traits with succeedingly lesser forces. We
demonstrate this in Figure 2 where the sub-clusters preserve the structure of
their parent clusters: trajectories of lesser personality traits also
converged to their respective attractors. Intuitively, people spend
differently according to their dominant personality trait. Within a group of
their peers, their lesser personality traits still differentiate them from
each other. Thus the hierarchical clustering of trajectories and their and
labeling is the model interpretation.

Finally, Figure 3 shows the long-term (six years) and short-term (one year)
trajectories of a single customer who changed their spending behaviour such
that their dominant personality type changed in the last year. In this figure
it is clear that, for the final year, both trajectories moved towards the same
attractor (conscientiousness), with the neuroticism attractor no longer acting
upon the long-term trajectory. We do not yet know the locations of the
hypothesised attractors and though it is not required for an interpretation we
nevertheless intend to locate them in future work by following methods
described in, e.g., [21, 22, 23].

Figure 3: Two trajectories - long-term and short-term - for the same customer
that seem to converge to a common attractor in the last year. In the long-term
trajectory, the customer’s dominant personalty trait is neuroticism as derived
from annual aggregated spending patterns for the first five years. In the
sixth year, the customer changed their spending pattern to conscientiousness,
which is reflected in the directional change of the trajectory towards the
conscientiousness attractor. The short-term trajectory corresponds to that
same customers’ spending behaviour derived from bi-monthly aggregated spending
for the last year.

##  5 Conclusions and Directions for Future Work

The financial sector is experiencing an increased demand in the level of
personalization offered to its customers, which requires more nuanced
segmentation techniques than the current offerings from traditional features
such as demographics. Representation learning offers such an alternative
technique for fine-grained segmentation, but it is plagued by the inherent
opacity introduced by deep learning; explainability and interpretability are
key in sensitive industries such as finance which must comply with regulations
regarding the responsible use of AI. We proposed a solution by extracting
temporal features from the state space of a RNN [7]. Our extracted features
formed trajectories which were associated with the Big-Five personality
traits. The dominant personality trajectories clustered and within each such
cluster, we found a hierarchy of sub-clusters which corresponded to
successively lesser personality traits. The clusters of feature trajectories
corresponding to the dominant personalities provide a coarse customer
segmentation, while the hierarchy of trajectories associated with lesser
personality traits offers the opportunity for micro-segmentation. We further
observed the convergence of trajectories in the state space and we
hypothesised the existence of attractors for each of the five personality
traits. Finally, we provided an explanation for the trajectories through a
high fidelity linear regression model which answers questions such as ”why was
Customer A classified in this way” by referring to their historic financial
transactions. In future work, we intend to find the locations of the
attractors in the state space which govern the directions of the customer
trajectories, and to use our explainable features in the development of
personal financial services such as personalized savings advice, advanced
product recommendations, and wealth forecasters.

#### Acknowledgments

This work was partially funded by The Norwegian Research Foundation, project
number 311465. We are grateful for fruitful discussions with Joe Gladstone on
the topic of personality traits and the determination of their corresponding
coefficients.

## References

  * [1] Matteo Stefanel and Udayan Goyal.  Artificial intelligence & financial services: Cutting through the noise.  Technical report, APIS partners, London, England, 2019.
  * [2] Prateek Kalia.  Product category vs demographics: Comparison of past and future purchase intentions of e-shoppers.  International Journal of E-Adoption (IJEA), 10(2):20–37, 2018.
  * [3] Solon Barocas and Andrew D. Selbst.  Big data’s disparate impact.  California Law Review, 104(671):671–732, 2016.
  * [4] Raghu Krishnapuram and Anirban Mondal.  Upcoming research challenges in the financial services industry: a technology perspective.  IDRBT Journal of Banking Technology, 1(1):66–84, 2017.
  * [5] Kenji Kuwayama, Hajime Miyaguchi, Yuko T. Iwata, Tatsuyuki Kanamori, Kenji Tsujikawa, Tadashi Yamamuro, Hiroki Segawa, and Hiroyuki Inoue.  Strong evidence of drug-facilitated crimes by hair analysis using lc–ms/ms after micro-segmentation.  Forensic Toxicology, 37(1):480–487, 2019.
  * [6] E.Y.L Nandapala, K.P.N Jayasena, and R.M.K.T Rathnayaka.  Behavior segmentation based micro-segmentation approach for health insurance industry.  2nd International Conference on Advancements in Computing (ICAC), 1(1):333–338, 2020.
  * [7] Charl Maree and Christian W. Omlin.  Clustering in recurrent neural networks for micro-segmentation using spending personality.  arXiv, 2109.09425, 2021.
  * [8] Joost van der Burgt.  General principles for the use of artificial intelligence in the financial sector.  Technical report, De Nederlandsche Bank, Amsterdam, The Netherlands, 2019\.
  * [9] European-Commission.  On artificial intelligence - a european approach to excellence and trust (whitepaper).  Technical report, European Commission, Brussels, Belgium, 2020.
  * [10] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  Information Fusion, 58(1):82–115, 2020.
  * [11] Bryce Goodman and Seth Flaxman.  European union regulations on algorithmic decision-making and a right to explanation.  AI Magazine, 38(3):50–57, 2017.
  * [12] Longbing Cao.  Ai in finance: Challenges, techniques and opportunities.  Banking & Insurance eJournal, 2021.
  * [13] Sandra Matz, Joe Gladstone, and David Stillwell.  Money buys happiness when spending fits our personality.  Psychological science, 27, 04 2016.
  * [14] Di Yao, Chao Zhang, Zhihua Zhu, Jianhui Huang, and Jingping Bi.  Trajectory clustering via deep representation learning.  International Joint Conference on Neural Networks (IJCNN), pages 3880–3887, 2017.
  * [15] S. Li and Handong Zhao.  A survey on representation learning for user modeling.  International Joint Conferences on Artificial Intelligence Organization (IJCAI), pages 4997–5003, 2020.
  * [16] Silviu-Marian Udrescu and Max Tegmark.  Ai feynman: a physics-inspired method for symbolic regression.  arXiv, 1905.11481, 2020.
  * [17] Yao Ming, Shaozu Cao, Ruixiang Zhang, Zhen Li, Yuanzhe Chen, Yangqiu Song, and Huamin Qu.  Understanding hidden memories of recurrent neural networks.  IEEE Conference on Visual Analytics Science and Technology (VAST), pages 13–24, 2017.
  * [18] Christian W. Omlin and Lee Giles.  Extraction of rules from discrete-time recurrent neural networks.  Neural Networks, 9(1):41–53, 1996.
  * [19] B. De Raad.  The big five personality factors: The psycholexical approach to personality.  Hogrefe & Huber Publishers, 2000.
  * [20] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [21] Andrea Ceni, Peter Ashwin, and Lorenzo Livi.  Interpreting recurrent neural networks behaviour via excitable network attractors.  Cognitive Computation, 12(2):330–356, 2019.
  * [22] Niru Maheswaranathan, Alex H. Williams, Matthew D. Golub, S. Ganguli, and David Sussillo.  Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics.  Advances in neural information processing systems, 32:15696–15705, 2019.
  * [23] Garrett E. Katz and James A. Reggia.  Using directional fibers to locate fixed points of recurrent neural networks.  IEEE Transactions on Neural Networks and Learning Systems, 29(8):3636–3646, 2018.