# Exploring language relations through syntactic distances and geographic
proximity

Juan De Gregorio  Raúl Toral  David Sánchez  Institute for Cross-Disciplinary
Physics and Complex Systems IFISC (UIB-CSIC), Campus Universitat de les Illes
Balears, E-07122 Palma de Mallorca, Spain

###### Abstract

Languages are grouped into families that share common linguistic traits. While
this approach has been successful in understanding genetic relations between
diverse languages, more analyses are needed to accurately quantify their
relatedness, especially in less studied linguistic levels such as syntax.
Here, we explore linguistic distances using series of parts of speech (POS)
extracted from the Universal Dependencies dataset. Within an information-
theoretic framework, we show that employing POS trigrams maximizes the
possibility of capturing syntactic variations while being at the same time
compatible with the amount of available data. Linguistic connections are then
established by assessing pairwise distances based on the POS distributions.
Intriguingly, our analysis reveals definite clusters that correspond to well
known language families and groups, with exceptions explained by distinct
morphological typologies. Furthermore, we obtain a significant correlation
between language similarity and geographic distance, which underscores the
influence of spatial proximity on language kinships.

Language distances; syntactic variation; POS dataset; POS n-grams; high-order
Markov processes; linguistic typology

##  I Introduction

The number of languages in the world is estimated to be around 7,000 Eberhard
_et al._ (2023). This leads to a broad diversity at all linguistic levels:
phonetic, morphosyntactic, semantic and pragmatic. The task to comprehend this
immense variation is overwhelming. However, researchers have managed to
pinpoint linguistic relationships that allow them to cluster languages in
groups and families. Such classification was first based on comparative
studies Hale (2007); Durie and Ross (1996) and now has increasingly been
supported with quantitative approaches Gray and Atkinson (2003); Gray  _et
al._ (2009).

A fertile approximation in historical linguistics describes languages as
species in a phylogenetic tree that shows the evolutionary history from proto-
languages to today’s descents Greenhill (2023). In this diachronic view,
languages change through linguistic innovations that cause two languages with
the same ancestor to become mutually unintelligible. There is also the
complementary view, followed in this work, that searches for relations among
languages in a given moment of time, in a synchronic manner de Saussure
(2011), with the goal of quantifying the distance between languages using an
appropriate metric Serva and Petroni (2008); Holman  _et al._ (2011).

Language distances are encoded in matrices whose entries measure the
similarity among certain linguistic features. The same method has been
successfully applied in dialectometry, which aims at quantifying the regional
differences among varieties of a given language Nerbonne (2009). Quantitative
measures of linguistic distances are useful not only for fundamental reasons
but also in applied linguistics with the aim of analyzing the learning
difficulties of minorities and immigrants Chiswick and Miller (2005). Another
application of language distances is for languages in contact since languages
that are more congruent to each other are more likely to coexist Mira and
Paredes (2005). An assessment of linguistic similarities is therefore helpful
for evidence-based policies and planning that seek to revitalize endangered
languages Fernando  _et al._ (2010).

Now, whilst most of the studies that compute distances either in synchronic
linguistics or in dialectometry focus on orthographic, phonetic or lexical
variations Nerbonne and Heeringa (1997); Downey  _et al._ (2008); Heeringa
_et al._ (2013); Donoso and Sánchez (2017); Gamallo  _et al._ (2017); Eden
(2018), less attention has been paid to morphosyntactic features, other than a
few exceptions Sanders (2010); Longobardi  _et al._ (2013); Dunn (2019). The
latter are interesting because syntax is more robust to change than phonetics
or semantics. Therefore, the resulting picture would show a larger time depth,
and unique cases of accelerated change would hence stand out.

A particularly simple but elucidating approach to analyze syntactic variations
is by means of parts of speech (POS). These denote word classes with well
defined grammatical functions that share common morphological properties
Manning and Schutze (1999). For instance, almost all languages distinguish
between verbs and nouns, i.e., roughly between actions and entities. As a
consequence, one can categorize words or lexical items as members of any of
the proposed POS, typically around 15. This classification has its own
limitations (e.g., certain languages do not distinguish between verbs and
adjectives) but it has the advantage of simplicity while capturing at the same
time a large amount of morphosyntactic information. The POS approach has been
proven especially useful in natural language processing tasks. The reason is
that POS reveal much not only about the syntactic category of a word but also
about that of its neighboring words, due to semantic restrictions. For
example, if a word is a noun it will most likely be surrounded by a
determiners and adjectives, forming a noun phrase. Therefore, we can gain
insight about the phrasal structure of a language by examining POS sequences.
This is precisely the main objective of this work: to model these sequences as
stochastic processes, analyze their correlations and compute syntactic
distances between languages using POS sequence distributions taken from a
multilingual corpus.

Statistics of POS sequences, specifically the analysis of POS $r$-grams,
defined as sequences of $r$ successive POS, has served for diverse research
goals. One may hypothesize that genres are characterized by different
syntactic structures, which would then allow for reliable genre
classification. As demonstrated in Ref. Feldman  _et al._ (2009), a careful
study of POS trigram histograms provides a high-performance genre classifier.
Strikingly, series of POS trigrams can be employed for building phylogenetic
language trees solely from translations Rabinovich  _et al._ (2017). The
premise here is that syntactic features are retained in the translation
process. Furthermore, assuming that POS tags can be predicted for historically
close languages it is possible to train a language model to measure proximity
among languages Samohi  _et al._ (2022). However, these previous works set the
POS block length in a somewhat heuristic manner. Below, we demonstrate using
information-theoretic methods that trigrams suffice to account for the
correlations present in POS sequences. In other words, it is not necessary to
consider $r$-grams with $r\geq 4$ to gain more information, a result that
considerably simplify analyses that involve POS series.

The mapping between a corpus and its corresponding POS series can be performed
with human or automatic parsers. For the purposes of our investigation, we
consider the Universal Dependencies library dataset Zeman  _et al._ (2023),
which includes manually annotated treebanks across many languages De Marneffe
_et al._ (2021). This dataset consists of texts and speech transcripts
originated from various sources: news, online social media, legal documents,
parliament speeches, literature, etc. The 17 universal parts of speech used in
this particular dataset are grouped into three classes: open, which includes
the tags adjective (ADJ), adverb (ADV), interjection (INTJ), noun (NOUN),
proper noun (PROPN) and verb (VERB); closed, with the tags adposition (ADP),
auxiliar (AUX), coordinating conjuntion (CCONJ), determiner (DET), numeral
(NUM), particle (PART), pronoun (PRON) and subordinating conjunction (SCONJ);
and others, which comprise punctuation (PUNCT), symbol (SYM) and other (X).

We then build a corpus of $67$ contemporary languages expressed by means of
these tags. Since the number of possible POS $r$-grams grows exponentially
with $r$, it is thus natural to ask what value of $r$ conveys the maximum
information about a language. As aforementioned, we find that $r=3$ suffices
to correctly characterize any of the studied languages. Then, we depict the
connections between languages assessing the pairwise distance between POS
trigram distributions. Interestingly, our found clusters can be identified
with well known families and groups. Exceptions can be understood due to
distinct linguistic typologies. This is natural since morphology constrains
the possible POS combinations that can form and, consequently, this is
reflected in the POS distributions and the distances calculated thereof.

Interestingly, we find a correlation between the obtained linguistic distance
and the geographic distance spanned between locations assigned to each
language. These are WALS locations, which correspond to the geographic
coordinates associated to the centre of the region where the analyzed
languages are spoken Dryer and Haspelmath (2013). Despite the fact that the
centres are obviously only approximate and that our calculation of the
linguistic distances has its own limitations, we clearly find that most of the
syntactically close languages are also geographically close. This is expected
since similar language usually lie in a continuum but there exist conspicuous
exceptions, as we shall discuss below.

##  II Methods

###  II.1 Data

The data utilized in this work is taken directly from the Universal
Dependencies library Zeman  _et al._ (2023), where each language is depicted
with one or several corpora that are manually tagged. Thus, each word is
classified into one of the possible POS categories as stated above. These tags
are deemed universal because they provide a common and consistent way to
represent the grammatical categories of words across different languages. For
example, the English sentence

Launching this way will hopefully avoid future disasters, giving more support
towards NASA revisiting the stars. (1)

is converted into the POS sequence

$\displaystyle S$ $\displaystyle=\text{VERB, DET, NOUN, AUX, ADV, VERB, ADJ,
NOUN, PUNCT, VERB, ADJ, }$ NOUN, SCONJ, PROPN, VERB, DET, NOUN, PUNCT (2)

Since we are interested in lexical classes that are either open or closed, we
combine the 3 categories in the others class in a single tag. Hence, we will
only consider $L=15$ distinct categories. As a consequence, in alphabetical
order, the possible tags are
$\\{z_{i}\\}_{i=0}^{14}=\\{\text{ADJ, ADP, ADV, AUX, CCONJ, DET, INTJ, NOUN,
NUM, PART,}$
$\text{PRON, PROPN, PUNCT, SCONJ, VERB}\\}$, where the categories in the
others class are included in the tag PUNCT. We can then count the occurrences
of each tag in $S$, excluding the period at the end of the sentence. For
example, the number of times the tag $z_{0}=\text{ADJ}$ occurs in Eq. (II.1)
is $2$. This way we gain access to unigram statistics. Similarly, we can group
$S$ in overlapping blocks of $2$ consecutive tags and count the number of
times each block is observed. For example, the block
$(z_{5},z_{7})=(\text{DET, NOUN})$ occurs in Eq. (II.1) $2$ times, thus
opening the path to bigram statistics, etc.

In general, the set of all $L^{r}$ possible $r$-grams, or blocks of size
$r\geq 1$, is given by
$\\{b_{j}^{(r)}:b_{(i_{0},\ldots,i_{r-1})_{L}}^{(r)}=(z_{i_{0}},\ldots,z_{i_{r-1}}),\quad
i_{0},\ldots,i_{r-1}=0,\ldots L-1\\}^{L^{{}^{r}}-1}_{{{j=0}}}$, where
$(i_{0},\ldots,i_{r-1})_{L}$ is a base $L$ number. Given the dataset of
language $\mathcal{L}$, formed by $R$ tagged sentences, we count the number of
appearances for each block of size $r$, with $1\leq r\leq r_{\text{max}}$,
with $r_{\text{max}}$ to be specified below. First, we arrange each of the $R$
sequences in overlapping blocks of size $r$ and calculate the occurrences
$\hat{n}_{j}^{(r)}$ of block $b^{(r)}_{j}$. With this information we build for
each value of $r$ the set of observations
$\\{\hat{n}_{j}^{(r)}\\}_{j=0}^{L^{{}^{r}}-1}$ for every language
$\mathcal{L}$.

$\mathcal{L}$ can be any of the $67$ contemporary European and Asian languages
that fulfill the criterion of having datasets of at least $10$ thousand tokens
in the Universal Dependencies library. The comprehensive list is included in
App. A, along with their respective language family and group, as well as
their corresponding morphological type (agglutinative, fusional, isolating).
The latter is a useful information about word formation. We emphasize that
these three types are just approximate categories since most of the languages
have morphological traits of the three types with variable relevance Comrie
(1989).

Importantly, we select languages within a contiguous region (except for
Afrikaans which is included since this belongs to the Indo-European family)
because we shall later explore possible correlations between linguistic and
geographic distances. Specifically, we focus on Eurasia since this is a single
continent with both a rich linguistic diversity and abundant data
availability.

Subsequently, we apply the previously discussed procedure of counting POS
$r$-grams occurrences for every language within our dataset. Following an
information-theoretic approach, we now argue in the next section that the
$r$-gram probability distribution of each language for $r=3$ is sufficient to
capture the correlations observed in POS sequences and, consequently, it can
serve as a reliable basis for calculating distances between languages.

###  II.2 Predictability gain and memory

The dynamics of many stochastic processes can be described by considering that
the transition probabilities to future outcomes depend on previous states.
Consider a random variable $X$ with $L$ possible outcomes
$z_{0},\ldots,z_{{L-1}}$ and probability distribution
$P(X)=\\{P(X=z_{i}),\quad i=0,\ldots,L-1\\}$, where $P(X=z_{i})$ is the
probability that $X$ takes the value $z_{i}$. In our case, $X$ are the POS
tags taking values specified above, and $P(X=z_{i})$ is the probability for
occurrence of tag $z_{i}$ in language $\mathcal{L}$. Given $k+1$ repetitions
of $X$, $X_{0},\ldots,X_{k}$, the accuracy of predicting the next outcome of
the process, $X_{k+1}$, generally grows as the number of past states
considered increases. For example, the information contained in the stationary
probability (or zeroth-order transition probability) $P(X_{k+1}=z_{j})$ for
outcome $z_{j}$ is always less or equally accurate than the information
provided by the first-order transition probability
$P(X_{k+1}=z_{j}|X_{k}=z_{l})$.

The predictability gain $\mathcal{G}_{u}$ Crutchfield and Feldman (2003)
quantifies the amount of information that one gains when performing
predictions taking into account $u+1$ past states instead of $u$. For
completeness, we prove in App. B that, for homogeneous systems in which the
transition probabilities are independent of time, the predictability gain
takes the form

$\mathcal{G}_{u}=-(H_{u+2}-2H_{u+1}+H_{u}),\quad u\geq 0,$ (3)

where $H_{r}$ is the block Shannon entropy Shannon (1948) of size $r\geq 1$
($H_{0}\equiv 0$). $H_{r}$ for $r\geq 1$ can be straightforwardly calculated
from the joint probability distribution $P_{r}(X)$ of $r$ consecutive
repetitions of the variable $X$,

$P_{r}(X)\equiv\\{P(X_{0}=z_{i_{0}},\ldots,X_{r-1}=z_{i_{r-1}}),\quad
i_{k}=0,\ldots,L-1\\},$ (4)

as

$\displaystyle\begin{split}H_{r}&=-\sum_{j=0}^{L^{r}-1}p(b_{j}^{(r)})\log(p(b_{j}^{(r)})),\end{split}$
(5)

where

$p(b_{(i_{0},\ldots,i_{r-1})_{L}}^{(r)})=P(X_{0}=z_{i_{0}},\ldots,X_{r-1}=z_{i_{r-1}}),\quad
0\leq i_{0},\ldots,i_{r-1}\leq L-1,$ (6)

is the probability for occurrence of block
$b_{(i_{0},\ldots,i_{r-1})_{L}}^{(r)}=(z_{i_{0}},\ldots,z_{i_{r-1}})$ with
size $r$, and $\log$ is hereafter understood as $\log_{2}$.

A stochastic process generated from consecutive repetitions of the variable
$X$ has order or memory $m\geq 1$ if the transition probabilities satisfy

$\begin{split}P(X_{k+1}=z_{j}|X_{0}=z_{v},\ldots,X_{k}=z_{l})=P(X_{k+1}=z_{j}|X_{k-(m-1)}=z_{o},\ldots,X_{k}=z_{l}).\end{split}$
(7)

These are usually referred to as $m$-th order Markovian processes Raftery
(1985). For the case $m=0$ the probabilities $P(X_{s}=z_{j})$ are independent
for all $s$.

In Ref. De Gregorio  _et al._ (2022) it was shown that a system has memory $m$
if and only if $H_{u}$ is a linear function of $u$ for $u\geq m$. This result
amounts to stating that a system has memory $m$ if and only if
$\mathcal{G}_{u}=0$ for all $u\geq m$. This can be proven directly from Eq.
(3). Hence, an analogous definition for the memory $m$ of a stochastic system
follows:

$m=\text{min}(\\{\mu:\mathcal{G}_{u}=0\text{, for all }u\geq\mu\\}).$ (8)

Therefore, $m$ is the minimum number of past states that we need to consider
in order to achieve maximum predictability of the random process. We can use
Eq. (8) as a benchmark to determine the memory of a system. Moreover, if a
stochastic process described with the random variable $X$ has memory $m$,
knowing the joint probability distribution $P_{m+1}(X)$ of $m+1$ consecutive
repetitions of $X$ is sufficient to capture all relevant information about the
process since $P_{m+1}$ can be used to compute the probabilities of larger and
smaller blocks. Hence, the $m+1$ block size is optimal for accurate
predictions and understanding the process dynamics, without adding redundant
information.

###  II.3 Estimation from finite data

Given the counts
$\\{\hat{n}^{(r)}\\}\equiv\\{\hat{n}_{j}^{(r)}\\}_{j=0}^{L^{{}^{r}}-1}$ of
language $\mathcal{L}$ obtained following the procedure outlined in Sec. II.1
for a value of $r$, we estimate the probability to observe block $b_{j}^{(r)}$
of $r$ consecutive POS tags in language $\mathcal{L}$ as

$\hat{p}_{{}_{\mathcal{L}}}(b_{j}^{(r)})=\dfrac{\hat{n}_{j}^{(r)}}{N^{(r)}},$
(9)

where $N^{(r)}=\sum_{j=0}^{L^{r}-1}\hat{n}_{j}^{(r)}$. It is well known that
Eq. (9) is an unbiased estimator for the probability
$p_{{}_{\mathcal{L}}}(b_{j}^{(r)})$. Similarly, from our observations we can
estimate the $(r-1)$th-order transition probabilities as

$\displaystyle\begin{split}\hat{p}_{{}_{\mathcal{L}}}(z_{i_{r}}|z_{i_{1}},\ldots,z_{i_{r-1}})\equiv\hat{P}(X_{r}=z_{i_{r}}|X_{1}=z_{i_{1}},\ldots,X_{r-1}=z_{i_{r-1}})=\dfrac{\hat{n}^{(r)}_{(i_{1},\ldots,i_{u},i_{r})_{L}}}{\sum\limits_{v=0}^{L-1}\hat{n}^{(r)}_{(i_{1},\ldots,i_{r-1},v)_{L}}},\end{split}$
(10)

with $i_{k}=0,\ldots,L-1$.

Our next step involves calculating the predictability gain in order to
determine the memory of the POS sequences by means of Eq. (8). Motivated by
Eq. (3) we can perform this task by estimating the block entropies.

Entropy estimation is a largely analyzed problem and, even though there is not
known unbiased estimator for the entropy Paninski (2003), numerous useful
estimators can be found in the literature Contreras Rodríguez  _et al._
(2021). Usually, all entropy estimators fail when the number of possible
outcomes is larger than the available data. Considering that the number of
possible blocks grows exponentially with increasing block size, there exists a
certain $r_{\text{max}}$ for which entropy estimation is unreliable for
$r>r_{\text{max}}$. The value of $r_{\text{max}}$ depends on the chosen
entropy estimator, but it is usual to take
$r_{\text{max}}\simeq\log(N^{(1)})/\log(L)$ De Gregorio  _et al._ (2022).

Hereafter we use the NSB entropy estimator Nemenman  _et al._ (2001, 2004),
which has shown to give good results for correlated sequences De Gregorio  _et
al._ (2024). Estimation of the block entropy of size $r$ using this method
only requires knowledge of the data observations $\\{\hat{n}^{(r)}\\}$. The
result of applying the estimator $\hat{H}$ to this set is denoted with
$\hat{H}\left[\\{\hat{n}^{(r)}\\}\right]$, for $1\leq r\leq r_{\text{max}}$.
Therefore, for finite data Eq. (3) becomes

$\hat{\mathcal{G}}_{u}[\\{\hat{n}\\}]=-\left(\hat{H}\left[\\{\hat{n}^{(u+2)}\\}\right]-2\hat{H}\left[\\{\hat{n}^{(u+1)}\\}\right]+\hat{H}\left[\\{\hat{n}^{(u)}\\}\right]\right),\quad
1\leq u\leq r_{\text{max}}-2,$ (11)

and

$\hat{\mathcal{G}}_{0}[\\{\hat{n}\\}]=-\left(\hat{H}\left[\\{\hat{n}^{(2)}\\}\right]-2\hat{H}\left[\\{\hat{n}^{(1)}\\}\right]\right).$
(12)

Due to the limitations of entropy estimation, the condition imposed in Eq. (8)
to determine the value of $m$ is too strict. Rather than attempting to
ascertain the minimum block size at which the predictability gain is $0$, a
more sensible approach is to compare the values of
$\hat{\mathcal{G}}_{u}[\\{\hat{n}\\}]$ with those obtained from a scenario
where we know that the estimated predictability gain should be $0$.

###  II.4 POS trigrams

In this section we provide evidence supporting our claim that in order to
capture the correlations in POS sequences it is enough to consider their
trigram probability distribution. We show this in two ways: in Sec. II.4.1,
calculating the predictability gain, and in Sec. II.4.2, analyzing the
accuracy in language detection as the order of Markovian models increases.

####  II.4.1 Predictability gain

Our first objective is to quantify the information gained in POS sequences of
varios languages when performing predictions considering $(u+1)$th-order
transition probabilities rather than $u$th-order.

In Fig. 1 we show $\hat{\mathcal{G}}_{u}$ for two representative languages of
the same group, namely, German (panel a) and Icelandic (panel b) from the
Germanic group, and two representative languages of different groups that also
differ from the previous one, namely, Portuguese (panel c) from the Romance
group and Czech (panel d) from the Slavic group. This way we can test
intragroup and intergroup variations, if they exist. Further, these three
groups constitute the most extensive clusters, characterized by a wealth of
available data, which make the results more trustworthy. We set
$r_{\text{max}}=5$ for the four languages; hence we ensure that the estimation
of their predictabilities is reliable up to $u=3$.

[figure]style=plain,subcapbesideposition=top

[] []
[] []

Figure 1: Estimated predictability gain when considering $(u+1)$th-order
instead of $u$th-order transition probabilities in POS sequences of (a)
German, (b) Icelandic, (c) Portuguese and (d) Czech, as extracted from the
Universal Dependencies library.

The values of $\hat{\mathcal{G}}_{u}$ at $u=0$ for the four languages
demonstrate a significant predictability gain when transitioning from zeroth-
order predictions to first-order predictions, suggesting that the POS state of
the sequence at a given step $k$ is highly informative about the next POS
outcome at step $k+1$. We observe that the values of $\hat{\mathcal{G}}_{0}$
significantly vary for each language, even for German and Icelandic which
belong to the same group. Then, the $\hat{\mathcal{G}}_{u}$ values at $u=1$
indicate that the information provided by the preceding POS outcome at step
$k-1$ is also substantial in terms of predictability. In contrast, the
predictability gain at $u=2$ drops to
$\hat{\mathcal{G}}_{2}\simeq\dfrac{\hat{\mathcal{G}}_{1}}{2}$ and remains
relatively constant at $u=3$. We obtain this decreasing pattern in the curves
of $\mathcal{\hat{G}}_{u}$ as a function of $u$ for all languages considered
in this work. Even though this does not prove that POS sequences have memory
$2$ (see App. C for further details), our results indeed show that considering
transition probabilities beyond order $2$ does not yield substantially more
information about the correlations present in our POS sequences.

####  II.4.2 Markov models

We now provide more evidence that POS trigrams (or equivalently, POS sequences
of memory $2$) describe most of the statistical information contained in the
represented languages.

From the corpus of language $\mathcal{L}$ we extract a tagged sentence
$S=x_{{}_{1}},\ldots,x_{{}_{N}}$ of length $N$ and compute the probability
$P(S|\mathcal{L}^{\prime})$ of observing $S$ in a certain language
$\mathcal{L}^{\prime}$. This probability can be computed either by considering
the estimated stationary distribution of language $\mathcal{L}^{\prime}$,

$\hat{P}^{(0)}(S|\mathcal{L}^{\prime})=\prod_{k=1}^{N}\hat{p}_{{}_{\mathcal{L}^{\prime}}}(x_{k}),$
(13)

or by incorporating $u$th-order transition probabilities for $u\geq 1$
($u$-order Markov model). For example, the probability of observing the
sequence $S$ given the language $\mathcal{L}^{\prime}$ considering first-order
transition probabilities reads

$\hat{P}^{(1)}(S|\mathcal{L}^{\prime})=\hat{p}_{{}_{\mathcal{L}^{\prime}}}(x_{1})\prod_{k=1}^{N-1}\hat{p}_{{}_{\mathcal{L}^{\prime}}}(x_{k+1}|x_{k}).$
(14)

Similarly, we can compute $\hat{P}^{(u)}(S|\mathcal{L}^{\prime})$ for $u>1$.
This procedure is repeated for various languages, and the language that yields
the highest probability is assigned to the one from which $S$ is generated.
Ideally, the obtained language would correspond to $\mathcal{L}$ for all
values of $u$.

For each of the four previously considered languages (German, Icelandic,
Portuguese, and Czech) we randomly select from their respective corpora $K$
sentences $S_{1},\ldots,S_{K}$, each comprising 5 to 20 word tokens.
Subsequently, for every sentence we compute
$\hat{P}^{(u)}(S_{l}|\mathcal{L}^{\prime})$ for the four languages and for
values of $u$ ranging from $0$ to $3$. The accuracy $A_{u}$ of correctly
identifying the language associated with each case is then assessed as the
fraction of correct sentence classifications:

$A_{u}(\mathcal{L})=\dfrac{\\#\\{l:\hat{P}^{(u)}(S_{l}|\mathcal{L})>\hat{P}^{(u)}(S_{l}|\mathcal{L}^{\prime})\text{
for all }\mathcal{L}^{\prime}\neq\mathcal{L}\\}_{l=1}^{{}^{K}}}{K}.$ (15)

We compute the estimated stationary and transition probabilities, as specified
in Eq. (9) and (10) respectively, without considering the influence of the $K$
sentences utilized for testing purposes. We then repeat this procedure $10$
times to find the mean and standard deviation of the accuracy for each
language, as a function of $u$. We consider $K=1000$. The results are
displayed in Fig. 2.

Figure 2: Accuracy in language identification, determined by computing and
comparing the probabilities of observing a given tagged sentence within each
considered language, based on an $u$th-order Markov model.

We observe that the accuracy for all languages significantly increases when
first-order transition probabilities are considered ($u=1$). Subsequently,
there is a slight increase at $u=2$, and afterwards the accuracy remains
relatively constant for $u=3$. These findings agree with the analysis
performed for the predictability gain, overall indicating that considering
memory values higher than $2$ does not provide significant additional
information. Conversely, choosing a low memory value proves advantageous as it
enhances the probability estimation accuracy. Consequently, this enables us to
incorporate a broader set of languages into our analysis.

###  II.5 Language distances

As explained in Sec. II.2, if a stochastic process generated from repetitions
of the variable $X$ has memory $m$, the probability distribution $P_{m+1}(X)$
is adequate for capturing all relevant information about the process. We
previously showed that sequences of POS tags can be modeled as processes with
memory $2$ with high accuracy. Therefore, we can define a distance metric
between languages $\mathcal{L}$ and $\mathcal{L}^{\prime}$ from the
statistical distance between their corresponding trigram distributions. To
this end, we consider the Jensen-Shannon (JS) distance Endres and Schindelin
(2003). Defining $\\{b_{j}\\}_{{j=0}}^{L^{{}^{3}}-1}$ as the set of all
possible POS trigrams, the JS distance between languages $\mathcal{L}$ and
$\mathcal{L}^{\prime}$ is determined by

$\displaystyle\begin{split}d_{{}_{JS}}(\mathcal{L},\mathcal{L}^{\prime})&=\sqrt{\dfrac{1}{2}\sum_{j=0}^{L^{{}^{3}}-1}\left(p_{{}_{\mathcal{L}}}(b_{j})\log\left(\dfrac{2p_{{}_{\mathcal{L}}}(b_{j})}{p_{{}_{\mathcal{L}}}(b_{j})+p_{{}_{\mathcal{L}^{\prime}}}(b_{j})}\right)+p_{{}_{\mathcal{L}^{\prime}}}(b_{j})\log\left(\dfrac{2p_{{}_{\mathcal{L}^{\prime}}}(b_{j})}{p_{{}_{\mathcal{L}}}(b_{j})+p_{{}_{\mathcal{L}^{\prime}}}(b_{j})}\right)\right)}\,\end{split}$
(16)

It is worth noting that the $d_{{}_{JS}}$ measure satisfies all the essential
properties expected for a metric and that $d_{{}_{JS}}$ ranges between 0 and
1.

[figure]style=plain,subcapbesideposition=top

[] []

Figure 3: Probability distribution of POS trigrams for (a) English and (b)
Japanese.

We estimate the JS distance by replacing the exact probabilities $p(b_{j})$ in
Eq. (16) with the maximum likelihood estimators given by Eq. (9). As an
illustration, we present in Fig. 3 the trigram probability distributions for
English (panel a) and Japanese (panel b). The numbers that designate the
trigrams correspond to their index. For example, trigram 0 refers to the block
$b_{0}=(z_{0},z_{0},z_{0})=(\text{ADJ, ADJ, ADJ})$; trigram 1 is
$b_{1}=(z_{0},z_{0},z_{1})=(\text{ADJ, ADJ, ADP})$; and so on until the last
trigram 3374, which corresponds to the block
$b_{3374}=(z_{14},z_{14},z_{14})=(\text{VERB, VERB, VERB})$.

From a simple inspection of Fig. 3 it is clear that the trigram probability
distributions of English ($\mathcal{E}$) and Japanese ($\mathcal{J}$) show
substantial differences. For example, in English the three most probable
trigrams are $b_{307}=\text{(ADP, DET, NOUN)}$, $b_{1231}=\text{(DET, NOUN,
ADP)}$ and $b_{1132}=\text{(DET, ADJ, NOUN)}$ whereas for Japanese we have
$b_{1597}=\text{(NOUN, ADP, NOUN)}$, $b_{1604}=\text{(NOUN, ADP, VERB)}$ and
$b_{331}=\text{(ADP, NOUN, ADP)}$. These trigrams differ because determiners
are generally absent from Japanese, unlike English, and adpositions follow the
Japanese nouns whereas in English adpositions can also appear before the noun.
These differences between the two languages can be quantified using Eq. (16).
We find $d_{{}_{JS}}(\mathcal{E},\mathcal{J})=0.79$, which is a high value due
to the strong morphosyntactic differences between Japanese and English.

Following a similar methodology, for each language pair within our dataset we
compute their corresponding JS distances using Eqs. (16), yielding a distance
matrix of dimension $67\times 67$, which bases our subsequent clustering
analysis. This methodology considers all available data for each language. An
alternative approach is presented in App. D, where we calculate distances
among text samples extracted from the same language group. We now present the
results obtained with the JS distance. Importantly, we also obtain similar
results employing a different metric (the Hellinger distance, see App. E),
which reinforces the validity of our findings.

##  III Results

We first discuss the general results obtained from calculating distances
between languages based on their POS distributions. Then, we inquire a
possible correlation between linguistic and physical (i. e., geographical)
distances.

###  III.1 Language distances and cluster analysis

####  III.1.1 Distance matrix

The distance matrix generated from the data with the aid of Eq. (16) can be
better visualized through a clustermap (see Fig. 4). This representation
employs hierarchical clustering Nielsen (2016) and heatmap visualization. We
use the complete linkage method Defays (1977) for clustering, organizing rows
and columns based on similarity, depicted as dendrograms in the same figure.
The heatmap, which represents distance values by a color spectrum as indicated
in the inset of Fig. 4, enables a comprehensive exploration of our data
relationships and structure.

Figure 4: Heatmap visualization of the Jensen-Shannon distance matrix,
calculated from POS trigram distributions. Rows and columns are organized
based on hierarchical clustering. The colour spectrum in the heatmap
illustrates data matrix values.

First, since darker colours indicate shorter distance (i.e, greater
morphosyntactic similarity) one can clearly distinguish large-scale cluster
formation. In the vicinity of the left upper corner we observe the most
extensive cluster, corresponding to Slavic languages. Within this language
group, we discern smaller clusters, the most clear being among Belarusian,
Russian and Ukrainian (the East branch of the Slavic family), as well as
between Serbian and Croatian (the South branch). Interestingly, our results
point to a close relationship between this group and two members of the Baltic
group (Latvian and Lithuanian). This suggests that spatial proximity might be
correlated with our POS distances, as we discuss below.

Germanic languages exhibit a more dispersed pattern, manifested itself in two
distinct clusters. The first cluster encompasses Afrikaans, Dutch, and German
(the West branch), while the second is primarily composed of North Germanic
languages along with English, which highlights its mixed character via a close
connection with Romance languages. The latter, in turn, exhibit the most
compact clustering, as evidenced by their tight proximity to one another in
the heatmap. An exception is Romanian, which falls outside this group,
probably because it is the only major Romance language with noun declension
and article enclitics. Also, its spatial distance from the West may play a
role. This interpretration is supported by the fact that Romanian is clustered
with Pomak, a Bulgarian variety spoken in the geographically close region of
Thrace.

We next find a small cluster that encompasses Maltese and Hebrew.
Surprisingly, Arabic, classified as a Semitic language as well, appears closer
to Austronesian languages, such as Indonesian and Javanese, as well as
Persian. The reason lies in the different typology of Arabic and its strong
dialect diversity. Further visible clusters correspond to Celtic languages,
Hindi and Urdu (both belonging to the Indic language group), the Turkic family
(Turkish, Kazakh and Uyghur), adjacent to Buryat (a Mongolic language),
followed by a larger cluster primarily comprising Uralic, Tungusic, and Sino-
Tibetan families. This is the most diverse cluster partly because the
Universal Dependencies library contains data of a few languages only as
compared with the previous families. However, there exist interesting
connections that we explore in the next sections.

####  III.1.2 Language tree and k-medoids clustering

In order to gain further insight on the relations among the distinct
languages, we perform a k-medoids clustering upon the distance matrix. Hence,
we utilize the Partition Around Medoids (PAM) algorithm Kaufman and Rousseeuw
(2009), for which the optimal number of clusters is determined through a
silhouette analysis Rousseeuw (1987). The corresponding figure and a list of
languages constituting each cluster are presented in App. F. For the moment,
we will make use of these results to build a single picture of the main
interlinguistic connections.

First, we construct a network whose nodes represent languages whereas the
edges’ weight are given by their pairwise distances. Only the edges that
generate a minimum spanning tree are considered as part of our analysis. The
minimum spanning tree is a structure that connects all the data points in our
dataset results with the minimum possible total edge weight Gower and Ross
(1969). We depict the tree in Fig. 5, employing the Kamada-Kawai layout Kamada
_et al._ (1989). This is a force-directed layout, implying that, for weighted
graphs as in our case, the length of the edges tend to be larger for heavier
weights. This visualization provides an intuitive representation of the
relationships encoded in the minimum spanning tree, offering insights into the
overall structure and connectivity patterns, not only among closely related
languages, but also between distinct language families. To enhance our
interpretability, we assign consistent colors to items within the same
cluster, obtained from the k-medoids analysis mentioned earlier. Nodes from
the same language group are linked with a full line; nodes from the same
family but different group are linked with a dashed line; finally, nodes
belonging to distinct families are connected with dotted lines. The shape of
the nodes represents the language type: circles are assigned to fusional
languages, squares to the agglutinative type and diamonds to isolating
languages. The languages plotted with half nodes are both isolating and
another type. For example, English is plotted with a half diamond node because
it is considered to be a fusional-isolating language Haselow (2011).

Figure 5: Minimum spanning tree generated from the Jensen-Shannon distance
matrix, with node colors representing clusters identified through k-medoids
analysis. Nodes’ shape represent the language typology. Full lines are
assigned to links between languages belonging to the same group; dashed lines
for languages of the same family but different group; finally, dotted lines
connect languages from distinct families.

Interestingly, the same cluster formations observed with our previous analysis
of Fig. 4 naturally emerge in the constructed tree, as well as connections
between language families hidden within the distance matrix, which are easier
to identify in this simpler visualization. We can find most of the Celtic
languages (clusters $6$, $28$) close to a few Semitic languages (cluster
$25$), specially Hebrew, whereas Maltese can be found in close proximity to
Romance languages, which are once again forming a compact cluster ($1$), with
the exception of Romanian which is clustered with Icelandic and Faroese
(cluster $21$). The Germanic group is subdivided into three groups (clusters
$3$, $20$ and $21$) that are relatively close to one another, and similarly
with the Slavic group (clusters $0$, $8$, $26$, $30$). The Baltic languages
are clustered together with the Slavic sub-group mostly consisting in East
Slavic languages (cluster $0$). The proximity of these two groups was also
observed in Fig. 4.

The Baltic languages are connected with the Uralic languages (clusters $5$,
$22$) and with the Armenian family (cluster $10$). An exception is Hungarian,
considered an Uralic language, but here appearing separately (cluster $23$)
near the Turkic languages, which are grouped together as expected, alongside
Buryat (cluster $2$). Around this cluster several isolated languages such as
Korean (cluster $19$), Tamil (cluster $16$) and Xibe (cluster $18$) can be
found.

Arabic is clustered with Austronesian languages (cluster $7$), in close
proximity to the Iranian cluster (cluster $9$). The Indic languages (cluster
$4$) are noticeably distant from the rest, and even further is Japanese
(cluster $13$). The remaining clusters are formed by only one language,
namely: Gheg ($12$), Thai ($14$), Basque ($24$), Greek ($30$), Cantonese
($11$) and Chinese ($15$). With the exception on the latter two, which belong
to the Sino-Tibetan family, all others are the sole representatives of their
respective language groups being considered in this analysis.

It is important to note that, even though this form of visualization can be
helpful to recognize clusters and connection among languages, some of these
connections are spurious and mainly arise from the inherent nature of the
minimum spanning tree, which is designed to avoid leaving isolated nodes.
Nevertheless, it is remarkable that most of the connections observed in Fig.
5, represented both by the cluster identification of the nodes and the edges
linking them, are between languages belonging to the same group. Moreover,
most of the languages displayed in the upper part of the figure belong to the
Indo-European family, and they all have fusional features. On the other hand,
the lower part of the chart mostly consists of agglutinative and isolating
languages.

Less known connections, such as among Hebrew and Celtic languages, or between
Basque and Armenian are also displayed in the tree. These similarities have
been already pointed out in Refs. Gensler (1993) and Tamrazian (1994)
respectively, with the proper observation that this does not imply a common
origin of the languages. A similar case is that of Altaic languages Starostin
_et al._ (2003), a putative family relationship between Turkic, Mongolic,
Tungusic, Korean and Japanese languages, a subject characterized by ongoing
debate within linguistic research Janhunen (2023). In Fig. 5 this group is
represented in the lower part of the tree by the connections among Turkish,
Kazakh, Uyghur, Xibe, Buryat and Korean. Despite the controversy surrounding
the existence of a genetic relationship, it is generally acknowledged that
there exist linguistic similarities among these languages, as evidenced by our
analysis.

Romanian language, which is considered to be a Romance language, can be found
in Figs. 4 and 5 closer to Germanic languages than to other Romance languages.
Interestingly, this fact has been observed in Ref. Samohi  _et al._ (2022),
also using Universal Dependencies data but considering a machine learning
approach to calculate linguistic distances. In Ref. Rabinovich  _et al._
(2017), a similar cluster analysis based on the analysis of POS trigrams of
translations locates Romanian language away from the Romance group.

Another interesting case is that of Arabic, a Semitic language which, in the
dendrogram in Fig. 4 is clustered alongside Persian and Kurmanji, both Iranian
languages. In the tree presented in Fig. 5, Arabic and Persian are linked but
the former is clustered by the k-medoids algorithm with Indonesian and
Javanese, which belong to the Austronesian family. In this figure, Arabic
appears in the lowest part of the tree, very distant from the other Semitic
languages, Hebrew and Maltese. However, this is misleading because their
linguistic distance in Fig. 4 is not high, specially between Arabic and
Hebrew. Further, the linguistic proximity observed between Arabic and Persian
is influenced by their geographic closeness.

Overall, our cluster results are consistent with well established families and
linguistic groups. We indeed observe departures that could be attributed to
methodology inaccuracies. However, another interpretation is possible. Very
recently, a family tree based on the phylogenetic signal of syntactic data has
been inferred Hartmann and Walkden (2024), pointing to salient deviations with
respect to the trees derived from the comparative method, which typically did
no take into account syntactic data. Therefore, language groups formed upon
syntactic analyses need not to fully agree with those groups that emerge from
phonetic or lexical similarities.

Strikingly, our results also suggest correlations between syntactic distances
and spatial proximity. In the following section, we explore the connection
between these two variables, which can account for the relationship observed
in Fig. 5 between, e.g., Semitic and Romance languages, Indonesian with
Vietnamese or Uralic with Slavic languages.

###  III.2 Relation between linguistic and geographic distances

We calculate the geographic (geodesic) distances between all language pairs by
assigning an spatial coordinate to each language considered. This geolocation
information is obtained from the World Atlas of Language Structures Online
(WALS) Dryer and Haspelmath (2013). For this analysis we exclude Afrikaans
since it is geographically isolated from the rest of the languages considered.

[figure]style=plain,subcapbesideposition=top

[] []
[] []

Figure 6: Linguistic versus geographic distances between a selected language
and all other languages considered: (a) German, (b) Portuguese, (c) Czech and
(d) Basque. $R_{p}$ denotes the Pearson correlation coefficient.

First, for a single language we compute its linguistic and geographic
distances with the rest of considered languages. The plots obtained for German
(panel a), Portuguese (panel b), Czech (panel c) and Basque (panel d) are
presented in Fig. 6, applying logarithmic scale to the geographic distances.
The obtained Pearson correlation coefficients ($R_{p}=0.721$, $0.700$,
$0.600$, and $-0.510$, respectively) point to a logarithmic relation between
the two distances (all p-values are $<0.001$). Surprisingly, out of the $66$
languages analyzed, only Basque [Fig. 6(d)] shows a significant negative
correlation between linguistic and geographic distances. This is because
Basque is categorized as an agglutinative language and while the majority of
Western European languages lean towards fusional characteristics,
agglutinative languages are predominantly found in Eastern Europe and Asia.
Consequently, Basque shares more linguistic similarities with languages spoken
at greater distances than with those geographically closer.

Figure 7: Linguistic and geographic distances between all language pairs
considered. The distance correlation coefficient is $R_{d}=0.447$.

Finally, we compute the linguistic and geographic distances for all language
pairs in order to globally explore the relation between these two variables.
We show the resulting plot in Fig. 7. Quite generally, there is a positive
correlation between the syntactic and spatial variables. We can quantify this
dependence with the the distance correlation coefficient $R_{d}$ Székely  _et
al._ (2007). This coefficient is more general than Pearson’s $R_{p}$ since it
does not only measures linear dependence between variables. We find
$R_{d}=0.447$, with a p-value $<0.001$ calculated from a permutation test.
Even though $R_{d}$ is not high due to the uncertainties associated to the
noisy data and the geographical locations, the value is significantly greater
than $0$ which indicates that geographical and linguistic distances are indeed
correlated.

##  IV Conclusions

In this work we have collected and analyzed POS tagged sentences available in
the Universal Dependencies library for $67$ contemporary languages located in
a geographically contiguous region. Following an information-theoretic
approach, we have provided evidence showing the effectiveness of utilizing the
trigram probability distribution of parts-of-speech for characterizing the
syntax statistics of languages. Through this method, we have computed
distances between languages by calculating statistical divergences between
trigram distributions, revealing both well-established language groupings and
less familiar but already documented linguistic connections. Furthermore, our
analysis has delved into the correlation between linguistic and geographic
distances. We have found that spatially proximate languages tend to exhibit
more similar morphosyntactic characteristics compared to those located farther
apart. The obtained results suggest a logarithmic relation between these
distances. This finding is in fact in good agreement with results reported in
Refs. Nerbonne (2010); Jäger (2018), where different measures of linguistic
distances are defined.

We emphasize that the methodology detailed in Sec. II for analyzing
correlations within discrete sequences is versatile and applicable across
numerous disciplines. These techniques hold relevance beyond the realm of
linguistics and can be effectively employed in various fields, including but
not limited to statistical physics, biology and data science, especially for
systems that can be modeled as short-memory stochastic processes with few
states.

Looking forward, our research suggests several promising avenues for future
exploration. Firstly, expanding the range of languages considered in our
analysis, even including different varieties of certain languages, could
provide a more comprehensive understanding of linguistic diversity and
relationships across different language families. Additionally, investigating
the evolution of morphosyntactic similarities over time holds great potential
for uncovering insights into the dynamical nature of grammar and word
formation systems and their historical interconnections. For that, however,
one would need multilingual diachronic corpora with a sufficiently long time
span.

##  V Data availability

The datasets of parts-of-speech for all languages analyzed during the current
study are available in the Universal Dependencies repository. The dataset of
the geographic location of each language is available in the WALS repository.

##  VI Acknowledgments

This work has been supported by the Agencia Estatal de Investigación (AEI,
MICIU, Spain) MICIU/AEI/10.13039/501100011033 and Fondo Europeo de Desarrollo
Regional (FEDER, UE) under Project APASOS (PID2021-122256NB-C21), the María de
Maeztu Program for units of Excellence in R&D, grant CEX2021-001164-M, and by
the Government of the Balearic Islands CAIB fund ITS2017-006 under project
CAFECONMIEL (PDR2020/51).

## References

  * Eberhard  _et al._ (2023) D. Eberhard, G. F. S. Simons, and C. D. Fennig (eds.),  _Ethnologue_ , 23rd ed. (SIL International, Dallas, Texas, 2023).
  * Hale (2007) M. Hale,  _Historical Linguistics: Theory and Method_ (Backwell Publishing, Hoboken, New Jersey, 2007).
  * Durie and Ross (1996) M. Durie and M. Ross,  _The comparative method reviewed: Regularity and irregularity in language change_ (Oxford University Press, Oxford, 1996).
  * Gray and Atkinson (2003) R. D. Gray and Q. D. Atkinson, “Language-tree divergence times support the Anatolian theory of Indo-European origin,” Nature 426, 435–439 (2003).
  * Gray  _et al._ (2009) R. D. Gray, A. J. Drummond, and S. J. Greenhill, “Language phylogenies reveal expansion pulses and pauses in Pacific settlement,” Science 323, 479–483 (2009).
  * Greenhill (2023) S. J. Greenhill, “Language phylogenies: modelling the evolution of language,” in  _The Oxford Handbook of Cultural Evolution_ (Oxford Academic, 2023).
  * de Saussure (2011) F. de Saussure,  _Course in General Linguistics_ (Columbia University Press, New York, 2011).
  * Serva and Petroni (2008) M. Serva and F. Petroni, “Indo-European languages tree by Levenshtein distance,” Europhysics letters 81, 68005 (2008).
  * Holman  _et al._ (2011) E. W. Holman, C. H. Brown, S. Wichmann, A. Müller, V. Velupillai, H. Hammarström, S. Sauppe, H. Jung, D. Bakker, P. Brown, _et al._ , “Automated dating of the world’s language families based on lexical similarity,” Current Anthropology 52, 841–875 (2011).
  * Nerbonne (2009) J. Nerbonne, “Data-driven dialectology,” Language and linguistics compass 3, 175–198 (2009).
  * Chiswick and Miller (2005) B. R. Chiswick and P. W. Miller, “Linguistic distance: A quantitative measure of the distance between english and other languages,” Journal of multilingual and multicultural development 26, 1–11 (2005).
  * Mira and Paredes (2005) J. Mira and Á. Paredes, “Interlinguistic similarity and language death dynamics,” Europhysics Letters 69, 1031 (2005).
  * Fernando  _et al._ (2010) C. Fernando, R.-L. Valijärvi, and R. A. Goldstein, “A model of the mechanisms of language extinction and revitalization strategies to save endangered languages,” Human biology 82, 47–75 (2010).
  * Nerbonne and Heeringa (1997) J. Nerbonne and W. Heeringa, “Measuring dialect distance phonetically,” in  _Computational phonology: third meeting of the acl special interest group in computational phonology_ (1997).
  * Downey  _et al._ (2008) S. S. Downey, B. Hallmark, M. P. Cox, P. Norquest, and J. S. Lansing, “Computational feature-sensitive reconstruction of language relationships: Developing the aline distance for comparative historical linguistic reconstruction,” Journal of Quantitative Linguistics 15, 340–369 (2008).
  * Heeringa  _et al._ (2013) W. Heeringa, J. Golubovic, C. Gooskens, A. Schüppert, F. Swarte, and S. Voigt, “Lexical and orthographic distances between Germanic, Romance and Slavic languages and their relationship to geographic distance,” in  _Phonetics in Europe_ , edited by C. Gooskens and R. van Bezooijen (P.I.E. - Peter Lang, 2013) pp. 99–137.
  * Donoso and Sánchez (2017) G. Donoso and D. Sánchez, “Dialectometric analysis of language variation in Twitter,” in  _Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)_, edited by P. Nakov, M. Zampieri, N. Ljubešić, J. Tiedemann, S. Malmasi, and A. Ali (Association for Computational Linguistics, Valencia, Spain, 2017) pp. 16–25.
  * Gamallo  _et al._ (2017) P. Gamallo, J. R. Pichel, and I. Alegria, “From language identification to language distance,” Physica A: Statistical Mechanics and its Applications 484, 152–162 (2017).
  * Eden (2018) S. E. Eden,  _Measuring phonological distance between languages_ , Ph.D. thesis, UCL (University College London) (2018).
  * Sanders (2010) N. C. Sanders,  _A Statistical Method for Syntactic Dialectometry_ (Indiana University, 2010).
  * Longobardi  _et al._ (2013) G. Longobardi, C. Guardiano, G. Silvestri, A. Boattini, and A. Ceolin, “Toward a syntactic phylogeny of modern Indo-European languages,” Journal of Historical Linguistics 3, 122–152 (2013).
  * Dunn (2019) J. Dunn, “Global syntactic variation in seven languages: Toward a computational dialectology,” Frontiers in Artificial Intelligence 2, 15 (2019).
  * Manning and Schutze (1999) C. Manning and H. Schutze,  _Foundations of statistical natural language processing_ (MIT press, Cambridge, 1999).
  * Feldman  _et al._ (2009) S. Feldman, M. A. Marin, M. Ostendorf, and M. R. Gupta, “Part-of-speech histograms for genre classification of text,” in  _2009 IEEE international conference on acoustics, speech and signal processing_ (IEEE, 2009) pp. 4781–4784.
  * Rabinovich  _et al._ (2017) E. Rabinovich, N. Ordan, and S. Wintner, “Found in translation: Reconstructing phylogenetic language trees from translations,” in  _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ (2017) pp. 530–540.
  * Samohi  _et al._ (2022) A. Samohi, D. W. Mitelman, and K. Bar, “Using cross-lingual part of speech tagging for partially reconstructing the classic language family tree model,” in  _Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change_ (2022) pp. 78–88.
  * Zeman  _et al._ (2023) D. Zeman _et al._ , “Universal dependencies 2.13,”  (2023), LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
  * De Marneffe  _et al._ (2021) M.-C. De Marneffe, C. D. Manning, J. Nivre, and D. Zeman, “Universal dependencies,” Computational linguistics 47, 255–308 (2021).
  * Dryer and Haspelmath (2013) M. S. Dryer and M. Haspelmath, eds.,  _WALS Online (v2020.3)_ (Zenodo, 2013).
  * Comrie (1989) B. Comrie,  _Language universals and linguistic typology: Syntax and morphology_ (University of Chicago press, 1989).
  * Crutchfield and Feldman (2003) J. P. Crutchfield and D. P. Feldman, “Regularities unseen, randomness observed: Levels of entropy convergence,” Chaos: An Interdisciplinary Journal of Nonlinear Science 13, 25–54 (2003).
  * Shannon (1948) C. E. Shannon, “A mathematical theory of communication,” Bell System Technical Journal 27, 379–423 (1948).
  * Raftery (1985) A. E. Raftery, “A model for high-order Markov chains,” Journal of the Royal Statistical Society: Series B (Methodological) 47, 528–539 (1985).
  * De Gregorio  _et al._ (2022) J. De Gregorio, D. Sánchez, and R. Toral, “An improved estimator of Shannon entropy with applications to systems with memory,” Chaos, Solitons & Fractals 165, 112797 (2022).
  * Paninski (2003) L. Paninski, “Estimation of entropy and mutual information,” Neural Computation 15, 1191–1253 (2003).
  * Contreras Rodríguez  _et al._ (2021) L. Contreras Rodríguez, E. J. Madarro-Capó, C. M. Legón-Pérez, O. Rojas, and G. Sosa-Gómez, “Selecting an effective entropy estimator for short sequences of bits and bytes with maximum entropy,” Entropy 23, 561 (2021).
  * Nemenman  _et al._ (2001) I. Nemenman, F. Shafee, and W. Bialek, “Entropy and inference, revisited,” in  _Advances in Neural Information Processing Systems_ , Vol. 14, edited by T. Dietterich, S. Becker, and Z. Ghahramani (MIT Press, 2001).
  * Nemenman  _et al._ (2004) I. Nemenman, W. Bialek, and R. de Ruyter van Steveninck, “Entropy and information in neural spike trains: Progress on the sampling problem,” Phys. Rev. E 69, 056111 (2004).
  * De Gregorio  _et al._ (2024) J. De Gregorio, D. Sánchez, and R. Toral, “Entropy estimators for Markovian sequences: A comparative analysis,” Entropy 26 (2024), 10.3390/e26010079.
  * Endres and Schindelin (2003) D. Endres and J. Schindelin, “A new metric for probability distributions,” IEEE Transactions on Information Theory 49, 1858–1860 (2003).
  * Nielsen (2016) F. Nielsen, “Hierarchical clustering,” in  _Introduction to HPC with MPI for Data Science_ (Springer International Publishing, Cham, 2016) pp. 195–211.
  * Defays (1977) D. Defays, “An efficient algorithm for a complete link method,” The Computer Journal 20, 364–366 (1977).
  * Kaufman and Rousseeuw (2009) L. Kaufman and P. J. Rousseeuw,  _Finding groups in data: an introduction to cluster analysis_ (John Wiley & Sons, 2009).
  * Rousseeuw (1987) P. J. Rousseeuw, “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,” Journal of Computational and Applied Mathematics 20, 53–65 (1987).
  * Gower and Ross (1969) J. C. Gower and G. J. Ross, “Minimum spanning trees and single linkage cluster analysis,” Journal of the Royal Statistical Society: Series C (Applied Statistics) 18, 54–64 (1969).
  * Kamada  _et al._ (1989) T. Kamada, S. Kawai, _et al._ , “An algorithm for drawing general undirected graphs,” Information processing letters 31, 7–15 (1989).
  * Haselow (2011) A. Haselow,  _Typological changes in the lexicon: Analytic tendencies in English noun formation_ , Vol. 72 (Walter de Gruyter, 2011).
  * Gensler (1993) O. Gensler,  _A typological evaluation of Celtic/Hamito-Semitic syntactic parallels_ , Ph.D. thesis, University of California (1993).
  * Tamrazian (1994) A. Tamrazian,  _The syntax of Armenian: Chains and the auxiliary_ , Ph.D. thesis, University of London, University College London (United Kingdom) (1994).
  * Starostin  _et al._ (2003) S. A. Starostin, A. V. Dybo, O. Mudrak, and I. Gruntov,  _Etymological dictionary of the Altaic languages_ , Vol. 3 (Brill Leiden, 2003).
  * Janhunen (2023) J. A. Janhunen, “The unity and diversity of Altaic,” Annual Review of Linguistics 9, 135–154 (2023).
  * Hartmann and Walkden (2024) F. Hartmann and G. Walkden, “The strength of the phylogenetic signal in syntactic data,” Glossa: a journal of general linguistics 9, 1–25 (2024).
  * Székely  _et al._ (2007) G. J. Székely, M. L. Rizzo, and N. K. Bakirov, “Measuring and testing dependence by correlation of distances,” The Annals of Statistics 35, 2769 – 2794 (2007).
  * Nerbonne (2010) J. Nerbonne, “Measuring the diffusion of linguistic change,” Philosophical Transactions of the Royal Society B: Biological Sciences 365, 3821–3828 (2010).
  * Jäger (2018) G. Jäger, “Global-scale phylogenetic linguistic inference from lexical resources,” Scientific Data 5, 1–16 (2018).
  * Cover and Thomas (2006) T. Cover and J. Thomas,  _Elements of Information Theory_ (John Wiley and Sons, 2006).
  * Altmann  _et al._ (2012) E. G. Altmann, G. Cristadoro, and M. D. Esposti, “On the origin of long-range correlations in texts,” Proceedings of the National Academy of Sciences 109, 11582–11587 (2012).

##  Appendix A List of languages

We present in Table LABEL:tab:languages a list of the $67$ languages included
in our study, categorized by language family, group, and morphological type.

Table A.1: List of languages included in this work, classified by language family, group and type. Family | Group | Language | Type
---|---|---|---
Afro-Asiatic | Semitic | Arabic | Isolating-Fusional
Hebrew | Fusional
Maltese | Fusional
Austro-Asiatic | Viet-Muong | Vietnamese | Isolating
Austronesian | Javanese | Javanese | Agglutinative
Malayo-Sumbawan | Indonesian | Agglutinative
Basque |  | Basque | Agglutinative
Dravidian |  | Tamil | Agglutinative
Indo-European | Albanian | Gheg | Fusional
Armenian | Armenian | Agglutinative
Western Armenian | Agglutinative
Baltic | Latvian | Fusional
Lithuanian | Fusional
Celtic | Breton | Fusional
Irish | Fusional
Manx | Fusional
Scottish Gaelic | Fusional
Welsh | Fusional
Germanic | Afrikaans | Isolating-Fusional
Danish | Fusional
Dutch | Fusional
English | Isolating-Fusional
Faroese | Fusional
| German | Fusional
| Icelandic | Fusional
| Norwegian | Fusional
| Swedish | Fusional
| Greek | Greek | Fusional
| Indic | Hindi | Fusional
| Urdu | Fusional
| Iranian | Kurmanji | Fusional-Agglutinative
| Persian | Agglutinative
| Romance | Catalan | Fusional
| French | Fusional
| Galician | Fusional
| Italian | Fusional
| Portuguese | Fusional
| Romanian | Fusional
| Spanish | Fusional
|  | Belarusian | Fusional
|  | Bulgarian | Isolating-Fusional
|  | Croatian | Fusional
|  | Czech | Fusional
|  | Polish | Fusional
|  | Pomak | Fusional
| Slavic | Russian | Fusional
|  | Serbian | Fusional
|  | Slovak | Fusional
|  | Slovenian | Fusional
|  | Ukrianian | Fusional
|  | Upper Sorbian | Fusional
Japanese |  | Japanese | Agglutinative
Korean |  | Korean | Agglutinative
Mongolic |  | Buryat | Agglutinative
Sino-Tibetan |  | Cantonese | Isolating
| Chinese | Isolating
Tai-Kadai |  | Thai | Isolating
Tungusic |  | Xibe | Agglutinative
Turkic | Northwestern | Kazakh | Agglutinative
Southeastern | Uyghur | Agglutinative
Southwestern | Turkish | Agglutinative
Uralic | Finnic | Estonian | Agglutinative
Finnish | Agglutinative
Mordvin | Erzya | Agglutinative
| Permic | Komi Zyrian | Agglutinative
| Sami | North Sami | Agglutinative
| Ugric | Hungarian | Agglutinative

##  Appendix B Predictability gain and block entropy

The predictability gain of going from $u$th-order to $(u+1)$th-order
transition probabilities, for $u\geq 1$, can be defined in terms of a
conditional relative entropy Cover and Thomas (2006) as

$\mathcal{G}_{u}=\sum_{x_{0},\ldots,x_{u+1}}p(x_{0},x_{1},\ldots,x_{u},x_{u+1})\log\left(\dfrac{p(x_{u+1}|x_{0},\ldots,x_{u})}{p(x_{u+1}|x_{1},\ldots,x_{u})}\right),$
(17)

where we adopt the notation that $p(x_{0},\ldots,x_{s})\equiv
P(X_{0}=x_{0},\ldots,X_{s}=x_{s})$, with $x_{i}\in\\{z_{j}\\}_{j=0}^{L-1}$,
and similarly for the transition probabilities.

We will use the following general result for the difference between the block
entropies $H_{r+1}-H_{r}$, demonstrated in the appendix of Ref. De Gregorio
_et al._ (2022):

$H_{r+1}-H_{r}=-\sum_{x_{1},\ldots,x_{r+1}}p(x_{1},\ldots,x_{r+1})\log(p(x_{r+1}|x_{1},\ldots,x_{r})).$
(18)

By the properties of the logarithm function, we can write Eq. (17) as

$\mathcal{G}_{u}=\sum_{x_{0},\ldots,x_{u+1}}p(x_{0},\ldots,x_{u+1})\log(p(x_{u+1}|x_{0},\ldots,x_{u}))-\sum_{x_{0},\ldots,x_{u+1}}p(x_{0},\ldots,x_{u+1})\log(p(x_{u+1}|x_{1},\ldots,x_{u})).$
(19)

Adding $1$ to all indices in the first sum in Eq. (19), which is allowed given
that we are considering homogeneous sequences, and using
$\sum\limits_{x_{0}}p(x_{0},x_{1},\ldots,x_{u+1})=p(x_{1},\ldots,x_{u+1})$, we
write Eq. (19) as

$\mathcal{G}_{u}=\sum_{x_{1},\ldots,x_{u+2}}p(x_{1},\ldots,x_{u+2})\log(p(x_{u+2}|x_{1},\ldots,x_{u+1}))-\sum_{x_{1},\ldots,x_{u+1}}p(x_{1},\ldots,x_{u+1})\log(p(x_{u+1}|x_{1},\ldots,x_{u})).$
(20)

Using Eq. (18) we express Eq. (20) as

$\displaystyle\begin{split}\mathcal{G}_{u}=-(H_{u+2}-H_{u+1})+(H_{u+1}-H_{u})=-(H_{u+2}-2H_{u+1}+H_{u}),\end{split}$
(21)

which is exactly the result we wanted to prove. The case $u=0$ can be
demonstrated similarly, considering that

$\mathcal{G}_{0}=\sum_{x_{0},x_{1}}p(x_{0},x_{1})\log\left(\dfrac{p(x_{1}|x_{0})}{p(x_{1})}\right).$
(22)

##  Appendix C Memory effects in POS sequences

In Sec. II.4, we have shown that POS sequences can be approximated as
stochastic processes with memory 2 and that this approximation is good.
However, this does not necessarily imply that the process has a memory $m$
exactly equal to 2. To see this, we compare the obtained value of
$\mathcal{\hat{G}}_{2}$ with the ones calculated under the null hypothesis.
From the original dataset of language $\mathcal{L}$, composed of $R$ tagged
sentences, we compute its second-order transition probabilities by means of
Eq. (10). We then generate $R$ sequences, each of equal length as the original
sentences, from which we build the set $\\{\hat{n}\\}_{1}$. We repeat this
procedure $K$ times. By construction, the sets of observations
$\\{\hat{n}\\}_{1},\ldots,\\{\hat{n}\\}_{K}$, built from the generated groups
of sequences, have the same size as the original set $\\{\hat{n}\\}$ of
language $\mathcal{L}$ to ensure equal amount of data for comparison.
Moreover, these numerical sequences have memory $m=2$ and consequently we
expect the values $\mathcal{\hat{G}}_{2}[\\{\hat{n}\\}_{k}]$ to be close to
$0$, for $k=1,\ldots,K$. Therefore, we can estimate the p-value $p$ as

$\hat{p}=\dfrac{\\#\\{k:\mathcal{\hat{G}}_{2}[\\{\hat{n}\\}_{k}]\geq\mathcal{\hat{G}}_{2}[\\{\hat{n}\\}]\\}}{K}.$
(23)

We apply this method to each of the four languages considered in Sec. II.4,
setting $K=1000$. We obtain $\hat{p}<0.001$ for all cases, which leads us to
reject the hypothesis that the POS sequences have memory exactly equal to $2$.
For comparison, we plot in Fig. C.1 with black color the curves corresponding
to the mean $\bar{\mathcal{G}}_{u}$ and standard deviation $s_{u}$ of the
estimated predictability gain for the generated sequences of memory $2$,
calculated as follows:

$\bar{\mathcal{G}}_{u}=\dfrac{1}{K}\sum_{k=1}^{K}\hat{\mathcal{G}}_{u}[\\{\hat{n}\\}_{k}],$
(24)

and

$s_{u}=\sqrt{\dfrac{1}{K-1}\sum_{k=1}^{K}\left(\hat{\mathcal{G}}_{u}[\\{\hat{n}\\}_{k}]-\bar{\mathcal{G}}_{u}\right)^{2}}.$
(25)

We repeat the analysis with the hypothesis that $m=3$ and obtain similar
results, indicating that the POS sequences for these languages possess a
memory of at least $4$. This is a possible hint of the observed long-range
correlations in texts Altmann  _et al._ (2012). However, we stress that for
the purposes of our work the $m=2$ approximation works rather well.

[figure]style=plain,subcapbesideposition=top

[] []
[] []

Figure C.1: Estimated predictability gain when considering $(u+1)$th-order
instead of $u$th-order transition probabilities in sequences of UPOS for (a)
German, (b) Icelandic, (c) Portuguese and (d) Czech, shown in red. The mean
and standard deviation of the predictability gain estimated on numerically
generated sequences of memory $2$, generated from the estimated transition
probabilities of each language, are represented in black.

##  Appendix D Distance between texts belonging to a single language group

We consider a single linguistic group. For each language in this group we
randomly select tagged sentences from our database until we reach
approximately $10^{4}$ tokens. We iterate this procedure a maximum of $20$
times, without replacement (for a few languages there is not enough data to
perform this procedure $20$ times). Then, for each text portion of each
language inside the given group we calculate their probability distribution of
POS trigrams and compute the pairwise distances among texts with the JS
metric. In Fig. D.1 we present heatmaps corresponding to the distance matrices
obtained for Germanic (panel a) and Slavic (panel b) texts. We can observe
that, in general, the distance between texts of the same language is smaller
when compared to texts coming from distinct languages. This holds even for
languages that are considered to be closely related, such as Croatian and
Serbian, and Belarusian, Russian and Ukrainian.

[figure]style=plain,subcapbesideposition=top

[] []

Figure D.1: Heatmap representation of the pairwise Jensen-Shannon distances
between portions of text of (a) Germanic and (b) Slavic languages.

This analysis demonstrates the reliability of our proposed metric to discern
texts originated from the same language, which is generally a desirable
result.

##  Appendix E Clustermap obtained from the Hellinger distance matrix

Following the same procedure outlined in Sec. III.1.1 of the main text, a
clustermap was built from the Hellinger (H) distance matrix. The metric in
this case is given by

$d_{{}_{H}}(\mathcal{L},\mathcal{L}^{\prime})=\sqrt{\dfrac{1}{2}\sum_{j=0}^{L^{{}^{3}}-1}\left(\sqrt{p_{{}_{\mathcal{L}}}(b_{j})}-\sqrt{p_{{}_{\mathcal{L}^{\prime}}}(b_{j})}\right)^{2}}.$
(26)

Similarly to the JS divergence, $d_{{}_{H}}$ fulfills the requirements of a
distance and lies in the $[0,1]$ interval.

Figure E.1 shows that the constructed representation resembles the one
obtained from the JS matrix. Not only the distributions of the colors match,
but also the ordering of the rows and columns obtained with the hierarchichal
clustering is similar. Only minor differences can be observed in the lower
half of the colormaps but, nevertheless, one can identify the same cluster
formations.

Figure E.1: Clustermap constructed from the Hellinger distance matrix.

##  Appendix F Silhoutte analysis and clusters obtained with k-medoids
algorithm

In order to determine the optimal number of clusters for the k-medoids
analysis from the JS distance matrix, we compute the silhouette score
Rousseeuw (1987) for various cluster numbers. The clusterization is then
performed using the cluster number that yields the maximum silhouette score.
We show the resulting plot in Fig. F.2. As seen, the maximum occurs for $31$
clusters.

Figure F.2: Silhouette score for various cluster numbers in a k-medoids
clustering. The highest score is achieved with a cluster number of $31$.

The clusters obtained using this method are presented in Table F.2.

Cluster | Languages
---|---
0 | Belarusian, Polish, Russia, Ukrainian, Latvian, Lithuanian
1 | Catalan, French, Galician, Italian, Portuguese, Spanish
2 | Buryat, Kazakh, Turkish, Uyghur
3 | Danish, English, Norwegian, Swedish
4 | Hindi, Urdu
5 | Estonian, Finnish, North Sami
6 | Irish, Scottish Gaelic, Welsh
7 | Arabic, Indonesian, Javanese
8 | Bulgarian, Czech, Slovak, Slovenian, Upper Sorbian
9 | Kurmanji, Persian
10 | Armenian, Wester Armenian
11 | Cantonese
12 | Gheg
13 | Japanese
14 | Thai
15 | Chinese
16 | Tamil
17 | Manx
18 | Xibe
19 | Korean
20 | Afrikaans, Dutch, German
21 | Faroese, Icelandic, Romanian
22 | Erzya, Komi Zyrian
23 | Hungarian
24 | Basque
25 | Hebrew, Maltese
26 | Croatian, Serbian
27 | Vietnamese
28 | Breton
29 | Pomak
30 | Greek
Table F.2: Clusters formed with k-medoids algorithm from the Jensen-Shannon
distance matrix.