#  Dependent measures in independent theories

Karim Khanaki
Arak University of Technology Partially supported by IPM grant 1400030118

###### Abstract

We introduce the notion of dependence, as a property of a Keisler measure, and
generalize several results of [HPS13] on generically stable measures (in $NIP$
theories) to arbitrary theories. Among other things, we show that this notion
is very natural and fundamental for several reasons: (i) all measures in $NIP$
theories are dependent, (ii) all types and all $fim$ measures in any theory
are dependent, and (iii) as a crucial result in measure theory, the Glivenko-
Cantelli class of functions (formulas) is characterized by dependent measures.

##  1 Introduction

In [HPS13], Hrushovski, Pillay, and Simon proved the following crucial
theorem:

Theorem111In this article, when we refer to the Hrushovski-Pillay-Simon
theorem, we mean this theorem.: (Assuming $NIP$) Let $\mu$ be a global Keisler
measure which is invariant over a small set. The following are equivalent:
(i) $\mu$ is both definable over and finitely satisfiable in a small model.
(ii) $\mu$ commutes with itself:
$\mu_{x}\otimes\mu_{y}=\mu_{y}\otimes\mu_{x}$.
(iii) $\mu$ is finitely approximated.
(iv) $\mu$ is $fim$.

It is known [CGH21] that, without assuming $NIP$, (iv) is strictly stronger
than (iii), and (iii) is strictly stronger than (i), and (ii) need not have
any special properties.222See [CGH21] Propositions 8.12, 7.1, and Example 5.3,
respectively. This highlights the fact that to generalize the Hrushovski-
Pillay-Simon theorem (to arbitrary theories) requires a weakened version of
the $NIP$.

This paper aims to generalize this theorem to arbitrary theories. Indeed, we
introduce the notion dependent measure and prove that, in any theory, every
dependent measure for which (i) above holds is symmetric, i.e. (ii) holds.
Then we show that, assuming a local version of $NIP$, for any symmetric
measure the conditions (i)—(iv) are equivalent. We will show that the notion
‘dependent measure’ and the local version of $NIP$ mentioned above are much
weaker than $NIP$, in several ways. We will argue that, from several
perspectives, these conditions are the minimum assumptions that lead to these
results. It is worth noting that some of the arguments presented in this
article are inherently similar to [HPS13], although the key notion ‘dependent
measure’ allows us to use the facts in measure theory and make a connection
between them and model theory. Surprisingly, in this case, model theory and
measure theory have organic relationships, and the results of one domain
corresponds to the results of another domain. We believe that the new notion
is valuable in itself and this approach can have more applications in future
work.

Let us give our motivation and point of view on the importance of this work.
It is very natural to generalize the results in the $NIP$ context to arbitrary
theories as this approach has already been pursued by generalizations of
stable to simple and $NIP$ theories. On the other hand, this study will
clarify why the arguments work in $NIP$ theories and how they can be
generalized to the outside this context. Finally, this study identifies the
deep links between two different areas of mathematics, namely model theory and
measure theory, and their importance in applications as the results of [HPS13]
are evidence of the usefulness of links between different domains, in the
latter case probability theory and model theory. Apart from these, studying
‘measures’ as mathematical objects, which are the natural generalization of
types, is interesting in itself and important in applications.

To simplify to read through the paper, we list some more important
observations and results: Theorems 4.1, 4.8, 5.3, 5.5, and Proposition 5.4,
and Corollary 4.11.

This paper is organized as follows: In the next section we review some basic
notions from measure theory. In Section 3 we introduce the notion dependent
measure and give some basic properties of dependent measures. In Section 4 we
generalize some results of [HPS13] on the Morley products of measures and
symmetric measures to arbitrary theories. In Section 5 we study $fim$ measures
and generalize the Hrushovski-Pillay-Simon theorem using a local version of
$NIP$. We also refine a result of [CGH21] and give some new ideas for the
future work.

##  2 Preliminaries from measure theory

In this section we give definitions of measure theory with which we shall
concerned, especially the notion of stable set of functions (or
$\mu$-stability) and its properties.

Let $X$ be a compact Hausdorff space. The space of continuous real-valued
functions on $X$ is denoted here by $C(X)$. The smallest $\sigma$-algebra
containing the open sets is called the class of Borel sets. By a Borel measure
on $X$ we mean a finite measure defined for all Borel sets. A Radon measure on
$X$ is a Borel measure which is regular. Recall that a measure is complete if
for any null measurable set $E$ every $F\subseteq E$ is measurable. It is
known that every Borel measure on a compact space has a unique extension to a
complete measure. In this paper, we always assume that every Radon measure is
complete.

In the following, given a measure $\mu$ and $k\geqslant 1$, the symbol
$\mu^{k}$ stands for $k$-fold product of $\mu$ and $\mu^{*}$ stands for the
outer measure of $\mu$.

The following fundamental notion has been invented by David H. Fremlin, namely
$\mu$-stability, which in the model theory context we will call dependent
measure. From a logical point of view, this notion was first studied in
[Kha16] in the framework of integral logic.

######  Definition 2.1 ($\mu$-stability).

Let $A\subseteq C(X)$ be a pointwise bounded family of real-valued continuous
functions on $X$. Suppose that $\mu$ is a Borel probability measure on $X$. We
say that $A$ is $\mu$-stable, if there is no measurable subset $E\subseteq X$
with $\mu(E)>0$ and $s<r$ such that for each $k=\\{1,\ldots k\\}$

$\mu^{k}\Big{\\{}\overline{w}\in E^{k}:\ \forall I\subseteq k\ \exists f\in A\
\bigwedge_{i\in I}f(w_{i})\leqslant s\wedge\bigwedge_{i\notin
I}f(w_{i})\geqslant r\Big{\\}}=(\mu E)^{k}.$

######  Remark 2.2.

(i) The notion $\mu$-stable is an adaptation of [Fre06, 465B]. Indeed, by
Proposition 4 of [T87], it is easy to check the equivalence. For this, notice
that every function $f\in A$ is continuous on $X$ and so the left set in the
equation above is measurable. This means that $(M)$ property of Proposition 4
of [T87] holds.
(ii) A measure $\mu$ is dependent iff its completion $\bar{\mu}$ is dependent.
Indeed, recall that the product measures of $\mu,\bar{\mu}$ are the same. (See
Proposition 465C(i) of [Fre06]–Version of 26.8.13.)

The following are important results connecting the notion of ‘stable’ set of
functions.

######  Fact 2.3 ([Fre06], Pro. 465D(b)).

Let $X$ be a compact Hausdorff space, $\mu$ a Radon probability measure on
$X$, and $A\subseteq C(X)$. If $A$ is $\mu$-stable, then every function in the
pointwise closure of $A$ is $\mu$-measurable.

As mentioned above, in this paper, we assume that all Radon measures are
complete. Note that the completeness of $\mu$ is absolutely necessary in Fact
2.3.

Recall that the convex hull of $A\subseteq C(X)$, denoted by $\Gamma(A)$ or
$\text{conv}(A)$, is the set of all convex combinations of functions in $A$,
that is, the set of functions of the form $\sum_{1}^{k}r_{i}\cdot f_{i}$ where
$k\in\mathbb{N}$,$f_{i}\in A$, $r_{i}\in{\mathbb{R}}^{+}$ and
$\sum_{1}^{k}r_{i}=1$.

The following theorem is the fundamental result on stable sets of functions.
In fact, it asserts that a set of function is stable iff it is a
Glivenko–Cantelli class iff its convex hull is a Glivenko–Cantelli class (cf.
[Kha21]).

######  Fact 2.4 ([Fre06]).

Let $X$ be a compact Hausdorff space, $\mu$ a Radon probability measure on
$X$, and $A\subseteq C(X)$ uniformly bounded. Then the following are
equivalent:

(i) $A$ is $\mu$-stable.

(ii) The convex hull of $A$ is $\mu$-stable.

(iii) $\lim_{k\to\infty}\sup_{f\in A}|\frac{1}{k}\sum_{1}^{k}f(w_{i})-\int
f|=0$ for almost all $w\in X^{\mathbb{N}}$.

(Here, $w=(w_{1},w_{2},\ldots)\in X^{\mathbb{N}}$ and the measure on
$X^{\mathbb{N}}$ is the usual product measure.)

Explanation. The direction (i) $\implies$(ii) is Theorem 465N(a) of [Fre06].
The converse is evident. (See also Proposition 465C(a)(i) of [Fre06].) The
equivalence (i) $\iff$(iii) is the equivalence (i) $\iff$(ii) of Theorem 465M
of [Fre06]. Again, we emphasize that the completeness of $\mu$ is necessary in
the direction (i) $\implies$(iii).

The last fact shows that, on stable set of functions, the topology of
pointwise convergence is stronger than the topology of convergence in measure.

######  Fact 2.5 ([Fre06], Thm. 465G).

Let $X$ be a compact Hausdorff space, $\mu$ a Radon probability measure on
$X$, and $A\subseteq C(X)$ a $\mu$-stable set. Let $(f_{i})$ be a net in $A$
such that $f_{i}\to f$ in the topology of pointwise convergence. Then
$\int|f_{i}-f|\to 0$.

##  3 Dependent Keisler measures

In this section we introduce the notion of dependent measure (Definition 3.2)
and give some results on the Morley products of measures in arbitrary
theories.

The model theory notation is standard, and a text such as [S15] will be
sufficient background. We fix a first order language $L$, a complete
$L$-theory $T$ (not necessarily $NIP$), an $L$-formula $\phi(x,y)$, and a
subset $A$ of the monster model of $T$. The monster model is denoted by $\cal
U$. We let $\phi^{*}(y,x)=\phi(x,y)$. We define $p=tp_{\phi}(a/A)$ as the
function $\phi(p,y):A\to\\{0,1\\}$ by $b\mapsto\phi(a,b)$. This function is
called a complete $\phi$-types over $A$. The set of all complete $\phi$-types
over $A$ is denoted by $S_{\phi}(A)$. We equip $S_{\phi}(A)$ with the least
topology in which all functions $p\mapsto\phi(p,b)$ (for $b\in A$) are
continuous. It is compact and Hausdorff, and is totally disconnected. Let
$X=S_{\phi^{*}}(A)$ be the space of complete $\phi^{*}$-types on $A$. Note
that the functions $q\mapsto\phi(a,q)$ (for $a\in A$) are continuous, and as
$\phi$ is fixed we can identify this set of functions with $A$. So, $A$ is a
subset of all bounded continuous functions on $X$, denoted by $A\subseteq
C(X)$.

A Keisler measure over $A$ in the variable $x$ is a finitely additive
probability measure on the Boolean algebra of $A$-definable sets in the
variable $x$, denoted by $L_{x}(A)$. Every Keisler measure over $A$ can be
represented by a regular Borel probability measure on $S_{x}(A)$, the space of
types over $A$ in the variable $x$. A measure over $\cal U$ is called a global
Keisler measure. The set of all measures over $A$ in the variable $x$ is
denoted by ${\mathfrak{M}}_{x}(A)$ or ${\mathfrak{M}}(A)$. We will sometimes
write $\mu$ as $\mu_{x}$ or $\mu(x)$ to emphasize that $\mu$ is a measure on
the variable $x$.

For a formula $\phi(x,y)$, a Keisler $\phi$-measure over $A$ in the variable
$x$ is a finitely additive probability measure on the Boolean algebra of
$\phi$-definable sets over $A$ in the variable $x$, denoted by $L_{\phi}(A)$.
Recall that a $\phi$-definable sets over $A$ is a Boolean combination of the
instances $\phi(x,b),b\in A$. The set of all $\phi$-measures over $A$ in the
variable $x$ is denoted by ${\mathfrak{M}}_{\phi}(A)$.

Given a model $M$, a $L(M)$-formula $\theta(x)$, and types
$p_{1}(x),\ldots,p_{n}(x)$ over $M$, the average measure of them (for
$\theta(x)$), denoted by $\text{Av}(p_{1},\ldots,p_{n})$, is defined as
follows:

$\text{Av}(p_{1},\ldots,p_{n};\theta(x)):=\frac{\big{|}\\{i:~{}\theta(x)\in
p_{i},i\leq n\\}\big{|}}{n}.$

We first revisit a useful dictionary of [Kha21] that is used in the rest of
the paper and can also be used in future work. For the definition of finitely
satisfiable (definable, Borel definable) measures see Definition 7.16 of
[S15].

######  Fact 3.1.

Let $T$ be a complete theory, $M$ a model of $T$ and $\phi(x,y)$ a formula.
(i)(Pillay) There is a correspondence between global $M$-finitely satisfiable
$\phi$-types $p(x)$ and the functions in the pointwise closure of all
functions $\phi(a,y):S_{\phi^{*}}(M)\to\\{0,1\\}$ for $a\in M$, where
$\phi(a,q)=1$ if and only if $\phi(a,y)\in q$.
(ii) The map $p\mapsto\delta_{p}$ is a correspondence between global
$\phi$-types $p(x)$ and Dirac measures $\delta_{p}(x)$ on $S_{\phi}({\cal
U})$, where $\delta_{p}(A)=1$ if $p\in A$, and $=0$ in otherwise. Moreover,
$p(x)$ is finitely satisfiable in $M$ iff $\delta_{p}(x)$ is finitely
satisfiable in $M$.
(iii) There is a correspondence between global $\phi$-measures $\mu(x)$ and
regular Borel probability measures on $S_{\phi}({\cal U})$. Moreover, a global
$\phi$-measures is finitely satisfiable in $M$ iff its corresponding regular
Borel probability measure is finitely satisfiale in $M$.
(iv) The closed convex hull of Dirac measures $\delta(x)$ on $S_{\phi}({\cal
U})$ is exactly all regular Borel probability measures $\mu(x)$ on
$S_{\phi}({\cal U})$. Moreover, the closed convex hull of Dirac measures on
$S_{\phi}({\cal U})$ which are finitely satisfiable in $M$ is exactly all
regular Borel probability measures $\mu(x)$ on $S_{\phi}({\cal U})$ which are
finitely satisfiable in $M$.
(v) There is a correspondence between global $M$-finitely satisfiable
$\phi$-measures $\mu(x)$ and the functions in the pointwise closure of all
functions of the form $\frac{1}{n}\sum_{1}^{n}\theta(a_{i},y)$ on
$S_{\theta^{*}}(M)$, where $\theta$ is a $\phi^{*}$-formula,333Recall that a
$\phi$-formula is a Boolean combination of instances of $\phi(x,b)$, $b\in M$.
$a_{i}\in M$, and $\theta(a_{i},q)=1$ if and only if $\theta(a_{i},y)\in q$.

The above fact actually shows the ideology that we follow in this article.
That is, the finitely satisfied (and invariant) measures in this paper are
functions on specific topological spaces. The reader of this article can
become more familiar with our approach as well as its historical trend by
reading [Kha20], [Kha17], [KP18] and [Kha21].

Convention. Recall that every regular Borel probability measure $\mu$ has a
unique completion $\bar{\mu}$. In this paper, without loose of generality we
can assume that every measure is complete. That is, $\mu=\bar{\mu}$. (The
crucial notion of this paper (i.e. Definition 3.2) is neutral to completion.
Cf. Remark 3.3(ii) below.)

Let $\mu\in{\mathfrak{M}}_{x}({\cal U})$ and $\mu_{\phi}$ its restriction to
$S_{\phi}(M)$ (equivalently, its restriction to Boolean algebra of
$\phi$-formulas). As the restriction map $r_{\phi}:S_{x}(M)\to S_{\phi}(M)$ is
a quotient map, it is easy to verify that
$\mu_{\phi}(X)=\mu(r_{\phi}^{-1}(X))$ for any Borel subset $X\subseteq
S_{\phi}(M)$. (See Remark 1.2 in [CGH21].) For $A\subseteq B$ and
$\mu\in{\mathfrak{M}}_{x}(B)$, $\mu|_{A}\in{\mathfrak{M}}_{x}(A)$ is the
restriction of $\mu$ by the quotient map $r:S_{x}(B)\to S_{x}(A)$. The
restriction of $\mu|_{A}$ to $S_{\phi}(A)$ is denoted by $\mu_{\phi,A}$.

The crucial notion of the paper is as follows. (Compare Definition 2.1).

######  Definition 3.2 (Dependent Measures).

Let $T$ be a complete theory, $M$ a model, and $\mu_{x}\in{\mathfrak{M}}(M)$.
(i) Suppose that $A\subseteq B\subseteq M$. We say that $\mu$ is dependent
over $B$ and in $A$, if for any formula $\phi(x,y)$ there is no $E\subseteq
S_{\phi}(B)$ measurable, $\mu_{\phi,B}(E)>0$ (where
$\mu_{\phi,B}=(\mu|_{B})_{\phi}$) such that for each $k$,
$(\mu_{\phi,B}^{k})D_{k}(A,B,E,\phi)=(\mu_{\phi,B}E)^{k}$ where

$\displaystyle D_{k}(A,B,E,\phi)=\big{\\{}\overline{p}\in E^{k}:\ \forall
I\subseteq k\ \exists b\in A\ \bigwedge_{i\in
I}\phi(p_{i},b)=0\wedge\bigwedge_{i\notin I}\phi(p_{i},b)=1\big{\\}}.$

(Recall that $\phi(p_{i},b)=1$ if $\phi(x,b)\in p_{i}$ and $\phi(p_{i},b)=0$
in otherwise.)
(ii) We say that $\mu$ is dependent, if for every $A\subseteq B\subseteq M$,
$\mu$ is dependent over $B$ and in $A$.

######  Remark 3.3.

(i) The notion dependent measure is an adaptation of Definition 2.1 above to
the model theory context. Indeed, note that every function $\phi(x,b)$ is
continuous on $S_{\phi}(B)$ and so $D_{k}(A,B,E,\phi)$ is
$\mu_{\phi,B}^{k}$-measurable. This means that
$(\mu_{\phi,B}^{k})^{*}(D_{k}(A,B,E,\phi))=\mu_{\phi,B}^{k}(D_{k}(A,B,E,\phi))$
and $(M)$ property of Proposition 4 of [T87] holds.
(ii) Recall from Remark 2.2(ii) that a Keisler measure $\mu$ is dependent iff
its completion $\bar{\mu}$ is dependent.
(iii) For $A\subseteq B$, as the restriction map $r:S_{x}(B)\to S_{x}(A)$ is a
quotient map, it is easy to verify that $\mu|_{A}(X)=\mu|_{B}(r^{-1}(X))$ for
any Borel subset $X\subseteq S_{x}(A)$. (See Remark 1.2 in [CGH21].) Now, by
Proposition 465C(d) of [Fre06] version of 26.8.13, if $\mu$ is dependent over
$A$ and in $A$, then it is dependent over $B$ and in $A$.
(iv) As the restriction map $r_{\phi}:S_{x}(B)\to S_{\phi}(B)$ is a quotient
map, it is easy to verify that $\mu_{\phi}(E)=\mu(r_{\phi}^{-1}(E))$ for any
Borel subset $E\subseteq S_{\phi}(B)$. (See Remark 1.2 in [CGH21].) Therefore,
by Proposition 465C(d) of [Fre06] again, if $\mu$ is dependent over $B$ and in
$A$, then whenever $E\subseteq S_{x}(B)$ is measurable, $\mu|_{B}(E)>0$, there
is some $k\geqslant 1$ such that $(\mu|_{B})^{k}D_{k}(A,B,E)<(\mu|_{B}E)^{k}$
where $D_{k}(A,B,E)=\big{\\{}\overline{p}\in E^{k}:\ \forall I\subseteq k\
\exists b\in A\ \bigwedge_{i\in I}\phi(p_{i},b)=0\wedge\bigwedge_{i\notin
I}\phi(p_{i},b)=1\big{\\}}$.

The following is an important property of the notion dependent measure.

######  Proposition 3.4.

Let $T$ be a complete theory, $M$ a model, and $\mu\in{\mathfrak{M}}_{x}(M)$.
If $\mu$ is dependent, then for any $A\subseteq M$ and any $L(A)$-formula
$\phi(x,y)$, every function in the pointwise closure of
$\\{\phi(x,b):S_{\phi}(A)\to\\{0,1\\}~{}|b\in A\\}$ is $\mu$-measurable.

###### Proof.

This is a consequence of Proposition 2.3. Indeed, recall that $\mu$ is
dependent iff its completion $\bar{\mu}$ is. Now, by the above convention,
$\mu_{\phi,A}$ (is complete and) satisfies in the assumptions of Proposition
2.3. ∎

Recall from [KP18, Def. 1.1] that a formula $\phi(x,y)$ has $NIP$ in a model
$M$ if there is no countably infinite sequence $(a_{i})\in M$ such that for
all finite disjoint subsets $I,J\subseteq\mathbb{N}$, $M\models\exists
y(\bigwedge_{i\in I}\phi(a_{i},y)\wedge\bigwedge_{i\in J}\neg\phi(a_{i},y))$.
The formula $\phi$ has $NIP$ for the theory $T$ iff it has $NIP$ in every
model $M$ of $T$ iff it has $NIP$ in some model $M$ of $T$ in which all types
over the empty set in countably many variables are realised.

######  Proposition 3.5.

(i) For any theory, every type is dependent.
(ii) A theory $T$ is $NIP$ iff in any model $M$ of $T$, every Keisler measure
over $M$ is dependent iff every Keisler measure over some model $M$ of $T$ in
which all types over the empty set in countably many variables are realised.
(iii) For any theory $T$ and any model $M$ of $T$, if every Keisler measure
over $M$ is dependent, then every formula has $NIP$ in $M$.

###### Proof.

(i) is evident, by definition. (See also Theorem 5 of [T87].)

(ii): It is easy to see that if $T$ is $NIP$, then for some $k$,
$D_{k}(A,B,E,\phi)=\emptyset$. (Cf. Definition 3.2.) For the converse, by
Proposition 3.4, for any model $M$ and any formula $\phi(x,y)$, every function
in the pointwise closure of $\\{\phi(x,b):S_{\phi}(M)\to\\{0,1\\}~{}|b\in
M\\}$ is measurable with respect to any Keisler measure over $M$ (in the
variable $x$). This means that every function in the pointwise closure is
universally Radon measurable. (Recall that every measure is complete and
regular and therefore Radon.) By the equivalence (iv) $\iff$ (vi) of Theorem
2F of [BFT78], it is easy to see that $\phi$ has $NIP$ in any model of $T$. By
Remark 2.1 of [KP18], this means that $\phi$ has $NIP$ for the theory $T$.

(iii): By Proposition 3.4, for any formula $\phi(x,y)$, every function in the
pointwise closure of $\\{\phi(x,b):S_{\phi}(M)\to\\{0,1\\}~{}|b\in M\\}$ is
measurable with respect to any Keisler measure over $M$. By the equivalence
(iv) $\iff$ (vi) of Theorem 2F of [BFT78], $\phi(x,y)$ has $NIP$ in $M$. ∎

######  Example 3.6.

(i) By Proposition 3.5 above, all types in any theory, and all measures in
$NIP$ theories are dependent.
(ii) We say that a measure $\mu$ is purely atomic if there are Dirac measures
$(\delta_{n}:n<\omega)$ such that $\mu=\sum_{1}^{\infty}r_{n}.\delta_{n}$
where $r_{n}\in[0,1]$ and $\sum_{1}^{\infty}r_{n}=1$. By definition, it is
easy to verify that any purely atomic measure is dependent (cf. also Theorem 5
of [T87]). In [CG20], a measure $\mu$ is called trivial, if (1) it is purely
atomic (i.e. $\mu=\sum_{1}^{\infty}r_{n}.\delta_{n}$) , and (2) any
$\delta_{n}$ is realized in $\cal U$, i.e. $\delta_{n}=tp(a_{n}/{\cal U})$ for
some $a_{n}\in\cal U$. It is shown [CG20, Theorem 4.9] that, in the theories
of the random graph and the random bipartite graph, every definable and
finitely satisfiable measure is trivial. This means that such measures are
dependent. (Similarly, all definable and finitely satisfiable measures in the
theories in [CG20, Corollary 4.10] are dependent.)
(iii) Furthermore, in Proposition 5.4 below, we show that any $fim$ measure
(in any theory) is dependent.
In Example 4.9 below we will present a measure that is not dependent.

The following result allows us to create new dependent measures from the
previous ones.

######  Proposition 3.7.

The set ${\mathfrak{M}}_{x}^{d}({\cal U})$ of all global dependent measures
(in the variable $x$) is convex.

###### Proof.

Let $\mu,\nu\in{\mathfrak{M}}_{x}^{d}({\cal U})$ and $r\in[0,1]$. Set
$\lambda=r\mu+(1-r)\nu$. Then, for formula $\phi(x,y)$, $b\in{\cal U}$ and
${\bar{p}}=(p_{1},\ldots,p_{k})$, we have

$\displaystyle|Av({\bar{p}};\phi(x,b))-\lambda(\phi(x,b))|$ $\displaystyle\leq
r|Av({\bar{p}};\phi(x,b))-\mu(\phi(x,b))|+$ $\displaystyle\ \ \ \
(1-r)|Av({\bar{p}};\phi(x,b))-\nu(\phi(x,b))|.$

We apply Fact 2.4 for $\mu,\nu$ and conclude that
$\lim_{k\to\infty}\sup_{b\in\cal
U}|Av({\bar{p}};\phi(x,b))-\lambda(\phi(x,b))|=0$ for almost all
$(p_{i})\in(S_{\phi}({\cal U}))^{\mathbb{N}}$. This means that $\lambda$ is
dependent. (Note that this argument works for all $A\subseteq B\subseteq\cal
U$ in Definition 3.2.) ∎

##  4 Dependence and symmetry

In this section we generalize some results of [HPS13] on the Morley products
of measures and symmetric measures. The following is the fundamental property
of the notion of dependent measure.

######  Theorem 4.1.

Let $T$ be a complete theory, $M$ a model, and $\mu_{x}\in{\mathfrak{M}}({\cal
U})$. If $\mu|_{M}$ is dependent, then for any $\mu|_{M}$-measurable subsets
$X_{1},\ldots,X_{m}\subseteq S_{x}(M)$ and $\epsilon>0$, there are
$n\in{\mathbb{N}}$ and $E\subseteq(S_{x}({\cal U}))^{n}$, with
$(\mu^{n})^{*}E\geq 1-\epsilon$, such that for every $b\in M$ and $k\leq m$,

$\big{|}\mu(\phi(x,b)\cap X_{k})-Av(p_{1}|_{M},\ldots,p_{n}|_{M};\phi(x,b)\cap
X_{k})\big{|}\leq\epsilon,\ \ \ (*)$

for all $(p_{1},\ldots,p_{n})\in E$. (Here $p_{i}|_{M}$ is the restriction of
$p_{i}$ to $S_{x}(M)$.)

###### Proof.

First note that, by the definition and Remark 3.3(iv), we can use $\mu|_{M}$
instead of $\mu_{\phi,M}$ (in Definition 3.2). As $\mu|_{M}$ is dependent
(equivalently the set $\\{\phi(x,b):S_{x}(M)\to\\{0,1\\}~{}|b\in M\\}$ is
$\mu|_{M}$-stable in the sense of Definition 2.1), by Proposition 465C(a)(i)
and (b)(ii) of [Fre06]–Version of 26.8.13, the set
$\bigcup_{k=1}^{m}\\{\phi(x,b)\times\chi_{X_{k}}:b\in M\\}$ is
$\mu|_{M}$-stable, where $\chi_{X_{k}}$ is the characteristic function of
$X_{k}$. Therefore, by Fact 2.4, we have

$\sup_{b\in
M}|\frac{1}{n}\sum_{1}^{n}\phi(p_{i},b)\times\chi_{X_{k}}-\mu(\phi(x,b)\times\chi_{X_{k}})|\to
0$

as $n\to\infty$ for all $k\leq m$ and for almost every $(p_{i})\in
S_{x}(M)^{\mathbb{N}}$ with the product measure $(\mu|_{M})^{\mathbb{N}}$.
Now, it is easy to verify that the claim holds. Indeed, one can see directly
(or using Theorem 11-1-1(c) of [T84]) that there are $n\in{\mathbb{N}}$ and
$F\subseteq(S_{x}(M))^{n}$, with $(\mu|_{M}^{n})^{*}F\geq 1-\epsilon$, such
that for every $b\in M$ and $k\leq m$, $\big{|}\mu(\phi(x,b)\cap
X_{k})-Av(p_{1}^{\prime},\ldots,p_{n}^{\prime};\phi(x,b)\cap
X_{k})\big{|}\leq\epsilon$ for all $(p_{i}^{\prime})\in F$. Finally, use
Remark 3.3(iii) above and find the desired set $E\subseteq(S_{x}({\cal
U}))^{n}$ such that $(*)$ holds. (Here, $p_{i}^{\prime}=p_{i}|_{M}$ for some
$p_{i}\in S_{x}({\cal U})$.) ∎

In the following, for $\mu\in{\mathfrak{M}}({\cal U})$, the support of $\mu$
is denoted by $S(\mu)$. (Cf. [S15, p. 99].)

######  Corollary 4.2.

Let $T$ be a complete theory, $M$ a model, and $\mu\in{\mathfrak{M}}_{x}({\cal
U})$. If $\mu|_{M}$ is dependent, then for any $\mu|_{M}$-measurable subsets
$X_{1},\ldots,X_{m}\subseteq S_{x}(M)$ and $\epsilon>0$, there are
$p_{1}|_{M},\ldots,p_{n}|_{M}\in S_{x}(M)$ such that for every $b\in M$ and
$k\leq m$,

$\big{|}\mu(\phi(x,b)\cap X_{k})-Av(p_{1}|_{M},\ldots,p_{n}|_{M};\phi(x,b)\cap
X_{k})\big{|}\leq\epsilon.$

Furthermore, we can assume that $p_{i}\in S(\mu)$ for all $i$.

###### Proof.

Immediate, by Theorem 4.1. (Recall from [G20, Pro. 2.10] that $\mu(S(\mu))=1$.
This assures us that we can assume that $p_{i}\in S(\mu)$ for all $i$.) ∎

The following result allows us to define the Morley product of a finitely
satisfiable measure and a dependent measure.

######  Proposition 4.3.

Let $\mu_{x}$ be global $M$-finitely satisfied measures, $\lambda_{y}$ a
global dependent measure and $\phi(x,y;b)$ an $L({\cal U})$-formula. Let
$N\supseteq Mb$ be a model and define the function
$f:S_{\phi^{*}}(N)\to[0,1]$,444Recall that $\phi^{*}(y,x;b)=\phi(x,y;b)$. by
$q\mapsto\mu(\phi(x,d;b))$ for some (any) $d\models q$. Then $f$ is
$\lambda_{y}|_{N}$-measurable.

###### Proof.

As $\mu$ is $M$-finitely satisfied, by Fact 3.1(v), $f$ is in the closure of
the convex hull of $\\{\phi(a,y;b):S_{\phi^{*}}(N)\to\\{0,1\\}~{}|a\in M\\}$.
Now, as $\lambda_{y}$ is dependent, by Facts 2.3 and 2.4 above, $f$ is
$\lambda_{y}|_{N}$-measurable. (Indeed, recall that $\lambda_{y}$ is a
complete measure, with the above convention.) ∎

######  Definition 4.4.

By the assumptions of Proposition 4.3 we define the Morley product measure
$\mu(x)\otimes\lambda(y)$ as follows:

$\mu(x)\otimes\lambda(y)(\phi(x,y;b))=\int_{S_{\phi^{*}}(N)}f~{}d\lambda|_{N}.$

It is easy to verify that the definition does not depend on the choice of $N$.
We will sometimes write $f$ as $f_{\mu}^{\phi}$ (or $f_{\mu,N}^{\phi}$) to
emphasize that it is related to $\mu$, $\phi$ (and $N$) as above.

The following is a generalization of Lemma 7.1 of [S15], although the proof is
essentially the same, using the previous observations.

######  Lemma 4.5.

Let $\mu_{x},\lambda_{y}$ be global dependent measures such that $\mu_{x}$ is
$M$-finitely satisfied (or Borel definable over $M$) and $\lambda_{y}$ is
$M$-finitely satisfied. If $\mu_{x}\otimes q_{y}=q_{y}\otimes\mu_{x}$ for any
$q_{y}\in S_{y}({\cal U})$ in the support of $\lambda_{y}$, then
$\mu_{x}\otimes\lambda_{y}=\lambda_{y}\otimes\mu_{x}$.

###### Proof.

Let $\phi(x,y;b)\in L({\cal U})$ and $N\supseteq Mb$ a model. Let
$f=f_{\mu,N}^{\phi}$ be as above. As $\mu_{x}$ is $M$-finitely satisfied and
$\lambda_{y}$ is dependent (or just $\mu_{x}$ is Borel definable over $M$),
the Morley product $\mu\otimes\lambda$ is well-defined. (Cf. Definition 4.4
and Proposition 4.3.) Fix $\epsilon>0$. Let $\sum_{1}^{n}r_{i}.\chi_{X_{i}}$
be a simple $\lambda|_{N}$-measurable functions such that
$|f(q)-\sum_{1}^{n}r_{i}.\chi_{X_{i}}(q)|<\epsilon$ for all $q\in
S_{\phi^{*}}(N)$. (That is, $X_{1},\ldots,X_{n}\in S_{\phi^{*}}(N)$ are
$\lambda|_{N}$-measurable, $\chi_{X_{i}}$ is the characteristic function of
$X_{i}$, and $r_{i}\in[0,1]$ for $i\leq n$.) By Corollary 4.2, there are
$q_{1},\ldots,q_{m}\in S(\lambda)$ such that if
${\tilde{\lambda}}=\frac{1}{m}\sum q_{i}$ then

(1) $|{\tilde{\lambda}}(X_{i})-\lambda(X_{i})|<\epsilon$ for all $i\leq n$,
and

(2) $|{\tilde{\lambda}}(\phi(a,y;b))-\lambda(\phi(a,y;b))|<\epsilon$ for all
$a\in\cal U$.

(Here, we let $r^{-1}(X_{i}):=X_{i}$ again, where $r:S_{\phi^{*}}({\cal U})\to
S_{\phi^{*}}(N)$ is the restriction map.) Note that, as the $q_{i}$’s are
types, the product $\mu\otimes\tilde{\lambda}$ is well-defined. The product
measure ${\tilde{\lambda}}\otimes\mu$ is well-defined since $\mu$ is dependent
and the $q_{i}$’s are $M$-finitely satisfied. As $\mu$ commutes with
$\tilde{\lambda}$ and $\epsilon$ is arbitrary, by the conditions (1),(2), it
is easy to see that
$\mu_{x}\otimes\lambda_{y}(\phi(x,y;b))=\lambda_{y}\otimes\mu_{x}(\phi(x,y;b))$.
∎

The above argument can be further visualized in the language of analysis.
Given $\epsilon>0$, and $r,s\in\mathbb{R}$, we write $r\approx_{\epsilon}s$ to
denote $|r-s|<\epsilon$. With the above assumptions, the argument of Lemma 4.5
is as follows:

$\displaystyle\mu\otimes\lambda(\phi(x,y;b))$ $\displaystyle=\int
f_{\mu}^{\phi}d\lambda\approx_{\epsilon}\int(\sum
r_{i}\cdot\chi_{X_{i}})d\lambda=\sum r_{i}\cdot\lambda(X_{i})$
$\displaystyle\approx_{\epsilon}\sum
r_{i}\cdot\tilde{\lambda}(X_{i})=\int(\sum
r_{i}\cdot\chi_{X_{i}})d\tilde{\lambda}\approx_{\epsilon}\int
f_{\mu}^{\phi}d\tilde{\lambda}$
$\displaystyle=\mu\otimes\tilde{\lambda}(\phi(x,y;b))=\tilde{\lambda}\otimes\mu(\phi(x,y;b))=\int
f_{\tilde{\lambda}}^{\phi^{*}}d\mu$ $\displaystyle\approx_{\epsilon}\int
f_{\lambda}^{\phi^{*}}d\mu=\lambda\otimes\mu(\phi(x,y;b)).$

######  Remark 4.6.

Assuming that $\mu$ and $\lambda$ are dependent, and using Lemma 4.5, one can
give a generalization of [HPS13, Lemma 3.1]. That is, if
$\mu\in{\mathfrak{M}}_{x}({\cal U})$ is definable over $M$ and dependent, and
$\lambda\in{\mathfrak{M}}_{y}({\cal U})$ is $M$-finitely satisfied and
dependent, then $\mu_{x}\otimes\lambda_{y}=\lambda_{y}\otimes\mu_{x}$. (See
also [S15, Pro. 7.22].) Nevertheless, we prove something stronger (cf. Theorem
4.8). In fact, the dependence of $\lambda$ is unnecessary. Although, for the
sake of completeness we give a model theoretic proof of the weaker result in
Appendix.

First we need a lemma that is interesting in itself and is a generalization of
[ChG21, Pro. 6.3].

######  Lemma 4.7.

Let $\mu\in{\mathfrak{M}}_{x}({\cal U})$, $\lambda\in{\mathfrak{M}}_{y}({\cal
U})$ such that $\mu$ is dependent and $\lambda$ is finitely satisfiable in a
small model $M$. Suppose that $(\bar{a}_{i})_{i}$ is a net in
$(M^{y})^{<\omega}$ such that $Av(\bar{a}_{i})\to\lambda$, then:

$Av(\bar{a}_{i})\otimes\lambda\to\lambda\otimes\mu.$

###### Proof.

Let $\phi(x,y)$ be a formula, and set $\lambda_{i}=Av(\bar{a}_{i})$. By Fact
2.4, as $\mu$ is dependent, the convex hull of the set
$\\{f_{tp(a)}^{\phi^{*}}:a\in M\\}$ is $\mu$-stable. Therefore,

$\lim_{i}[\lambda_{i}\otimes\mu](\phi(x,y))=\lim_{i}\int
f_{\lambda_{i}}^{\phi^{*}}d\mu\stackrel{{\scriptstyle(*)}}{{=}}\int(\lim_{i}f_{\lambda_{i}}^{\phi^{*}})d\mu=\int
f_{\lambda}^{\phi^{*}}d\mu=\lambda\otimes\mu.$

The second equality $\stackrel{{\scriptstyle(*)}}{{=}}$ holds by Fact 2.5. ∎

The following is a generalization of [HPS13, Lemma 3.1] and [CGH21,
Proposition 5.1].

######  Theorem 4.8.

Let $\mu_{x}\in{\mathfrak{M}}({\cal U})$ be a dependent measure and definable
over $M$, and $\lambda_{y}\in{\mathfrak{M}}({\cal U})$ be $M$-finitely
satisfied. Then $\mu_{x}\otimes\lambda_{y}=\lambda_{y}\otimes\mu_{x}$.

###### Proof.

By Fact 3.1(iv), there is $(\bar{a}_{i})\in(M^{y})^{<\omega}$ such that
$Av(\bar{a}_{i})\to\lambda$. Set $\lambda_{i}=Av(\bar{a}_{i})$.

$\displaystyle\lambda\otimes\mu(\phi(x,y;d))$
$\displaystyle\stackrel{{\scriptstyle(1)}}{{=}}[\lim_{i}(\lambda_{i}\otimes\mu)]\phi(x,y)$
$\displaystyle\stackrel{{\scriptstyle(2)}}{{=}}[\lim_{i}(\mu\otimes\lambda_{i})]\phi(x,y)$
$\displaystyle\stackrel{{\scriptstyle(3)}}{{=}}[\mu\otimes(\lim_{i}\lambda_{i})]\phi(x,y)=\mu\otimes\lambda(\phi(x,y;d)).$

As the $\lambda_{i}$’s are smooth over $M$, (2) follows. (1) and (3) follows
from Lemma 4.7 and definability of $\mu$, respectively. (Cf. also [CGH21,
Lemma 5.4] for a proof of (3).) ∎

######  Example 4.9 (Non-example).

(i) In Proposition 7.14 of [CGH21], it is shown that there is a complete
theory $T$, a global definable measure $\mu_{x}$ and a finitely satisfiable
(and definable) type $q_{y}$ such that $\mu\otimes q\neq q\otimes\mu$.
Therefore, by Theorem 4.8, $\mu$ is not dependent. (A direct examination of
the fact that $\mu$ is not dependent can be instructive and we will do it
elsewhere.)
(ii) By Proposition 3.5, in any non-$NIP$ theory, there is a measure that is
not dependent.

######  Remark 4.10.

An obvious question is whether the argument of Theorem 4.8 works with a weaker
condition than dependence of measures. Although a localized version of it
works (cf. Remark 4.13 below), but the answer is negative from one
perspective: As measurability is necessary for the definition of
$\mu\otimes\lambda$, such a condition must require measurability. On the other
hand, there is a weaker notion of ‘dependent measure’ which is equivalent to
measurability, namely $R$-stable. ( See [Fre06, 465S] or [T84].) The only
difference is in the definition of product of measures. Therefore, the above
arguments work if and only if we use the notion $R$-stable (or $R$-dependent
measure) instead of dependent measure.

For short, we say that $\mu\in{\mathfrak{M}}({\cal U})$ is dfs over $M$ if it
is both definable over and finitely satisfiable in $M$. We say that $\mu$ is
$ddfs$ (over $M$) if it is both $dfs$ (over $M$) and dependent.

######  Corollary 4.11.

Let $\mu\in{\mathfrak{M}}({\cal U})$ be $ddfs$. Then $\mu$ is symmetric, that
is, for any $n\in\mathbb{N}$ and any permutation $\sigma$ of
$\\{1,\ldots,n\\}$, $\mu_{x_{1}}\otimes\cdots\otimes\mu_{x_{n}}=\mu_{\sigma
x_{1}}\otimes\cdots\otimes\mu_{\sigma x_{n}}$.

###### Proof.

This follows from Theorem 4.8 and associativity of $\otimes$ for definable
measures. (See [CGH21, Thm 2.18] for a proof of associativity of $\otimes$ for
definable measures.) ∎

In the next section, we only need a local version of dependent measures, which
we describe below.

######  Definition 4.12.

We say that $M^{*}$ is a good extension of $M$, if it is an elementary
extension of $M$ such that any $n$-type ($n\in\mathbb{N}$) over $M$ is
realized in $M^{*}$.

######  Remark 4.13.

Let $M$ be a model, $M^{*}$ a good extension of $M$, and $\mu_{x},\lambda_{y}$
two global $M$-invariant measures. It is easy to verify that all the arguments
of this section are established with a weaker assumption than dependence of
measures. In fact, it is enough to assume that the measures are dependent over
$M^{*}$ and in $M^{*}$ (cf. Definition 3.2(a)). For example in Definition 4.4,

$\mu(x)\otimes\lambda(y)(\phi(x,y;b))=\mu(x)\otimes\lambda(y)(\phi(x,y;b^{\prime})),$

where $tp(b/M)=tp(b^{\prime}/M)$ for some $b^{\prime}\in M^{*}$. Similarly, in
Proposition 4.3, we can define the function $f:S_{\phi^{*}}(M^{*})\to[0,1]$,
by $q\mapsto\mu(\phi(x,d;b^{\prime}))$ for some (any) $d\models q$, where
$tp(b/M)=tp(b^{\prime}/M)$ and $b^{\prime}\in M^{*}$.

This remark will be used in the next section.

##  5 $fim$ and local $NIP$

In this section we show that, assuming a local version of $NIP$, any $dfs$
measure is $fim$. Recall from [HPS13] that: a global measure $\mu$ is $fim$
(over $M$) if (i) for each $\epsilon>0$ there is an $L(M)$-formula
$\theta_{\epsilon}(x_{1},\ldots,x_{n})$ such that
$\mu^{(n)}(\theta_{\epsilon})\geq 1-\epsilon$, and (ii) for all $b$,
$|\mu(\phi(x,b))-Av(a_{1},\ldots,a_{n};\phi(x,b))|\leq\epsilon$ for all
$(a_{1},\ldots,a_{n})\in\theta_{\epsilon}({\cal U})$.

######  Definition 5.1.

Let $M$ be a model and $\phi(x,y)$ a formula. We say that $\phi(x,y)$ is
uniformly $NIP$ in $M$ if there is a natural number $n=n_{\phi,M}$ such that
there is no $a_{1},\ldots,a_{n}\in M$ such that for any
$I\subseteq\\{1,\ldots,n\\}$, $M\models\exists y\bigwedge_{i\in
I}\phi(a_{i},y)\wedge\bigwedge_{i\notin I}\neg\phi(a_{i},y)$.555In the notion,
‘uniformly’ emphasized that, in contrast to ‘$NIP$ in a model’ in [KP18],
there is a natural number $n_{\phi,M}$ for any formula $\phi$.

It is easy to see that $\phi$ has $NIP$ for the theory $T$ iff it is uniformly
$NIP$ in the monster model of $T$ iff it is uniformly $NIP$ in some model of
$T$ in which all types over the empty set in countably many variables are
realised.

######  Remark 5.2.

Suppose that $M^{*}$ is a good extension of $M$ such that every formula is
uniformly $NIP$ in $M^{*}$. Then all results in the previous section hold
without assuming ‘dependence of measure’. Indeed, recall from Remark 4.13
that, as measures are $M$-invariant, we just need the measures to be dependent
over $M^{*}$ and in $M^{*}$. On the other hand, it is easy to check that
uniformly $NIP$ in $M^{*}$ is stronger than the dependence over $M^{*}$ and in
$M^{*}$.

######  Theorem 5.3.

Let $\mu\in{\mathfrak{M}}({\cal U})$ be $dfs$ over $M$. Suppose that $M^{*}$
is a good extension of $M$ such that every formula is uniformly $NIP$ in
$M^{*}$. Then $\mu$ is $fim$ (over $M$).

###### Proof.

By Remark 5.2 and Corollary 4.11, $\mu$ is symmetric. Now, the argument of
Lemma 3.4 of [HPS13] works well by using $\mu|_{M^{*}}$ instead of $\mu$.
Therefore, similar to [HPS13, Corollary 3.5], for each $\epsilon>0$ there is
an $L(M)$-formula $\theta_{\epsilon}(x_{1},\ldots,x_{n})$ such that
$\mu^{(n)}(\theta_{\epsilon})\geq 1-\epsilon$, AND (ii)∗ for all
$b^{\prime}\in M^{*}$,
$|\mu(\phi(x,b^{\prime}))-Av(a_{1}^{\prime},\ldots,a_{n}^{\prime};\phi(x,b^{\prime}))|\leq\epsilon$
for all $(a_{1}^{\prime},\ldots,a_{n}^{\prime})\in\theta_{\epsilon}(M^{*})$.
Suppose that $(a_{1},\ldots,a_{n})\in\theta_{\epsilon}({\cal U})$ and
$b\in{\cal U}$. As $M^{*}$ is a good extension, we can find
$a_{1}^{\prime},\ldots,a_{n}^{\prime},b^{\prime}$ in $M^{*}$ such that
$(a_{1}^{\prime},\ldots,a_{n}^{\prime},b^{\prime})\equiv_{M}(a_{1},\ldots,a_{n},b)$.
Therefore, (ii) for all $b\in\cal U$,
$|\mu(\phi(x,b))-Av(a_{1},\ldots,a_{n};\phi(x,b))|\leq\epsilon$ for all
$(a_{1},\ldots,a_{n})\in\theta_{\epsilon}({\cal U})$. This means that $\mu$ is
$fim$. ∎

Recall from [CGH21] that there are $fam$ types that are not $fim$, and there
are $dfs$ types that are not $fam$. As types are dependent, in Theorem 5.3,
one should not expect to be able to exchange ‘uniformly $NIP$ in $M^{*}$’ with
‘dependence of measure’.

An obvious question is whether each $fim$ measure is dependent. A positive
answer indicates that the notion ‘dependent measure’ is necessary and the
least possible one expects.

######  Proposition 5.4.

Every $fim$ measure is dependent.

###### Proof.

Let $\mu_{x}\in{\mathfrak{M}}({\cal U})$ be $fim$ over a small model $M$. For
any formula $\phi(x,y)$, there are formulas
$\theta_{\epsilon_{n}}(x_{1},\ldots,x_{n})\in L(M)$ such that
$\mu^{(n)}(\theta_{\epsilon_{n}})\to 1$, ${\epsilon_{n}}\to 0$ and for all
$b$, $|\mu(\phi(x,b))-\frac{1}{n}\sum_{1}^{n}\phi(a_{i},b)|\leq{\epsilon_{n}}$
for all $(a_{1},\ldots,a_{n})\in\theta_{\epsilon_{n}}({\cal U})$. Therefore,
it is easy to verify that for any $\epsilon>0$,

$(\mu^{n})^{*}\Big{\\{}(p_{1},\ldots,p_{n})\in(S_{x_{1}}({\cal U}))^{n}\ :\
\sup_{b\in{\cal
U}}\big{|}\mu(\phi(x,b))-\frac{1}{n}\sum_{1}^{n}\phi(a_{i},b)\big{|}\leq\epsilon\Big{\\}}\to
1.$

Indeed, note that if $p\in S_{x_{1},\ldots,x_{n}}({\cal U})$ and
$\theta_{\epsilon_{n}}\in p$ then $(p|_{x_{1}},\ldots p|_{x_{n}})$ is in the
measured set on the left. This is enough, by the equivalence (i) $\iff$ (iii)
of Fact 2.4. (Recall also Definition 3.2 and Remark 3.3(i).) ∎

From one perspective, the following result generalizes Theorem 5.16 of
[CGH21]. It uses Proposition 5.4 to ensure that products of $fim$ and finitely
satisfied measures make sense.

######  Theorem 5.5.

Let $\mu_{x}\in{\mathfrak{M}}({\cal U})$ be $fim$ (over $M$), and
$\lambda_{y}$ be $M$-finitely satisfied. Then $\mu$ commutes with $\lambda$.

###### Proof.

This follows from Theorem 4.8 and Proposition 5.4. ∎

We can give a proof similar to [CGH21, Pro. 5.15]. The point here is that, as
$\lambda$ is $M$-finitely satisfied and $\mu$ is dependent (by Proposition
5.4), every fiber function $f_{\lambda,N}^{\phi^{*}}$ is
$\mu|_{N}$-measurable. (Cf. Definition 4.4.) We just have to check everything
still works well.

At the end paper let us ask the following questions:

######  Question 5.6.

(i) Is the product of two global $ddfs$ measures $ddfs$?
(ii) Is every $dfs$ measure dependent? If so, by Theorem 4.8, the answer to
Question 5.10 of [CGH21] is positive (i.e., any two $dfs$ global measures
commute.) As the product of two $dfs$ measures is $dfs$, a positive answer to
(ii) also automatically gives a positive answer to (i).

Although we strongly believe that the answer to (i) is negative. Indeed,
suppose that $f:S_{\phi}({\cal U})\times S_{\phi^{*}}({\cal U})\to[0,1]$ is a
function and for all $p\in S_{\phi}({\cal U})$ and $q\in S_{\phi^{*}}({\cal
U})$, $x$-sections $f_{p}:S_{\phi^{*}}({\cal U})\to[0,1]$ and $y$-sections
$f^{q}:S_{\phi}({\cal U})\to[0,1]$ are measurable. There is no guarantee that
$f$ will be measurable. A similar idea may lead to the rejection of a claim in
the initial version of [CGH21] that $fim$ measures are closed under Morley
product, i.e. the products of $fim$ measures are $fim$. On the other hand, it
is easy to verify that, assuming every formula is uniformly $NIP$ in a good
extension of a model $M$ (cf. Theorem 5.3), the product of two $fim$ (over
$M$) measures is $fim$.666This is a consequence of Theorem 5.3 and the fact
that the products of $dfs$ measures are $dfs$. Finally, we believe that the
answer to Question 5.10 of [CGH21] is negative, however, we should wait for
such counterexamples in future work. In $NIP$ theories, the answer to these
questions is clearly positive. So we can make the questions more accurate.

######  Question 5.7.

(i) In which theories is the product of two global $ddfs$ measures $ddfs$?
(ii) In which theories is every $dfs$ measure dependent?

##  Appendix A Appendix

In this appendix, for the sake of completeness, we give a model theoretic
proof of a weaker version of Theorem 4.8 using the previous observations.

######  Proposition A.1.

Let $\mu_{x}\in{\mathfrak{M}}({\cal U})$ be a global dependent measure and
definable over $M$, and $\lambda_{y}\in{\mathfrak{M}}({\cal U})$ be a global
dependent measure and $M$-finitely satisfied. Then
$\mu_{x}\otimes\lambda_{y}=\lambda_{y}\otimes\mu_{x}$.

###### Proof.

By Lemma 4.5, we can assume that $\lambda_{y}=q_{y}$ is a type. Assume, for a
contradiction, that $\mu_{x}\otimes q_{y}(\phi(x,y))<r-2\epsilon$ and
$q_{y}\otimes\mu_{x}(\phi(x,y))>r+2\epsilon$, for some $r\in[0,1]$ and
$\epsilon>0$. Let $N\supseteq Md$ be a model where $\phi(x,y)=\phi(x,y;d)$.
(Cf. Definition 4.4.) Recall that, as $q$ is $M$-finitely satisfied and $\mu$
is dependent, $f_{q,N}^{\phi^{*}}$ is $\mu|_{N}$-measurable and so
$q_{y}\otimes\mu_{x}$ is well-defined. By definability of $\mu$, there is a
formula $\psi(y)\in L({\cal U})$ such that: (1) $q\vdash\psi(y)$ and for all
$b\in\psi({\cal U})$, $\mu(\phi(x,b))<r-\epsilon$. (See also Fact 1.1(a) of
[CGH21].)

As $f_{q,N}^{\phi^{*}}$ is $\mu|_{N}$-measurable, there is a
$\mu|_{N}$-measurable set $X\subseteq S_{\phi}(N)$ such that: (2)
$q\vdash\phi(a,y)$ iff $tp(a/N)\in X$.

By Corollary 4.2, pick $p_{1},\ldots,p_{n}\in S_{\phi}(N)$ such that: (3) for
all $b\in N$, $|Av(p_{1},\ldots,p_{n};\phi(x,b))-\mu(\phi(x,b))|<\epsilon$
and, (4) $|Av(p_{1},\ldots,p_{n};X)-\mu(X)|<\epsilon$.

Let $a_{i}\models p_{i}$ ($i\leq n$). By finite satisfiability of $q$ there is
$b_{0}\in\psi(M)$ such that: (5) $\models\phi(a_{i},b_{0})$ iff
$q\vdash\phi(a_{i},y)$ ($i\leq n$).

It is easy to see, by (1) and (3), that
$Av(p_{1},\ldots,p_{n};\phi(x,b_{0}))<r$. Also, by (5) and (2),
$Av(p_{1},\ldots,p_{n};\phi(x,b_{0}))=\frac{1}{n}\sum
q(\phi(a_{i},y))=\frac{1}{n}|\\{i:a_{i}\in X\\}|=Av(p_{1},\ldots,p_{n};X)$
which is within $\epsilon$ of
$\mu(X)=q_{y}\otimes\mu_{x}(\phi(x,y))>r+2\epsilon$, by (4). This is a
contradiction. ∎

Acknowledgements. I want to thank Gabriel Conant for his interest in reading a
preliminary version of this article and for his comments.

## References

  * [BFT78] J. Bourgain, D. H. Fremlin, and M. Talagrand. Pointwise compact sets of baire-measurable functions. American Journal of Mathematics, 100(4):pp. 845-886, 1978.
  * [ChG21] A. Chernikov and K. Gannon, Definable convolution and idempotent Keisler measures, arXiv:2004.10378, 2021.
  * [CG20] G. Conant, K. Gannon, Remarks on generic stability in independent theories, Ann. Pure Appl. Logic 171 (2020), no. 2, 102736, 20. MR 4033642
  * [CGH21] G. Conant, K. Gannon, J. Hanson, Keisler measures in the wild, arxiv 2021
  * [Fre06] D. H. Fremlin, Measure Theory, vol.4, (Topological Measure Spaces, Torres Fremlin, Colchester, 2006). (See https://www1.essex.ac.uk/maths/people/fremlin/cont46.htm )
  * [G20] K. Gannon, Approximation theorems for Keisler measures, Ph.D. thesis, University of Notre Dame, 2020.
  * [HPS13] E. Hrushovski, A. Pillay, and P. Simon. Generically stable and smooth measures in NIP theories. Transactions of the American Mathematical Society 365.5 (2013): 2341-2366.
  * [Kha16] K. Khanaki, Amenability, extreme amenability, model-theoretic stability, and NIP in integral logic, Fundamenta Mathematicae 234 (2016), 253-286
  * [Kha21] K. Khanaki, Glivenko-Cantelli classes and $NIP$ formulas, submitted, arXiv:2103.10788v4, 2021.
  * [Kha17] K. Khanaki, NIP formulas and Baire 1 definability, unpublished note, arXiv:1703.08731, 2017.
  * [Kha20] K. Khanaki, Stability, the NIP, and the NSOP; Model Theoretic Properties of Formulas via Topological Properties of Function Spaces, Math. Log. Quart. 66, No. 2, 136-149 (2020) / DOI 10.1002/malq.201500059
  * [KP18] K. Khanaki, A. Pillay, Remarks on $NIP$ in a model, Math. Log. Quart. 64, No. 6, 429-434 (2018) / DOI 10.1002/malq.201700070
  * [S15] P. Simon. A guide to NIP theories. Cambridge University Press, 2015.
  * [T84] M. Talagrand, Pettis integral and measure theory. Mem. Amer. Math. Soc. 307 (1984).
  * [T87] M. Talagrand, The Glivenko–Cantelli problem, Ann. Probab. 15 (1987), 837-870.


# Using Contextual Information for Sentence-level Morpheme Segmentation

Prabin Bhandari
Department of Computer Science
George Mason University
<EMAIL_ADDRESS>
Abhishek Paudel
Department of Computer Science
George Mason University
<EMAIL_ADDRESS>

###### Abstract

Recent advancements in morpheme segmentation primarily emphasize word-level
segmentation, often neglecting the contextual relevance within the sentence.
In this study, we redefine the morpheme segmentation task as a sequence-to-
sequence problem, treating the entire sentence as input rather than isolating
individual words. Our findings reveal that the multilingual model consistently
exhibits superior performance compared to monolingual counterparts. While our
model did not surpass the performance of the current state-of-the-art, it
demonstrated comparable efficacy with high-resource languages while revealing
limitations in low-resource language scenarios.

##  1 Introduction

The problem of morpheme segmentation deals with decomposing a word into a
sequence of morphemes that represent the smallest meaningful unit of words
like prefixes, suffixes, and root words. For example, the word _pokers_ can be
decomposed into its morphemes as _poke@@er@@s_ , where _@@_ represents
separations between morphemes. Recent advances in morpheme segmentation focus
mostly on word-level segmentation (Batsuren et al., 2022; Peters and Martins,
2022) and do not take the context of the word in the sentence into
consideration. Instead, we focus on the task of sentence-level morpheme
segmentation in which the context of the word in a sentence is taken into
account for morpheme segmentation.

The SIGMORPHON 2022 Shared Task on Morpheme Segmentation (Batsuren et al.,
2022) formulates the task of morpheme segmentation into two subtasks: word-
level segmentation and sentence-level segmentation. Although many submissions
in this task show significant improvements over baselines on both subtasks
(Rouhe et al., 2022; Wehrli et al., 2022; Peters and Martins, 2022), most
submitted approaches for sentence-level segmentation subtask ignore the
context of words by design treating the problem as word-level morpheme
segmentation, i.e. they treat each word in a sentence as a separate example.
However, in many languages, the context of a word might provide a piece of
meaningful information to disambiguate the morphology of the word (Batsuren et
al., 2022) and help improve morpheme segmentation. Consider the following
sentences in Mongolian and their corresponding morpheme segmentation:

  1. 1.

Гэрт эмээ хоол хийв. $\rightarrow$ Гэр @@т эмээ хоол хийх @@в.

  2. 2.

Би өдөр эмээ уусан. $\rightarrow$ Би өдөр эм @@ээ уух @@сан.

In the above examples, the word эмээ has different meanings in each sentence.
In the first sentence, it means _grandmother_ and is not segmentable. However,
in the second one, it means _medicine_ and is segmentable as эм @@ээ.
Therefore, the context of a word in a sentence is an important factor for
determining how the word should be segmented into its morphemes.

We focus on the task of sentence-level morpheme segmentation while taking the
whole sentence in context and implement a sequence-to-sequence transformer
model (Vaswani et al., 2017) inspired by DeepSPIN-3 (Peters and Martins, 2022)
which is the winner of word-level morpheme segmentation in the SIGMORPHON 2022
Shared Task on Morpheme Segmentation (Batsuren et al., 2022). While most
existing methods treat sentence-level morpheme segmentation as a zero-shot
solution of word-level morpheme segmentation (Batsuren et al., 2022), we treat
each sentence as a whole as one training example to preserve its context and
treat the problem as a sequence to sequence generation task. The sentence-
level dataset provided for the shared task consists of three languages: Czech,
English, and Mongolian.

We perform various experiments with monolingual and multilingual sequence-to-
sequence transformer models and show that the multilingual model generally
performs better over monolingual models, especially for low-resource
languages. Additionally, we experiment with data augmentation in which we
increase the training dataset by combining samples for word-level morpheme
segmentation. We also experiment with upsampling the sentence-level data given
the lack of enough training data for low-resource languages like Czech and
Mongolian. Although we are not able to outperform the winners of the shared
task, our results are close for high resource languages like English (F1
score: 95.10) but relatively underperform for low-resource languages like
Czech (F1 score: 75.79) and Mongolian (F1 score: 72.54).

##  2 Approach

We implement sequence to sequence transformer model similar to DeepSPIN-3
(Peters and Martins, 2022) for sentence-level morpheme segmentation while
treating each sentence as one training example. Our implementation is done
using fairseq (Ott et al., 2019). This sequence-to-sequence task setting for
sentence-level morpheme segmentation is similar to machine translation
settings but differs in the sense that the source language corresponds to
sentence examples and the target language corresponds to the sentence in which
the words are segmented into their corresponding morphemes. We train
monolingual and multilingual models with the same transformer architecture.
The overall approach for both monolingual and multilingual models are
described below.

###  2.1 Tokenization

We use Google’s SentencePiece tokenizer with subword regularization using
Unigram Language Model (ULM) (Kudo, 2018). ULM is a top-down technique where
the model is initially initialized with a vast vocabulary of overlapping sub-
words and a score is generated based on expectation maximization for each sub-
word. Up until the anticipated vocabulary size is attained, the lowest-scoring
sub-words are trimmed.

The SentencePiece tokenizer is trained using both source and target data, and
all tokens for sentence-level data are pre-computed before training as
expected by fairseq’s training workflow. The vocabulary size for English is
chosen to be 6000 to 8000 for English and 1000 to 5000 for Czech and Mongolian
depending upon the data augmentation which could allow for higher vocabulary
sizes. Additionally, for multilingual model, we use the vocabulary size to be
9000 to 12000 so as to take into account a large number of diverse tokens
present in multiple languages.

###  2.2 Model Architecture

Our transformer model (Vaswani et al., 2017) consists of 6 encoder and 6
decoder blocks with 8 multi-head attentions. We use an embedding size of 256
and dropout of 0.3 determined through hyperparameter tuning on the dev set.
The size of each feed-forward layer is 1024 which is also determined through
hyperparameter tuning on the dev set. We use identical architecture for all
three languages: Czech, English, and Mongolian.

###  2.3 Training

We train for a maximum of 400,000 updates and perform early stopping based on
the validation loss. We use learning rate warm-up for 4000 steps for different
languages determined through hyperparameter tuning. We also use inverse square
root learning rate scheduling during our training. We use a batch size of 8192
for all the experiments.

Our model is trained using entmax-loss (Peters et al., 2019) with alpha of 1.5
similar to Peters and Martins (2022) since it seemed to show better
performance compared to cross-entropy loss in our preliminary experiments. The
basic idea behind entmax loss is to penalize the model when it produces
probability distributions that are too peaked or overly concentrated on a
single class. One advantage of entmax loss over cross entropy loss is that it
encourages the model to produce probability distributions that are more evenly
distributed over the possible classes. This can help the model make more
nuanced predictions and avoid over-confident mistakes caused due to overly
confident predictions on a single class. This makes entmax loss a better
choice over cross-entropy loss in many applications. However, entmax loss is
computationally more expensive to compute than cross-entropy loss. We
therefore use existing implementation of entmax loss rather than implementing
it from scratch.

###  2.4 Inference and Evaluation

For generating morpheme segmentation for each sentence, we use beam search
with a beam size of 5. The tokenizer trained using the training data is used
to tokenize sentences during inference. Additional postprocessing is done to
clean up raw output from the model to obtain final segmentation. We use F1
score as our primary metric to evaluate the morpheme segmentation at the
sentence-level. This means that the generated segmentation for a sentence is
considered to be correct only if all the words in that sentence is segmented
correctly. Precision, recall and Levenshtein distance, although computed, have
not been reported in this paper.

| Czech | English | Mongolian
---|---|---|---
Monolingual Sequence-to-Sequence Transformer | 22.62 | 87.94 | 21.00
\+ Upsampling | 37.58 | 88.62 | 39.40
\+ Word Augmentation | 25.87 | 92.46 | 16.76
\+ Word Augmentation + Upsampling | 46.62 | 95.10 | 38.00
Multilingual Sequence-to-Sequence Transformer | 66.77 | 88.54 | 68.12
\+ Word Augmentation + Upsampling | 75.79 | 92.93 | 72.54
Official highest score in Batsuren et al. (2022) | 91.99 | 96.31 | 82.88
DeepSPIN-sent (Peters and Martins, 2022) | _93.23_ | _98.24_ | _83.59_
Table 1: F1 scores obtained for sentence-level test sets for Czech, English and Mongolian languages in monolingual and multilingual settings (top two blocks) along with highest scores of official shared task (Batsuren et al., 2022) winners (AUUH_B (Rouhe et al., 2022) for English and CLUZH-3 (Wehrli et al., 2022) for Czech and Mongolian ) and DeepSPIN-3 (unofficial highest scorer). The bold numbers are highest scores for our implementation, and italicized numbers are highest overall scores. Language | train | dev | test
---|---|---|---
Czech | 1000 | 500 | 500
English | 11007 | 1783 | 1845
Mongolian | 1000 | 500 | 500
Table 2: Break down of sentence-level morpheme segmentation for Czech, English
and Mongolian in train, dev and test set.

##  3 Experiments

We evaluate our approach on the sentence-level morpheme segmentation dataset
provided by Batsuren et al. (2022). This dataset consists of train, dev and
test samples in Czech, English and Mongolian languages as shown in Table 2. We
experiment with monolingual models (separate models for each language) and
multilingual model (one model for all languages) that are discussed below.

###  3.1 Monolingual Experiments

We experiment with monolingual models in which we train three different models
separately on Czech, English and Mongolian datasets. We conduct few different
experiments using the sequence to sequence transformer model as described in 2
using only the sentence-level dataset, augmenting with word dataset and
upsampling the dataset.

####  3.1.1 Sentence-level Dataset Only

In this experiment, we take the sentence-level dataset in three languages and
train the sequence to sequence transformer model for each of these languages.
The results are shown in the Table 1 (second row). We observe that for high
resource language like English, the F1 score is 87.94 while drops
significantly for low-resource language likes Czech (F1 score: 22.62) and
Mongolian (F1 score: 21.00).

####  3.1.2 Data Upsampling

Since there are not many training examples especially in Czech and Mongolian,
the sentence-level training set is upsampled 100 times for Czech and Mongolian
and 10 times for English. The results with data upsampling are shown in Table
1 (third row). We observe that the results for all three languages improve but
the gap between English, which is a high-resource language, and low-resource
languages like Czech and Mongolian is still large.

####  3.1.3 Word-level Dataset Augmentation

Language | Number of Samples
---|---
Czech | 38682
English | 577374
Mongolian | 18966
Table 3: Break down of word-level morpheme segmentation dataset for Czech,
English and Mongolian used to augment sentence-level dataset.

Another way to increase the number of training data we employ is to augment
the sentence-level training set with word-level dataset from their
corresponding languages. The results with word-level datset augmentation are
shown in Table 1 (fourth row). We observe that augmenting the word-level
dataset significantly improves the F1 score for English to 92.46 compared to
87.94 obtained without any upsampling or word-level dataset augmentation.
However, there is only small improvement for Czech language (from 22.62
without any upsampling or word-level dataset augmentation to 25.87 in this
case). Surprisingly, augmenting the word-level dataset reduced the performance
for Mongolian language. This could be because of the prevalence of context-
dependent morpheme segmentation examples like the one discussed in Sec. 1. We
believe that understanding the core reason for this drop in performance might
require knowledge of Mongolian language’s morphology.

####  3.1.4 Word Augmentation with Upsampling

We combine word-level data augmentation and then perform upsampling similar to
Sec. 3.1.2 and use this dataset to train the transformer model. The results
are shown in Table 1 (fifth row). We observe that this achieves the highest
overall F1 score of 95.10 for English. We also see improvements for Czech with
F1 scores of 46.62. Although there is improvement for Mongolian with F1 score
of 38.00 compared to word-level dataset augmentation, this is still lower than
F1 score of 39.40 obtained with upsampling, and therefore this improvement is
most likely only contributed by upsampling rather than word-level dataset
augmentation which most likely slightly hampered the performance instead.

###  3.2 Multilingual Experiments

We experiment with multilingual model in which we concatenate training
datasets from Czech, English and Mongolian into one large multilingual dataset
and train one tokenizer model and one sequence-to-sequence transformer on this
multilingual dataset. We do not use any language identifier token to identify
the samples from different languages and instead perform simple concatenation
only. We also experiment with word-level dataset augmentation and then perform
upsampling on this augmented multilingual dataset since this approach
demonstrated better performance with monolingual models. The results are shown
in Table 1 (middle block).

We observe that the multilingual model outperforms monolingual models for
Czech and Mongolian in which the F1 scores are 75.79 and 72.54 respectively.
This shows that Czech and Mongolian languages significantly benefited from
multilingual settings. For English, the performance drops slightly to 92.93 in
multilingual settings from 95.10 in monolingual settings. Overall, we observe
that low-resource languages see much more performance improvements in
multilingual settings as expected.

##  4 Related Work

Many NLP applications require tokenization of words into subwords. However,
most subword tokenization approaches like Byte-Pair Encoding (BPE) (Gage,
1994; Sennrich et al., 2016) and Unigram Language Model (ULM) (Kudo, 2018)
ignore the morphophology of words during tokenization and instead tokenize
words based on statistical co-occurence of subwords. Batsuren et al. (2022)
proposed SIGMORPHON 2022 Shared Task on Morpheme Segmentation citing the
necessity of linguistically motivated subword tokenization in which many
approaches demonstrated state-of-the-art performance over existing baseline
systems like WordPiece (Schuster and Nakajima, 2012), ULM (Kudo, 2018) and
Morfessor2 (Virpioja et al., 2013).

For example, DeepSPIN-2 and DeepSPIN-3 (Peters and Martins, 2022) collectively
demonstrate a superior performance on word-level morpheme segmentation on all
9 languages considered on the shared task. AUUH (Rouhe et al., 2022), CLUZH
(Wehrli et al., 2022) and others also show significantly improved performance
over baselines for both subtasks. However, most submitted approaches ignore
the context of words in a sentence for sentence-level morpheme segmentation.

Our approach formulates sentence-level morpheme segmentation by considering
sentence as a whole to preserve the context (including the unofficial highest
scorer DeepSPIN-sent) and performs morpheme segmentation without separating
the sentence into individual words before morpheme segmentation.

##  5 Conclusion and Future Work

Our results show that our approach with multilingual model shows better
performance over monolingual models. Additionally, the performance improvement
is more significant for low-resource languages like Czech and Mongolian than
English for which comparatively large amount of training data was available.
We also show that word-level dataset augmentation and data upsampling show
improved performance particularly for low resource languages. In this regard,
we emphasize that simple techniques like data upsampling could serve as a
first step for low-resource languages in some NLP tasks. Overall, although our
results do not outperform state of the art, they are competitive for high-
resource languages like English and have the potential for improved
performance upon tackling issues with regards to the lack of enough data for
low-resource languages like Czech and Mongolian.

Future directions of this research could try to tackle lack of enough data for
low-resource languages by investigating techniques like semi-supervised
learning in which a trained model is used to generate labels for newly
collected data and then integrate this newly labelled data to expand the
training data if the predicted probabilities for the labels cross certain
threshold. This could help generate new labelled data that could then be used
to train new models. Additionally, our multilingual model do not distinguish
between language samples. Adding an additional language identifier token in
data based on language could potentially help improve the performance of
multilingual model further.

## References

  * Batsuren et al. (2022) Khuyagbaatar Batsuren, Gábor Bella, Aryaman Arora, Viktor Martinović, Kyle Gorman, Zdeněk Žabokrtskỳ, Amarsanaa Ganbold, Šárka Dohnalová, Magda Ševčíková, Kateřina Pelegrinová, et al. 2022.  The SIGMORPHON 2022 shared task on morpheme segmentation.  _arXiv preprint arXiv:2206.07615_.
  * Gage (1994) Philip Gage. 1994.  A new algorithm for data compression.  _C Users Journal_ , 12(2):23–38.
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of NAACL-HLT 2019: Demonstrations_.
  * Peters and Martins (2022) Ben Peters and André FT Martins. 2022.  Beyond characters: Subword-level morpheme segmentation.  In _Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 131–138.
  * Peters et al. (2019) Ben Peters, Vlad Niculae, and André FT Martins. 2019.  Sparse sequence-to-sequence models.  _arXiv preprint arXiv:1905.05702_.
  * Rouhe et al. (2022) Aku Rouhe, Stig-Arne Grönroos, Sami Virpioja, Mathias Creutz, and Mikko Kurimo. 2022.  Morfessor-enriched features and multilingual training for canonical morphological segmentation.  In _Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_. The Association for Computational Linguistics.
  * Schuster and Nakajima (2012) Mike Schuster and Kaisuke Nakajima. 2012.  Japanese and korean voice search.  In _2012 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 5149–5152. IEEE.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers) 1715–1725 association for computational linguistics https://www. aclweb. org/anthology.  _P16-1162_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Virpioja et al. (2013) Sami Virpioja, Peter Smit, Stig-Arne Grönroos, Mikko Kurimo, et al. 2013.  Morfessor 2.0: Python implementation and extensions for morfessor baseline.
  * Wehrli et al. (2022) Silvan Wehrli, Simon Clematide, and Peter Makarov. 2022.  Cluzh at sigmorphon 2022 shared tasks on morpheme segmentation and inflection generation.  In _Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 212–219.