# Disordered crystals reveal soft quasilocalized glassy excitations

E. Lerner<EMAIL_ADDRESS>Institute for Theoretical Physics, University of
Amsterdam, Science Park 904, Amsterdam, Netherlands  E. Bouchbinder
<EMAIL_ADDRESS>Chemical and Biological Physics Department,
Weizmann Institute of Science, Rehovot 76001, Israel

###### Abstract

Structural glasses formed by quenching a melt are known to host a population
of low-energy quasilocalized (nonphononic) excitations whose frequencies
$\omega$ follow a universal $\sim\\!\omega^{4}$ distribution as
$\omega\\!\to\\!0$, independently of the glass formation history, the
interparticle interaction potential or spatial dimension. Here, we show that
the universal quartic law of nonphononic excitations also holds in disordered
crystals featuring finite long-range order, which is absent in their glassy
counterparts. We thus establish that the degree of universality of the quartic
law extends beyond structural glasses quenched from a melt. We further find
that disordered crystals, whose level of disorder can be continuously
controlled, host many more quasilocalized excitations than expected based on
their degree of mechanical disorder — quantified by the relative fluctuations
of the shear modulus — as compared to structural glasses featuring a similar
degree of mechanical disorder. Finally, we show that the stability bound on
nonlinear quasilocalized excitations — previously established in structural
glasses — also holds in disordered crystals. Our results are related to glass-
like anomalies experimentally observed in disordered crystals. More broadly,
they constitute an important step towards tracing the essential ingredients
necessary for the emergence of universal nonphononic excitations in disordered
solids.

##  I Introduction

It is now well established that structural glasses formed by quenching a melt
through the glass transition temperature generically feature a population of
soft, quasilocalized (nonphononic) excitations (QLEs) [1]. These excitations
are expected to play key roles in determining various static, mechanical,
transport and dynamic properties of glasses, and possibly of some supercooled
liquids as well [2, 3, 4, 5]. QLEs have been envisioned, and their importance
has been underlined, since the late 1980s by several workers, as described in
detail in a recent review [1]. A first-principle understanding of the
emergence of low-energy quasilocalized excitations in structural glasses is
still incomplete, despite some recent progress in identifying and solving
potentially relevant mean-field models [6, 7, 8, 9].

Intriguingly, the vibrational density of states (VDoS) ${\cal D}(\omega)$ of
low-energy QLEs — of vibrational frequency $\omega$ — has been shown to follow
a universal law in structural glasses, of the form ${\cal
D}(\omega)\\!\sim\\!\omega^{4}$, independently of the interparticle
interaction potential [10, 11], spatial dimension [12] or the glass formation
protocol [13, 14, 15]. Does this nonphononic quartic law remain valid in a
broader class of disordered solids, beyond structural glasses formed by
quenching a melt? Can it be observed in disordered solids featuring some long-
range order? In this work, we address these questions by studying the
statistical, structural and mechanical properties of soft excitations that
emerge in a class of _disordered crystals_. To this aim, we study a variant of
a model system put forward by Barrat and coworkers [16, 17, 18] in which
microscopic disorder can be gradually introduced into an initially perfectly-
crystalline solid, until the occurrence of a global amorphization transition
of the disordered crystal into a glassy state. Similar procedures were carried
out in two-dimensional systems in [19, 20, 21].

We employ a variant of these aforementioned three-dimensional disordered
crystal models in which the external pressure is fixed under variations of the
microscopic disorder, controlled by a dimensionless parameter
$\delta\\!\in\\![0,1]$ ($\delta\\!=\\!0$ corresponds to a perfect crystal).
Additional details about the model can be found below and in Appendix A. After
identifying the amorphization transition point, we study the vibrational
spectra of disordered crystals _prior_ to amorphization, i.e. in states that
feature finite long-range crystalline order. We find that these disordered
crystals host a population of QLEs whose frequencies obey the universal
$\sim\\!\omega^{4}$ nonphononic law. We thus establish that the degree of
universality of the nonphononic quartic law of QLEs in disordered media
extends beyond structural glasses quenched from a melt. In addition, our
results shed new light on low-temperature, glass-like thermodynamic, transport
and vibrational anomalies observed in laboratory disordered crystals [22, 23,
24, 25, 26, 27, 28, 29, 30, 31].

---
Figure 1: Dimensionless elastic properties across the amorphization
transition: (a) Poisson’s ratio $\nu$, (b) nonaffine-to-affine shear moduli
ratio $G_{\rm na}/G$, (c) relative sample-to-sample fluctuations of the shear
modulus $\chi$, see precise definitions in Appendix B. These data were
measured over about 700 independent realizations of systems of
$N\\!=\\!108000$ particles at fixed vanishing pressure. The vertical lines
mark an estimation (based on $\chi$) of the transition point at $\delta_{\rm
c}\\!\approx\\!0.49$. The black squares mark the disordered-crystal ensembles
studied below, namely $\delta\\!=\\!0.38,0.40,0.42,0.44$, all of which are
located below the amorphization transition.

We then go on to assess the degree of mechanical disorder of these disordered
crystals, and compare their relative abundance of QLEs to that found in
structural glasses featuring similar degrees of mechanical disorder. This
analysis reveals — surprisingly — that these disordered crystals host many
more QLEs as compared to structural glasses. Finally, we study the
micromechanical properties of QLEs in disordered crystals, e.g. the degree of
their spatial localization and stability bounds, and find striking
similarities to the corresponding QLEs’ properties in structural glasses. The
micromechanical properties of QLEs in disordered crystals also reveal
signatures of the proximity to the amorphization transition.

##  II Disordered crystals model and the amorphization transition

We employ a binary system of $N$ Lennard-Jones particles amongst which a
fraction $x$ is of one species (‘small’ particles) and the remaining particles
are of another species (‘large’ particles). One species may be regarded as
‘impurity atoms’ relative to the other species (‘host atoms’), as is
extensively used in laboratory disordered crystals. The parameter $\delta$
determines the effective interaction of the two species, such that
$\delta\\!=\\!0$ corresponds to a mono-disperse system, while $\delta\\!=\\!1$
corresponds to the largest-considered interaction contrast between the two
species. Details about the pairwise interaction potential and system sizes
employed are provided in Appendix A. Throughout this study we fix
$x\\!=\\!1/2$, such that $\delta\\!=\\!1$ corresponds to a 50:50 binary
mixture extensively employed in computer glass-forming models. We have
verified that the key results reported below do not qualitatively depend on
the precise choice of $x$.

---
Figure 2: Pair correlation functions $g(r)$ between the ‘large’ particle-
species (see Appendix A for details), measured for ensembles of disordered
solids of $N\\!=\\!108000$ particles created with different $\delta$ values,
as indicated in the legends. We indeed find that all the way up to the
amorphization transition at $\delta_{\rm c}\\!\approx\\!0.49$, long-range
order persists.

Disordered crystals are generated by first placing the two particle species
randomly on a perfect fcc lattice, and setting $\delta\\!=\\!0$. Then, for
each value of the parameter $\delta\\!>\\!0$, the potential energy is relaxed
under fixed zero confining pressure to obtain a disordered crystal. This
energy relaxation step spontaneously generates positional disorder and
internal stresses, absent in the reference fcc lattice. Continuously varying
$\delta$ gives rise to a class of disordered crystals with a varying degree of
disorder. Once $\delta$ exceeds a critical value — to be determined next —,
the disordered crystal collapses into a fully disordered, glassy state via a
so-called ‘amorphization transition’ [16, 17, 18].

---
Figure 3: (a) The vibrational density of states of disordered crystals, for
different values of the parameter $\delta$ (as indicated in the legend),
plotted against rescaled frequency $\omega/\omega_{0}$, where
$\omega_{0}\\!\equiv\\!c_{\rm s}/a_{0}$ with $c_{\rm s}$ being the shear wave-
speed and $a_{0}\\!\equiv\\!(V/N)^{1/3}$ is an interparticle distance. The
continuous lines correspond to fits to the $\sim\\!\omega^{4}$ law, from which
the prefactors $A_{\rm g}$ are extracted. Inset: the spatial decay of (two
randomly selected) soft quasilocalized excitations $\bm{\pi}$ (obtained by the
nonlinear framework presented in Appendix B) follows the expected continuum-
like $\sim\\!r^{-2}$ scaling in three dimensions; the colors match the
$\delta$ values in the legend. (b) Dimensionless prefactors $A_{\rm
g}\omega_{0}^{5}$ vs. $\delta$ on a log-lin scale. (c) The products $N\bar{e}$
— representing the mean size of the disordered cores of QLEs ($e$ is the
participation ratio, defined in the text) — plotted vs. $\delta$. The vertical
bars cover the 2nd and 3rd quartiles, see text for additional discussion.

To quantitatively identify this transition, we study the behavior of three
dimensionless observables while varying the control parameter $\delta$:
Poisson’s ratio $\nu$, the ratio of the nonaffine-to-affine shear modulus
$G_{\rm na}/G$, and the relative fluctuations of the shear modulus $\chi$.
Figure 1 displays our results, see figure caption for details. It is observed
that while $\nu$ and $G_{\rm na}/G$ vary smoothly and monotonically with
$\delta$, $\chi(\delta)$ exhibits a rather sharp peak, which enables the
identification of the location of the amorphization transition. The
corresponding onset threshold $\delta_{\rm c}$, which for $x\\!=\\!1/2$ takes
the value $\delta_{\rm c}\\!\approx\\!0.49$, is marked in all three panels by
the vertical dashed line. The black squares pertain to the disordered-crystal
ensembles that we study next, for which $\delta\\!<\\!\delta_{\rm c}$.

To further substantiate the amorphization transition at $\delta_{\rm c}$ and
to establish the existence of finite long-range order in our disordered
crystals for $\delta\\!<\\!\delta_{\rm c}$, we consider the pair correlation
function $g(r)$ among the ‘large’ particle-species; results for
$\delta\\!=\\!0.38,0.44,0.49,0.70$ are shown in Fig. 2. We find that solids
with $\delta\\!\lesssim\\!\delta_{\rm c}$ all feature finite long-range order,
indicated by the presence of peaks in $g(r)$ that persist to the largest
distances $r$ considered. In particular, long-range order is clearly present
in the ensembles pertaining to $\delta\\!=\\!0.38$ to $\delta\\!=\\!0.44$,
which are studied next.

##  III Universal nonphononic VDoS

In order to address the emergence of QLEs and their possible universal
nonphononic VDoS in disordered crystals, it is important to select
sufficiently small systems such that a phonon-free frequency window opens
below the first shear-wave frequency, as explained at length in [1].
Furthermore, since QLEs that emerge in structural glasses are characterized by
a core size $\xi_{\rm g}$ — usually on the order of a few interparticle
distances [1, 14] — it is also important that the linear system size is chosen
to be sufficiently larger than $\xi_{\rm g}$, in order for QLEs to fit
comfortably in the simulation box [1, 32]. To accommodate these two
requirements, we opt for studying emergent QLEs in disordered crystals of
$N\\!=\\!1372$ particles. We prepared 667900, 297600, 150000, and 109600
independent realizations of disordered crystals with
$\delta\\!=\\!0.38,0.40,0.42$ and $0.44$ (all below $\delta_{\rm c}$),
following the protocol described above and in Appendix A. For each disordered
crystal, we calculated the first 30 nontrivial vibrational modes
$\bm{\psi}_{\omega}$ as defined via Eq. (3).

The resulting VDoS are shown in Fig. 3a, which presents a key result of this
work. We find that disordered crystals featuring finite long-range order host
a population of quasilocalized modes whose frequencies obey the universal
$\sim\\!\omega^{4}$ law. Having demonstrated the validity of the nonphononic
quartic scaling law in disordered crystals, we turn now to discussing the non-
universal prefactor $A_{\rm g}$ in the full expression of the VDoS, ${\cal
D}(\omega)\\!=\\!A_{\rm g}\,\omega^{4}$. The prefactor $A_{\rm g}$, of
dimension $[\mbox{time}]^{5}$, has been suggested to serve as an indicator of
the abundance of soft QLEs [13, 33]. As such, it was the focus of several
recent studies [14, 34, 11, 35, 36]; notably, $A_{\rm g}$ was shown to
strongly correlate with the tensile fracture toughness of glassy solids [34].
The prefactors $A_{\rm g}$ of our disordered crystals — obtained here by fits
(represented by the straight lines) to a $\sim\\!\omega^{4}$ power-law at low
frequencies — are reported in Fig. 3b. We find an approximate exponential
dependence of $A_{\rm g}$ on the parameter $\delta$, which is reminiscent of
the Boltzmann-like dependence of $A_{\rm g}$ on the _equilibrium_ parent
temperature of structural glasses quenched from deeply supercooled liquids
[14, 37]. A similar exponential decay of $A_{\rm g}$ was also observed
recently in a mean-field spin-glass model [6].

##  IV QLE properties

We next study some of the structural and mechanical properties of the low-
frequency QLEs that emerge in our disordered crystals. To this aim, we invoke
the nonlinear framework of [38, 39], which provides a robust representation of
soft QLEs, denoted as $\bm{\pi}$ (see also Appendix B). We first focus on the
spatial decay of QLEs, which is presented in the inset of Fig. 3a for two
randomly selected nonlinear modes. It is observed that similarly to QLEs in
structural glasses [1], QLEs in our disordered crystals feature a disordered
core of a few interparticle distances, followed a power-law $\sim\\!r^{-2}$
tail (i.e. $\sim\\!r^{-{\,\mathchar 22\relax\mkern-12.0mud}+1}$ in three
dimensions, ${\,\mathchar 22\relax\mkern-12.0mud}\\!=\\!3$).

The degree of spatial localization of QLEs is further discussed next, by
considering the participation ratio $e$, defined as
$e(\bm{\pi})\\!\equiv\\!N^{-1}\big{(}\sum_{i}\bm{\pi}_{i}\cdot\bm{\pi}_{i}\big{)}^{2}\\!/\\!\sum_{i}\big{(}\bm{\pi}_{i}\cdot\bm{\pi}_{i}\big{)}^{2}$.
Here, $\bm{\pi}_{i}$ denotes the ${\,\mathchar
22\relax\mkern-12.0mud}$-dimensional vector of Cartesian components pertaining
to the $i^{\mbox{\tiny th}}$ particle of a QLE $\bm{\pi}$. The participation
ratio $e(\bm{\pi})$ quantifies the degree of spatial localization of
$\bm{\pi}$; in particular, $\bm{\pi}$’s that are localized on a core of
$N_{\rm c}$ particles are expected to follow $Ne\\!\approx\\!N_{\rm c}$,
whereas spatially extended $\bm{\pi}$ are expected to follow $Ne\\!\sim\\!N$
(or $e\\!\sim\\!1$). In Fig. 3c, we plot the factored means $N\bar{e}$ vs.
$\delta$; here, $\bar{e}$ was calculated over 1000 QLEs (for each value of
$\delta$) in systems of $N\\!=\\!16384$. Also shown are the 2nd and 3rd
quartiles of the products $Ne$ of the QLEs, which are covered by the error
bars. We find that moving away from the amorphization transition, giving rise
to less disordered states, leads to a stronger localization of QLEs. This
enhanced localization is reminiscent of the stronger localization of QLEs
observed in structural glasses with increasing glass stability, as shown in
e.g. [14, 15].

---
Figure 4: The square of the cubic Taylor series coefficient of QLEs,
$b_{3}^{2}$, is scatter-plotted against the product of the corresponding
quadratic and quartic expansion coefficients $b_{2}b_{4}$, see text for
further details. The continuous lines represent the stability bound
$b_{3}^{2}\\!=\\!3b_{2}b_{4}$. The emerging picture, and its relation to the
corresponding picture in structural glasses quenched from a melt [39], are
discussed in the text.

We next turn to the micromechanical properties of QLEs. In particular, for
each QLE $\bm{\pi}$ we calculate the first 3 nonzero Taylor series
coefficients — denoted $b_{2},b_{3},b_{4}$ respectively — of the energy
difference $U(s)\\!-\\!U(0)$, assuming particles are displaced a distance $s$
along the generalized direction defined by $\bm{\pi}$ about the local minimum
of the potential energy $U$. In Fig. 4, we scatter-plot $b_{2}^{2}$ vs. the
products $b_{2}b_{4}$ for QLEs of different $\delta$-ensembles; from stability
considerations, one expects $b_{3}^{2}\\!\leq\\!3b_{2}b_{4}$, which means that
the system sits in a single-well local potential, or in the stable well
amongst a double-well local potential, see further discussions e.g. in [39,
1].

---
Figure 5: Relations between quantifiers of mechanical disorder. (a) $G_{\rm
na}/G$ vs. $\chi$ for disordered crystals and structural glasses, as indicated
by the legend. The two data sets appear to approximately form a single
function, see text for further discussion. (b) The dimensionless nonphononic
VDoS prefactor $A_{\rm g}\omega_{0}^{5}$ vs. $\chi$ for disordered crystals
and structural glasses (the same symbols as in panel (a)). In sharp constract
to the results of panel (a), we find that the two classes of disordered solids
follow very different curves. In particular, the disordered crystals appear to
have anomalously low $\chi$ values relative to the number of QLEs they host.

We find that QLEs in disordered crystals with $\delta\\!=\\!0.38$ typically
reside quite below the stability bound $b_{3}^{2}\\!=\\!3b_{2}b_{4}$
(represented by the continuous lines in Fig. 4), but less so as $\delta$ is
increased. For $\delta\\!=\\!0.44$, we find that in fact a small fraction of
QLEs violate the stability bound. This scenario is reminiscent of the behavior
of QLEs observed in structural glasses with various degrees of stability, as
shown in [39]. There, glasses quenched from very deeply supercooled states
featured QLEs for which typically $b_{3}^{2}\\!<\\!3b_{2}b_{4}$, while glasses
made by quenching high-energy liquid states featured a sizeable fraction QLEs
that violate this stability bound. Also similar to the phenomenology of
structural glasses is the appearance of a lower bound of the form
$b_{3}^{2}\\!\propto\\!b_{2}b_{4}$, but with a smaller prefactor compared to
the aforementioned stability bound. These two bounds suggest the existence of
a deep relation between excitations’ frequencies and their associated cubic
anharmonicity, as discussed at length in [39, 1].

##  V Mechanical-disorder quantifiers

Having characterized the emergent nonphononic VDoS in disordered crystals and
the properties of the underlying excitations, we turn now to examining how
dimensionless quantifiers of mechanical disorder relate to each other in
disordered crystals, and how those compare to the corresponding relations in
structural glasses. For example, a scaling relation between $A_{\rm g}$ and
the strength of spatial fluctuation of the shear modulus quantified by $\chi$
has been demonstrated in computer structural glasses [34, 35, 36]. The
generality of such relations between different measures of mechanical disorder
remains largely unexplored across various classes of disordered solids. In
order to shed light on this important issue, we employ — in addition to our
disordered crystals dataset — a dataset of a simple glass-forming model (see
Appendix C for details) that can be supercooled down to extremely low
temperatures using the Swap Monto-Carlo algorithm [41], spanning a broad range
of mechanical disorder.

Our results for the interrelations between the dimensionless quantifiers of
mechanical disorder $G_{\rm na}/G$, $\chi$ and $A_{\rm g}$ are presented in
Fig. 5. In Fig. 5a, we plot $G_{\rm na}/G$ vs. $\chi$ for both disordered
crystals and structural glasses. Interestingly, while the data are somewhat
noisy, both datasets for these two systems appear to lie on the same
approximate power-law $G_{\rm na}/G\\!\sim\\!\chi^{0.6}$. This remarkable
commonality of the two classes of disordered solids suggests that the two
quantifiers in both classes are sensitive to mechanical disorder in a similar
fashion.

The situation is dramatically different, however, when comparing the relation
between the prefactor $A_{\rm g}$ — made dimensionless by multiplying it by
$\omega_{0}^{5}$ — and $\chi$, considered in both disordered crystals and
structural glasses, as done in Fig. 5b. Here, we find that these two classes
of disordered solids exhibit very different $A_{\rm g}\omega_{0}^{5}$–$\chi$
relations. Most notably, for similar values of the dimensionless prefactor
$A_{g}\omega_{0}^{5}$ — which encodes information about both the number and
characteristic frequency of QLEs [33, 14] —, $\chi$ of disordered crystals is
significantly smaller. Put differently, we observe that disordered crystals
appear to feature many more QLEs than expected from the
$A_{g}\omega_{0}^{5}$–$\chi$ relation of structural glasses. Finally, we note
that in a recent work it was found that $\chi$ of a broad variety of
(computer) structural glasses appears to be bounded from below by
$\chi_{0}\\!\approx\\!0.85$ [42], marked by the vertical, dash-dotted line in
Fig. 5b. Interestingly, in our disordered crystals, for the range of $\delta$
studied here, $\chi$ breaks this apparent bound by a significant factor.

##  VI Discussion and outlook

In this work, we theoretically studied the emergence of glass-like properties
in disordered crystals, most notably the emergence of low-energy,
quasilocalized excitations (QLEs), whose existence has been recently
established in structural glasses quenched from a melt [13, 12, 10, 1]. By
continuously varying the degree of mechanical disorder in disordered crystals
in a model that features an amorphization transition, we found that below the
transition — where the disordered crystals feature finite long-range order —
QLEs emerge and their frequencies follow the universal $\sim\\!\omega^{4}$
nonphononic VDoS as seen in many classes of structural glasses [13, 12, 10, 1]
and other models [43, 6, 7, 9, 44, 45, 46]. As such, our results extend the
degree of universality of the $\omega^{4}$ law to include disordered crystals.

We showed that the non-universal prefactor $A_{\rm g}$ of the universal
$\sim\\!\omega^{4}$ VDoS, which is related to the number of QLEs [14], varies
substantially with a control parameter of the model, similarly to an
apparently related variation observed in structural glasses [14, 15]. We
further found that the micromechanical properties of QLEs in disordered
crystals, e.g. their spatial localization properties, resemble those observed
in structural glasses, and that they satisfy a similar stability bound.
Finally, considering interrelations between different quantifiers of
mechanical disorder in both disordered crystals and structural glasses,
striking similarities and differences emerged. We discovered that the relation
between a measure of elastic nonaffinity, $G_{\rm na}/G$ (cf. (7)), and a
measure of shear modulus fluctuations, $\chi$ (cf. (6)), appears to form a
unique function for both disordered crystals and structural glasses. On the
other hand, not only disordered crystals feature $\chi$ values lower than the
lowest value ever obtained in structural glasses [42], but their $A_{\rm
g}(\chi)$ relation also implies that they host many more QLEs than the
corresponding structural glasses of the same $\chi$.

The extended validity of the universal glassy $\sim\\!\omega^{4}$ VDoS that
includes also disordered crystals, as established in this work, appears to be
intimately related to known glass-like anomalies experimentally observed in
laboratory disordered crystals at cryogenic temperatures [22, 23, 24, 25, 26,
27, 28, 29, 30, 31]. Most notably, our findings regarding the low-frequency
$\sim\\!\omega^{4}$ VDoS appear to be related to the existence of a
$\sim\\!T^{5}$ contribution to the specific heat at low temperatures $T$ in
disordered crystals [27, 28, 29, 30, 31], in addition to a $\sim\\!T$
contribution associated with two-level systems [47, 48, 49] and Debye’s
$\sim\\!T^{3}$ contribution of phonons [50]. They may also be related to low-
temperature anomalies in the heat conductivity [26] and vibrational anomalies,
such as the emergence of a Boson peak [30], documented in disordered crystals.
Overall, our findings support the experimentally-inferred suggestion that the
absence of long-range order is neither sufficient nor necessary for the
existence of the low-energy excitations [26, 25].

In order to explore in more quantitative terms the relations between the model
studied in this work and laboratory disordered crystals, future work should
more realistically represent the interactions between different species (e.g.
between impurity and host atoms, captured here by the parameter $\delta$) and
vary the species concentration $x$ (fixed here at $x\\!=\\!1/2$), which is the
key control parameter in laboratory binary disordered crystals. In particular,
as the prefactor of the $\sim\\!T^{5}$ contribution to the specific heat would
be proportional to the prefactor $A_{\rm g}$ of the $\sim\\!\omega^{4}$ VDoS,
such future studies would need to resolve $A_{\rm g}(\delta,x)$ (the results
for $A_{\rm g}(\delta,x\\!=\\!1/2)$, for $\delta$ defined in (2), are shown in
Fig. 3b). Likewise, extensions to multi-component disordered crystals can be
explored.

Our results also raise basic questions about disordered solids in a broader
context. One class of such questions is concerned with the identification of
the essential ingredients necessary for the emergence of universal nonphononic
excitations in disordered solids. In this context, we note that our disordered
crystals results appear to echo the theoretical framework recently developed
in [51]. There, it was shown that the self-consistent introduction of
positional disorder and internal stresses (generated also in our model) into
small groups of particles that feature perfect crystalline order — termed
“minimal complexes” — may lead to the emergence of the $\sim\\!\omega^{4}$
VDoS [51]. Future work should clarify whether indeed such a connection between
our findings and those of [51] exist, and if so, explore its possible
implications. It would be also interesting to explore the relations between
our findings and the model of [45], which a priori assumes the existence of
soft quasilocalized excitations. Finally, another interesting question is
concerned with the origin and degree of universality of the approximate
relation $G_{\rm na}/G\\!\sim\\!\chi^{0.6}$, observed in Fig. 5a to hold in
both disordered crystals and structural glasses, which is yet another topic
for future investigation.

###### Acknowledgements.

We warmly thank Geert Kapteijns and Jeppe Dyre for discussions that led to
this work. E.L. acknowledges support from the NWO (Vidi grant no.
680-47-554/3259). E.B. acknowledges support from the Ben May Center for
Chemical Theory and Computation and the Harold Perlman Family.

##  Appendix A Disordered crystals model

We employ a model of $N$ equal-mass, point-like particles in three dimensions,
which derives from the model studied in [16, 17, 18]. A fraction $x\\!=\\!1/2$
of the particles are randomly selected and labelled, to be effectively
“inflated” in order to introduce quenched microscopic disorder into the
system, as explained below. Pairs of particles interact via a Lennard-Jones
pairwise potential of the form

$\varphi_{\mbox{\tiny
LJ}}(r_{ij},\sigma_{ij})=4\varepsilon\bigg{[}\big{(}{\textstyle\frac{\sigma_{ij}}{r_{ij}}}\big{)}^{12}-\big{(}{\textstyle\frac{\sigma_{ij}}{r_{ij}}}\big{)}^{6}+\sum_{\ell=0}^{2}c_{2\ell}\big{(}{\textstyle\frac{r_{ij}}{\sigma_{ij}}}\big{)}^{2\ell}\bigg{]}\,,$
(1)

for $r_{ij}/\sigma_{ij}\\!<\\!2.5$, and $\varphi_{\mbox{\tiny
LJ}}(r_{ij},\sigma_{ij})\\!=\\!0$ otherwise. Here $\varepsilon$ sets the
microscopic units of energy, and the length parameters $\sigma_{ij}$ are set
as

$\sigma_{ij}=\left\\{\begin{array}[]{cc}\lambda_{\mbox{\tiny SS}}&\mbox{both
$i,j$ are unlabelled}\\\ \delta(\lambda_{\mbox{\tiny SL}}-\lambda_{\mbox{\tiny
SS}})+\lambda_{\mbox{\tiny SS}}&\mbox{either $i$ or $j$ are labelled}\\\
\delta(\lambda_{\mbox{\tiny LL}}-\lambda_{\mbox{\tiny
SS}})+\lambda_{\mbox{\tiny SS}}&\mbox{both $i$ and $j$ are
labelled}\end{array}\right.\,.$ (2)

We chose the length parameter $\lambda_{\mbox{\tiny SS}}$ as the microscopic
unit of length, and set $\lambda_{\mbox{\tiny
SL}}\\!=\\!1.18\lambda_{\mbox{\tiny SS}}$ and $\lambda_{\mbox{\tiny
LL}}\\!=\\!1.4\lambda_{\mbox{\tiny SS}}$. The parameter $\delta$, which
controls the effective interaction of labelled particles, determines the
degree of microscopic disorder: $\delta\\!=\\!0$ corresponds to a mono-
component system, whereas $\delta\\!=\\!1$ corresponds to a 50:50 binary
mixture of ‘large’ and ‘small’ particles of 1.4:1 ratio size, as used in
popular glass-forming models. The coefficients
$c_{4}\\!=\\!\mbox{6.201261686784e-04}$,
$c_{2}\\!=\\!\mbox{-9.70155098112e-03}$ and
$c_{0}\\!=\\!\mbox{4.0490237952e-02}$ are determined such that the potential
is continuous up to two derivatives at the dimensionless cutoff
$r^{c}_{ij}/\sigma_{ij}\\!=\\!2.5$.

We construct disordered crystals by setting $\delta\\!=\\!0$ and placing our
$N$ particles on a fcc lattice. Then, we increment the parameter $\delta$ and
minimize the potential energy by a standard nonlinear conjugate-gradient
method, combined with a Berendsen-like barostat [52, 53] fixing the pressure
at zero. This minimization step introduces positional disorder and internal
stresses into the system. For $\delta\\!<\\!0.35$, we use increments of
$0.05$, whereas for higher $\delta$’s the increments are set to $0.01$. We
studied a wide range of system sizes, from $N\\!=\\!1372$ to $N\\!=\\!108000$.

##  Appendix B Observables

The observables we consider are detailed next; we consider vibrational modes’
frequencies $\omega$ obtained via the eigenvalue equation

$\mbox{\boldmath${\cal
H}$}\cdot\bm{\psi}_{\omega}=\omega^{2}\bm{\psi}_{\omega}\,,$ (3)

where $\mbox{\boldmath${\cal H}$}\\!=\\!\frac{\partial^{2}U}{\partial\mathitbf
x\partial\mathitbf x}$ with $U\\!=\\!\sum_{i<j}\varphi_{\mbox{\tiny
LJ}}(r_{ij},\sigma_{ij})$ being the potential energy and $\mathitbf x$ the
particles’ spatial coordinates. $\bm{\psi}_{\omega}$ is the eigenvector
pertaining to the eigenvalue $\omega^{2}$, and we note that all masses are set
to unity. The harmonic (linear) normal modes $\bm{\psi}_{\omega}$ are used to
determine the VDoS.

The properties of QLEs are studied using the nonlinear framework discussed
e.g. in [38, 39]. Within this framework, QLEs are solutions $\bm{\pi}$ to the
equation

$\mbox{\boldmath${\cal H}$}\cdot\bm{\pi}=\frac{\mbox{\boldmath${\cal
H}$}:\bm{\pi}\bm{\pi}}{{\bm{U}^{\prime\prime\prime}}:\\!\cdot\,\bm{\pi}\bm{\pi}\bm{\pi}}{\bm{U}^{\prime\prime\prime}}:\bm{\pi}\bm{\pi}\,,$
(4)

where ${\bm{U}^{\prime\prime\prime}}\\!\equiv\\!\frac{\partial
U}{\partial\mathitbf x\partial\mathitbf x\partial\mathitbf x}$ denotes the
rank-3 tensor of derivatives of the potential energy, and
$\cdot,:,:\\!\cdot\,$ denote single, double or triple contractions over
particle indices and Cartesian components, respectively. In [38, 39], it was
shown that QLEs defined via Eq. (4) closely resemble quasilocalized
vibrational (harmonic) modes, in the absence of hybridizations with low-
frequency phonons. Low-energy QLEs were calculated as explained in Appendix B
of [39], which presumably provides the single, lowest-energy QLE in a given
disordered solid or glass sample. The spatial decay of QLEs was calculated
following [13].

We considered the athermal shear and bulk moduli, denoted $G$ and $K$
respectively, whose precise definitions can be found e.g. in Appendix A of
[11]. We also considered the widely studied Poisson’s ratio $\nu$, given by

$\nu=\frac{3K-2G}{6K+2G}\,.$ (5)

In order to further quantify mechanical disorder, we consider the sample-to-
sample relative fluctuations of the shear modulus, namely

$\chi\equiv\frac{\sqrt{N\overline{(G-\overline{G})^{2}}}}{\overline{G}}\,,$
(6)

where $\overline{\bullet}$ denotes the sample-to-sample average. The
quantifier $\chi$ was discussed at length in [54, 55, 35, 11, 56]; it has been
shown to determine wave attenuation rates in the harmonic regime [55, 56].
Here, we use a jackknife-like method to approximate $\chi$, see details in
[35].

Finally, we considered another dimensionless quantifier of mechanical disorder
defined as the ratio between what is known as the nonaffine term of the shear
modulus — denoted $G_{\rm na}$ — to the total shear modulus $G$, namely

$\frac{G_{\rm
na}}{G}=\frac{\frac{\partial^{2}U}{\partial\gamma\partial\mathitbf
x}\cdot\mbox{\boldmath${\cal
H}$}^{-1}\cdot\frac{\partial^{2}U}{\partial\mathitbf x\partial\gamma}}{VG}\,,$
(7)

where $\gamma$ is the shear-strain parameter as defined e.g. in [11].

##  Appendix C Structural glass model

In order to compare the behavior of our disordered crystals to structural
glasses formed by quenching a melt, we employed the dataset of soft-sphere
structural glasses quenched from a broad range of equilibrium temperatures
$T_{\rm p}$ as described in [14, 35, 40]. This model can be equilibrated down
to very low temperatures using the Swap-Monte-Carlo method [41]. Our glass
ensembles consist of 2000 independent samples of $N\\!=\\!16384$ particles
each.

## References

  * Lerner and Bouchbinder [2021] E. Lerner and E. Bouchbinder, Low-energy quasilocalized excitations in structural glasses, J. Chem. Phys. 155, 200901 (2021).
  * Widmer-Cooper  _et al._ [2008] A. Widmer-Cooper, H. Perry, P. Harrowell, and D. R. Reichman, Irreversible reorganization in a supercooled liquid originates from localized soft modes, Nature Phys. 4, 711 (2008).
  * Widmer-Cooper  _et al._ [2009] A. Widmer-Cooper, H. Perry, P. Harrowell, and D. R. Reichman, Localized soft modes and the supercooled liquid’s irreversible passage through its configuration space, J. Chem. Phys. 131, 194508 (2009).
  * Kapteijns  _et al._ [2021a] G. Kapteijns, D. Richard, E. Bouchbinder, T. B. Schrøder, J. C. Dyre, and E. Lerner, Does mesoscopic elasticity control viscous slowing down in glassforming liquids?, J. Chem. Phys. 155, 074502 (2021a).
  * Lerbinger  _et al._ [2021] M. Lerbinger, A. Barbot, D. Vandembroucq, and S. Patinet, On the relevance of shear transformations in the relaxation of supercooled liquids, arXiv preprint arXiv:2109.12639 (2021).
  * Rainone  _et al._ [2021] C. Rainone, P. Urbani, F. Zamponi, E. Lerner, and E. Bouchbinder, Mean-field model of interacting quasilocalized excitations in glasses, SciPost Phys. Core 4, 8 (2021).
  * Bouchbinder  _et al._ [2021] E. Bouchbinder, E. Lerner, C. Rainone, P. Urbani, and F. Zamponi, Low-frequency vibrational spectrum of mean-field disordered systems, Phys. Rev. B 103, 174202 (2021).
  * Folena and Urbani [2021] G. Folena and P. Urbani, Marginal stability of local energy minima in soft anharmonic mean field spin glasses, arXiv preprint arXiv:2106.16221 (2021).
  * Franz  _et al._ [2022] S. Franz, F. Nicoletti, G. Parisi, and F. Ricci-Tersenghi, Delocalization transition in low energy excitation modes of vector spin glasses, SciPost Phys. 12, 16 (2022).
  * Richard  _et al._ [2020] D. Richard, K. González-López, G. Kapteijns, R. Pater, T. Vaknin, E. Bouchbinder, and E. Lerner, Universality of the nonphononic vibrational spectrum across different classes of computer glasses, Phys. Rev. Lett. 125, 085502 (2020).
  * González-López  _et al._ [2021] K. González-López, M. Shivam, Y. Zheng, M. P. Ciamarra, and E. Lerner, Mechanical disorder of sticky-sphere glasses. I. effect of attractive interactions, Phys. Rev. E 103, 022605 (2021).
  * Kapteijns  _et al._ [2018] G. Kapteijns, E. Bouchbinder, and E. Lerner, Universal nonphononic density of states in 2D, 3D, and 4D glasses, Phys. Rev. Lett. 121, 055501 (2018).
  * Lerner  _et al._ [2016] E. Lerner, G. Düring, and E. Bouchbinder, Statistics and properties of low-frequency vibrational modes in structural glasses, Phys. Rev. Lett. 117, 035501 (2016).
  * Rainone  _et al._ [2020] C. Rainone, E. Bouchbinder, and E. Lerner, Pinching a glass reveals key properties of its soft spots, Proc. Natl. Acad. Sci. U.S.A. 117, 5228 (2020).
  * Wang  _et al._ [2019] L. Wang, A. Ninarello, P. Guan, L. Berthier, G. Szamel, and E. Flenner, Low-frequency vibrational modes of stable glasses, Nat. Commun. 10, 26 (2019).
  * Mizuno  _et al._ [2013] H. Mizuno, S. Mossa, and J.-L. Barrat, Elastic heterogeneity, vibrational states, and thermal conductivity across an amorphisation transition, Europhys. Lett. 104, 56001 (2013).
  * Mizuno  _et al._ [2014] H. Mizuno, S. Mossa, and J.-L. Barrat, Acoustic excitations and elastic heterogeneities in disordered solids, Proc. Natl. Acad. Sci. U.S.A. 111, 11949 (2014).
  * Mizuno  _et al._ [2016] H. Mizuno, S. Mossa, and J.-L. Barrat, Relation of vibrational excitations and thermal conductivity to elastic heterogeneities in disordered solids, Phys. Rev. B 94, 144303 (2016).
  * Tong  _et al._ [2015] H. Tong, P. Tan, and N. Xu, From crystals to disordered crystals: A hidden order-disorder transition, Scientific Reports 5, 15378 (2015).
  * Tan  _et al._ [2021] X. Tan, Y. Guo, D. Huang, and L. Zhang, A structural approach to vibrational properties ranging from crystals to disordered systems, Soft Matter 17, 1330 (2021).
  * Jiang and Xu [2022] Z. Jiang and N. Xu, Emergent physics on the route from crystals to amorphous solids, Chin. J. Chem. 40, 1091 (2022).
  * Ackerman  _et al._ [1981] D. A. Ackerman, D. Moy, R. C. Potter, A. C. Anderson, and W. N. Lawless, Glassy behavior of crystalline solids at low temperatures, Phys. Rev. B 23, 3886 (1981).
  * Anderson [1985] A. C. Anderson, Low temperature glassy properties of disordered crystals, Phase Transit. 5, 301 (1985).
  * Cahill  _et al._ [1992] D. G. Cahill, S. K. Watson, and R. O. Pohl, Lower limit to the thermal conductivity of disordered crystals, Phys. Rev. B 46, 6131 (1992).
  * Ramos  _et al._ [1997] M. A. Ramos, S. Vieira, F. J. Bermejo, J. Dawidowski, H. E. Fischer, H. Schober, M. A. González, C. K. Loong, and D. L. Price, Quantitative assessment of the effects of orientational and positional disorder on glassy dynamics, Phys. Rev. Lett. 78, 82 (1997).
  * Pohl  _et al._ [2002] R. O. Pohl, X. Liu, and E. Thompson, Low-temperature thermal conductivity and acoustic attenuation in amorphous solids, Rev. Mod. Phys. 74, 991 (2002).
  * Ramos  _et al._ [2003] M. Ramos, C. Talón, R. Jiménez-Riobóo, and S. Vieira, Low-temperature specific heat of structural and orientational glasses of simple alcohols, J. Phys. Condens. Matter. 15, S1007 (2003).
  * Vdovichenko  _et al._ [2015] G. A. Vdovichenko, A. I. Krivchikov, O. A. Korolyuk, J. L. Tamarit, L. C. Pardo, M. Rovira-Esteva, F. J. Bermejo, M. Hassaine, and M. A. Ramos, Thermal properties of halogen-ethane glassy crystals: Effects of orientational disorder and the role of internal molecular degrees of freedom, J. Chem. Phys. 143, 084510 (2015).
  * Gebbia  _et al._ [2017] J. F. Gebbia, M. A. Ramos, D. Szewczyk, A. Jezowski, A. I. Krivchikov, Y. V. Horbatenko, T. Guidi, F. J. Bermejo, and J. L. Tamarit, Glassy anomalies in the low-temperature thermal properties of a minimally disordered crystalline solid, Phys. Rev. Lett. 119, 215506 (2017).
  * Moratalla  _et al._ [2019] M. Moratalla, J. F. Gebbia, M. A. Ramos, L. C. Pardo, S. Mukhopadhyay, S. Rudić, F. Fernandez-Alonso, F. J. Bermejo, and J. L. Tamarit, Emergence of glassy features in halomethane crystals, Phys. Rev. B 99, 024301 (2019).
  * Ramos [2020] M. A. Ramos, Are universal ”anomalous” properties of glasses at low temperatures truly universal?, Low Temp. Phys. 46, 104 (2020).
  * Lerner [2020] E. Lerner, Finite-size effects in the nonphononic density of states in computer glasses, Phys. Rev. E 101, 032120 (2020).
  * Lerner and Bouchbinder [2018] E. Lerner and E. Bouchbinder, A characteristic energy scale in glasses, J. Chem. Phys. 148, 214502 (2018).
  * Richard  _et al._ [2021] D. Richard, E. Lerner, and E. Bouchbinder, Brittle-to-ductile transitions in glasses: Roles of soft defects and loading geometry, MRS Bulletin 46, 902 (2021).
  * Kapteijns  _et al._ [2021b] G. Kapteijns, E. Bouchbinder, and E. Lerner, Unified quantifier of mechanical disorder in solids, Phys. Rev. E 104, 035001 (2021b).
  * Lerner [2022] E. Lerner, What controls the poisson’s ratio of highly incompressible metallic glasses?, arXiv preprint arXiv:2202.10993 (2022).
  * Ji  _et al._ [2020] W. Ji, T. W. J. de Geus, M. Popović, E. Agoritsas, and M. Wyart, Thermal origin of quasilocalized excitations in glasses, Phys. Rev. E 102, 062110 (2020).
  * Gartner and Lerner [2016] L. Gartner and E. Lerner, Nonlinear modes disentangle glassy and Goldstone modes in structural glasses, SciPost Phys. 1, 016 (2016).
  * Kapteijns  _et al._ [2020] G. Kapteijns, D. Richard, and E. Lerner, Nonlinear quasilocalized excitations in glasses: True representatives of soft spots, Phys. Rev. E 101, 032130 (2020).
  * Lerner [2019] E. Lerner, Mechanical properties of simple computer glasses, J. Non-Cryst. Solids 522, 119570 (2019).
  * Ninarello  _et al._ [2017] A. Ninarello, L. Berthier, and D. Coslovich, Models and algorithms for the next generation of glass transition studies, Phys. Rev. X 7, 021039 (2017).
  * González-López  _et al._ [2020] K. González-López, E. Bouchbinder, and E. Lerner, Does a universal lower bound on glassy mechanical disorder exist?, arXiv preprint arXiv:2012.03634 (2020).
  * Baity-Jesi  _et al._ [2015] M. Baity-Jesi, V. Martín-Mayor, G. Parisi, and S. Perez-Gaviro, Soft modes, localization, and two-level systems in spin glasses, Phys. Rev. Lett. 115, 267205 (2015).
  * Gurarie and Chalker [2003] V. Gurarie and J. T. Chalker, Bosonic excitations in random media, Phys. Rev. B 68, 134207 (2003).
  * Gurevich  _et al._ [2003] V. L. Gurevich, D. A. Parshin, and H. R. Schober, Anharmonicity, vibrational instability, and the boson peak in glasses, Phys. Rev. B 67, 094203 (2003).
  * Das  _et al._ [2020] P. Das, H. G. E. Hentschel, E. Lerner, and I. Procaccia, Robustness of density of low-frequency states in amorphous solids, Phys. Rev. B 102, 014202 (2020).
  * Zeller and Pohl [1971] R. C. Zeller and R. O. Pohl, Thermal conductivity and specific heat of noncrystalline solids, Phys. Rev. B 4, 2029 (1971).
  * Anderson  _et al._ [1972] P. W. Anderson, B. I. Halperin, and C. M. Varma, Anomalous low-temperature thermal properties of glasses and spin glasses, Philos. Mag. 25, 1 (1972).
  * Phillips [1972] W. Phillips, Tunneling states in amorphous solids, J. Low Temp. Phys. 7, 351 (1972).
  * Kittel [2005] C. Kittel,  _Introduction to solid state physics_ (Wiley, 2005).
  * Moriel [2021] A. Moriel, Internally stressed and positionally disordered minimal complexes yield glasslike nonphononic excitations, Phys. Rev. Lett. 126, 088004 (2021).
  * Berendsen  _et al._ [1984] H. J. C. Berendsen, J. P. M. Postma, W. F. van Gunsteren, A. DiNola, and J. R. Haak, Molecular dynamics with coupling to an external bath, J. Chem. Phys. 81, 3684 (1984).
  * Allen and Tildesley [1989] M. P. Allen and D. J. Tildesley,  _Computer simulation of liquids_ (Oxford university press, 1989).
  * Bouchbinder and Lerner [2018] E. Bouchbinder and E. Lerner, Universal disorder-induced broadening of phonon bands: from disordered lattices to glasses, New J. Phys. 20, 073022 (2018).
  * Kapteijns  _et al._ [2021c] G. Kapteijns, D. Richard, E. Bouchbinder, and E. Lerner, Elastic moduli fluctuations predict wave attenuation rates in glasses, J. Chem. Phys. 154, 081101 (2021c).
  * Mahajan and Ciamarra [2021] S. Mahajan and M. P. Ciamarra, Unifying description of the vibrational anomalies of amorphous materials, Phys. Rev. Lett. 127, 215504 (2021).


The work in this thesis has been carried out under the auspices of the Center for Language
and Cognition Groningen (CLCG) of the Faculty of Arts of the University of Groningen.

Groningen Dissertations in Linguistics 164

ISSN: 0928-0030

ISBN: 978-94-034-0224-6 (printed version)

ISBN: 978-94-034-0223-9 (electronic version)

2017, Johannes Bjerva

Document prepared with  and typeset by pdf
(Droid Serif and Lato fonts)

Cover art: Cortical Columns.  2014, Greg Dunn

21K, 18K, and 12K gold, ink, and dye on aluminized panel.

Printed by Off Page (<www.offpage.nl>) on G-print 115g paper.

One Model to Rule them all

Multitask and Multilingual Modelling
for Lexical Analysis

ter verkrijging van de graad van doctor aan de

Rijksuniversiteit Groningen

op gezag van de

rector magnificus prof. dr. E. Sterken

en volgens besluit van het College voor Promoties.

De openbare verdediging zal plaatsvinden op

donderdag 7 december 2017 om 14.30 uur

Johannes Bjerva

geboren op 21 maart 1990

te Oslo, Noorwegen

Prof. dr. ing. J. Bos

Dr. B. Plank

Prof. dr. A. Søgaard

Prof. dr. J. Tiedemann

Prof. dr. L. R. B. Schomaker

CHAPTER: ACKNOWLEDGEMENTS

This has been a bumpy ride, to say the least, and having reached the end of this four-year journey, I owe a debt of gratitude to all of my friends, family, and colleagues.
Your support and guidance has certainly helped smooth out most of the ups and downs I've experienced.

First of all, I would like to thank my PhD supervisors.
Johan, the freedom you allowed me during the past four years has been one of the things I've appreciated the most in the whole experience.
This meant that I could pursue the track of research I felt was the most interesting, without which writing a whole book would have been much more arduous – thank you!

Barbara, thank you for agreeing to join as co-supervisor so late in my project, and for putting in so much time during the last couple of months of my PhD.
The thesis would likely have looked quite different if you hadn't started in Groningen when you did.
I owe you a huge thanks, especially for the final weeks – reading and commenting on the whole thesis in less than 24 hours during your vacation. Just, wow!

Next, I would like to thank Anders Søgaard, Jörg Tiedemann, and Lambert Schomaker for agreeing to form my thesis assessment committee.
I feel honoured that you took the time to read this thesis, and that you deemed it scientifically sound.
I also want to thank everyone who I have collaborated with throughout these years, both in Groningen and in Stockholm.
Thanks Calle for agreeing to work with a sign language novice such as myself.
Raf, it was an enlightening experience to work with you and see the world of `real' humanities research.
Robert, we should definitely continue with our one-week shared task submissions (with various degrees of success).

Most of the last few years were spent at the computational linguistics group in Groningen.
I would especially like to thank all of my fellow PhD students throughout the years.
Thanks Kilian, Noortje, Valerio, and Harm for welcoming me with open arms when I joined the group.
Special thanks to Kilian for being so helpful with answering all of my billions and billions of questions involved in finishing this thesis.
Also, a special thanks to both Noortje and Harm for the times we shared when I had just moved here (especially that first New Year's eve!).
Hessel, thanks for all the help with administrative matters while I was abroad, especially for sending a gazillion of travel declarations for me.
Rik and Anna, it was great getting to know you both better during the last few months – hopefully you find the bookmark to be sufficiently sloth-y.
Dieke and Rob, thanks for being such great laid back drinking buddies and travel companions.
To all of you, and Pauline – I hope we will continue the tradition of going all out whenever I come to visit Groningen!

A big thanks to the rest of the computational linguistics group, especially Gertjan, Malvina, Gosse, John, Leonie, and Martijn.
Also thanks to all other PhD students who started with me: Luis, Simon, Raf, Ruben, Aynur, Jonne, and everyone else whose name I've failed to mention.
A special thanks to Ben and Kaja - I hope we keep up our board-game centred visits to one another, no matter where we happen to pursue our careers.

I spent most of my final year in Stockholm, and it was great being in that relaxed atmosphere during one of the more intense periods of the past few years.
Most of all, I am sincerely grateful to Calle, to Johan (and Klara, Iris, and Vive), and to Bruno.
Your support is truly invaluable, and by my lights, there is not much more to say than <dank>:(since a pal's always needed).
I'd also like to thank Johan and Calle especially for agreeing to observing weird Dutch traditions by being my paranymphs.
Josefina, Elísabet, and David, thank you all for being there for me during the past year.
Thanks to the computational linguistics group for hosting me during this period, especially Mats, Robert, Kina, and Gintarė.
Finally, thanks to everyone else at the Department of Linguistics at Stockholm University.

Having moved to Copenhagen this autumn has been an extremely pleasant experience.
I would like to thank the entire CoAStal NLP group for simply being the coolest research group there is.
Especially, I'd like to thank Isabelle for making the whole process of moving to Denmark so easy.
Thanks to Anders, Maria, Mareike, Joachim, Dirk, and Ana for being so welcoming.
I'd also like to thank everyone else at the image section at DIKU, especially the running buddies at the department, and most especially Kristoffer and Niels.

Finally, I would like to thank my family.
This bumpy ride would have been challenging to get through without their support through everything.
Kiitos, mamma! Takk/tack Aksel, Paulina, Julian, och Lucas!
Takk Olav, og takk Amanda!

Let's see where the journey goes next!

Copenhagen, November 2017

In Natural Language Processing (NLP), we are generally concerned with the automatic analysis of natural language, often with the goal of solving a particular natural language task.
The traditional approach is to consider a single task for a single language at a time.

However, neural networks are particularly well suited for broadening this approach, by considering data for multiple tasks and languages simultaneously, and have become one of the most popular techniques in NLP in recent years.
This is an important approach to explore further, as the key to improving the reliability of NLP, especially for low-resource languages, is to take advantage of all relevant data, whenever possible.
In Part I of this thesis, we begin with going through a crash-course in (deep) neural networks with a focus on NLP (Chapter 2).
We will then look at some ways in which neural networks can consider multiple tasks and languages at the same time (Chapter 3).
Specifically, we will consider multitask learning (MTL), and several common multilingual approaches.

In Part II of this thesis, I look at exploiting the fact that many NLP tasks are highly related to one another.
This is done by experimenting with MTL using hard parameter sharing, which has proven beneficial for a variety of NLP tasks.
In spite of such successes, however, it is not clear when or why MTL is beneficial in NLP.
Chapter 4 contains a case study in which semantic tagging is shown to be beneficial for POS tagging.
This further highlights the question of when MTL is beneficial in NLP tagging tasks, which is explored using information-theoretic measures in Chapter 5.

Multilingual models can leverage the fact that many languages share commonalities with one another.
These resemblances can occur on various levels, with languages sharing, for instance, syntactic, morphological, or lexical features.

While there are many possibilities for exploiting these commonalities, the focus in this thesis is on using multilingual word representations, as they allow for straight-forward integration in a neural network.
As with MTL, it is not clear in which cases it is an advantage to go multilingual.
In Part III, I begin with presenting a case study on multilingual semantic textual similarity (Chapter 6).
Following this, I explore how similar languages need to be, and in which way, in order to for it to be useful to go multilingual (Chapter 7).

In Part IV of this thesis, I experiment with a combined paradigm, in which a neural network is trained on several languages and tasks simultaneously (Chapter 8).

Finally, the thesis is concluded in Part V (Chapter 9).
The experiments in this thesis are run on a large collection of mainly lexically oriented tasks, both semantic and morphosyntactic in nature, and on a total of 60 languages, representing a relatively wide typological range.

CHAPTER: INTRODUCTION

When learning a new skill, you take advantage of your preexisting skills and knowledge.
For instance, if you are a skilled violinist, you will likely have an easier time learning to play cello.
Similarly, when learning a new language you take advantage of the languages you already speak.
For instance, if your native language is Norwegian and you decide to learn Dutch, the lexical overlap between these two languages will likely benefit your rate of language acquisition.
This thesis deals with the intersection of learning multiple tasks and learning multiple languages in the context of Natural Language Processing (NLP), which can be defined as the study of computational processing of human language.
Although these two types of learning may seem different on the surface, we will see that they share many similarities.

Traditionally, NLP practitioners have looked at solving a single problem for a single task at a time.
For instance, considerable time and effort might be put into engineering a system for part-of-speech (PoS) tagging for English.
However, although the focus has been on considering a single task at a time, fact is that many NLP tasks are highly related.
For instance, different lexical tag sets will likely exhibit high correlations with each other.
As an example, consider the following sentence annotated with Universal Dependencies (UD) PoS tags <cit.>, and semantic tags <cit.>.[PMB 01/3421. Original source: Tatoeba. UD tags obtained using UD-Pipe <cit.>][The semantic tag set consists of 72 tags, and is developed for multilingual semantic parsing. The tag set is described further in Chapter <ref>.]

* We must draw attention to the distribution of this form in those dialects .

* We must draw attention to the distribution of this form in those dialects .

While these tag sets are certainly different, the distinctions they make compared to one another in this example are few, as there are only two apparent systematic differences.
Firstly, the semantic tags offer a difference between definite (), proximal (), and distal determiners (), whereas UD lumps these together as  (highlighted in green).
Secondly, the semantic tags also differentiate between relations () and conjunctions (), which are both represented by the  PoS tag, highlighted in blue.
Hence, although these tasks are undoubtedly different, there are considerable correlations between the two, as the rest of the tags exhibit a one-to-one mapping in this example.

This raises the question of how this fact can be exploited, as it seems like a colossal waste to not take advantage of such inter-task correlations.
In this thesis I approach this by exploring multitask learning (MTL, , ), which has been beneficial for many NLP tasks.
In spite of such successes, however, it is not clear when or why MTL is beneficial.

Similarly to how different tag sets correlate with each other, languages also share many commonalities with one another.
These resemblances can occur on various levels, with languages sharing, for instance, syntactic, morphological, or lexical features.
Such similarities can have many different causes, such as common language ancestry, loan words, or being a result of universals and constraints in the properties of natural language itself (see, e.g., ,, and , ).
Consider, for instance, the following German translation of the previous English example, annotated with semantic tags.[PMB 01/3421. Original source: Tatoeba.]

* Wir müssen die Verbreitung dieser Form in diesen Dialekten beachten .

Comparing the English and German annotations, there is a high overlap between the semantic tags used, and a high lexical overlap.
As in the case of related NLP tasks, this begs the question of how multilinguality can be exploited, as it seems like an equally colossal waste to not consider using, e.g., Norwegian PoS data when training a Swedish PoS tagger.
There are several approaches to exploiting multilingual data, such as annotation projection and model transfer, as detailed in Chapter <ref>.
The approach in this thesis is a type of model transfer, in which such inter-language relations are exploited by exploring multilingual word representations, which have also been beneficial for many NLP tasks.
As with MTL, in spite of the fact that such approaches have been successful for many NLP tasks, it is not clear in which cases it is an advantage to go multilingual.

Given the large amount of data available for many languages in different annotations, it is tempting to investigate possibilities of combining the paradigms of multitask learning and multilingual learning in order to take full advantage of this data.
Hence, as the title of the thesis suggests, the final effort in this thesis is to arrive at One Model to rule them all.

This thesis approaches these two related aspects of NLP by experimenting with deep neural networks, which represent a family of learning architectures which are exceptionally well suited for the aforementioned purposes (described in Chapter <ref>).
For one, it is fairly straightforward to implement the sharing of parameters between tasks, thus enabling multitask learning (discussed in Chapter <ref>).
Additionally, providing such an architecture with multilingual input representations is also straightforward (discussed in Chapter <ref>).
Experiments in this thesis are run on a large collection of tasks, both semantic and morphosyntactic in nature, and a total of 60 languages are considered, depending on the task at hand.

§ CHAPTER GUIDE

The thesis is divided into five parts, totalling 9 chapters, aiming to provide answers to the following general research questions (RQs):

RQ <ref> To what extent can a semantic tagging task be informative for other NLP tasks? rq:stag

RQ <ref> How can multitask learning effectivity in NLP be quantified? rq:mtl

RQ <ref> To what extent can multilingual word representations be used to enable zero-shot learning in semantic textual similarity? rq:sts

RQ <ref> In which way can language similarities be quantified, and what correlations can we find between multilingual model performance and language similarities?  rq:multiling

RQ <ref> Can a multitask and multilingual approach be combined to generalise across languages and tasks simultaneously?  rq:mmmt

§.§ Part I – Background

The goal of Part I is to provide the reader with sufficient background knowledge to understand the material in this thesis.
Chapter <ref> contains a crash-course in neural networks, introducing the main concepts and architectures used in NLP.
Chapter <ref> provides an introduction to multitask learning and multilingual learning, which are the two central topics of this work.

§.§ Part II – Multitask Learning

In Part II, the goal is to investigate multitask learning (MTL), in particular by looking at the effects of this paradigm in NLP sequence prediction tasks.
In Chapter <ref> we present a semantic tag set tailored for multilingual semantic parsing.
We attempt to use this tag set as an auxiliary task for PoS tagging, observing what effects this yields, answering RQ <ref>.
Chapter <ref> then delves deeper into MTL, attempting to find when MTL is effective, and how this effectiveness can be predicted by using information-theoretic measures (RQ <ref>).

§.§ Part III – Multilingual Learning

Having looked at similarities between tasks, we turn to similarities between languages in Part III.
In Chapter <ref>, we attempt to make a language-agnostic solution for semantic textual similarity, by exploiting multilingual word representations, thus answering RQ <ref>.
Having seen the results of combining related languages in this task, we try to quantify these effects in Chapter <ref>, aiming to answer RQ <ref>.

§.§ Part IV – Combining Multitask and Multilingual Learning

In Part IV we want to combine the paradigms of multitask learning and multilingual learning in order to make One Model to rule them all.
Chapter <ref> presents a pilot study taking a step in this direction, looking at predicting labels for an unseen task–language combination while exploiting other task–language combinations.

§.§ Part V – Conclusions

Finally, Chapter <ref> contains an overview of the conclusions from this thesis.
In addition to this, we provide an outlook for future work in this direction, in particular focussing on the combined multitask–multilingual paradigm.

§ PUBLICATIONS

This thesis is based on the following publications:

* bjerva:2016:semantic

* bjerva:2017:mtl

* bjerva:2017:sts

* bjerva:2017:sts:semeval

* bjerva:2017:multilingual

Some parts of the thesis may also refer to the following peer-reviewed publications completed in the course of the PhD:

* bjerva:14:eacl

* bjerva:14:semeval

* bjerva:2015

* bjerva:2016:cl4lc

* bjerva:2016:dsl

* gronup

* haagsma:metaphor

* bjerva:2016:amr

* bjerva:jdmdh

* gmb:hla

* pmb

* bjerva:2017:sigmorphon

* sjons:2017

* rug:nli

* bjerva:nli

Multitask Learning

CHAPTER: [CHAPTER ADAPTED FROM:
BJERVA, J., PLANK, B., AND BOS, J. (2016). SEMANTIC TAGGING WITH DEEP RESIDUAL NETWORKS. IN PROCEEDINGS OF COLING 2016, THE 26TH INTERNATIONAL CONFERENCE ON COMPUTATIONAL LINGUISTICS: TECHNICAL PAPERS, PAGES 3531–3541. THE COLING 2016 ORGANIZING COMMITTEE]
MULTITASK SEMANTIC TAGGING
WITH RESIDUAL NETWORKS

CHAPTER: [CHAPTER ADAPTED FROM: BJERVA, J. (2017)
WILL MY AUXILIARY TAGGING TASK HELP? ESTIMATING AUXILIARY TASKS EFFECTIVITY IN MULTI-TASK LEARNING, IN PROCEEDINGS OF THE 21ST NORDIC CONFERENCE ON COMPUTATIONAL LINGUISTICS, NODALIDA, 22-24 MAY 2017, GOTHENBURG, SWEDEN, NUMBER 131, PAGES 216–220. LINKÖPING UNIVERSITY ELECTRONIC PRESS, LINKÖPINGS UNIVERSITET. BEST SHORT-PAPER AWARD.]
INFORMATION-THEORETIC PERSPECTIVES ON MULTITASK LEARNING EFFECTIVITY

Multilingual Learning

CHAPTER: [CHAPTER ADAPTED FROM:
BJERVA, J. AND ÖSTLING, R. (2017A). CROSS-LINGUAL LEARNING OF SEMANTIC TEXTUAL SIMILARITY WITH MULTILINGUAL WORD REPRESENTATIONS. IN PROCEEDINGS OF THE 21ST NORDIC CONFERENCE ON COMPUTATIONAL LINGUISTICS, NODALIDA, 22-24 MAY 2017, GOTHENBURG, SWEDEN, NUMBER 131, PAGES 211–215. LINKÖPING UNIVERSITY ELECTRONIC PRESS, LINKÖPINGS UNIVERSITET.]
SEMANTIC TEXTUAL SIMILARITY

CHAPTER: [CHAPTER ADAPTED FROM: BJERVA, J. (IN REVIEW)
QUANTIFYING THE EFFECTS OF MULTILINGUALITY IN NLP SEQUENCE PREDICTION TASKS.]
QUANTIFYING THE EFFECTS OF MULTILINGUALITY IN NLP SEQUENCE PREDICTION TASKS

Combining Multitask and Multilingual Learning

CHAPTER: ONE MODEL TO RULE THEM ALL:

While traditional NLP approaches consider a single task or language at a time, the aim of this thesis was to answer several research questions dealing with pushing past this boundary.
In doing so, the hope is that in the long term, low-resource languages can benefit from the advances made in NLP which are currently to a large extent reserved for high-resource languages.
This, in turn, may then have positive consequences for, e.g., language preservation, as speakers of minority languages will have a lower degree of pressure to using high-resource languages.
In the short term, answering the specific research questions posed should be of use to NLP researchers working towards the same goal.
We will now see the conclusions which can be drawn from each research part of this thesis.

§ PART II - MULTITASK LEARNING

In the first research part of the thesis, we began by exploring the following research question in Chapter <ref>.

'To what extent can a semantic tagging task be informative for other NLP tasks?'

–RQ <ref>

We found that semantic tags are informative for the task of PoS tagging.
Furthermore, the results obtained when exploiting this were state-of-the-art results at the time.
Additionally, we found that using coarse-grained semantic tags was not informative for semantic tagging.
This then raised more questions.
Why were the semantic tags useful for PoS tags, while coarse-grained semantic tags were not useful for semantic tagging?
A look at correlations between the tag sets showed that these were high in both cases.
Coarse-grained semantic tags have a one-to-one mapping with semantic tags.
Semantic tags do not have a one-to-one mapping with PoS tags, but still exhibit large correlations.
The idea was, then, that such high correlations between tag sets might correlate with auxiliary task effectivity, given differing data sets.
Thus, we aimed at answering the next research question in Chapter <ref>:

'How can multitask learning effectivity in NLP be quantified?'

–RQ <ref>

Taking an information-theoretic perspective, we found that these correlations could be quantified fairly well by using mutual information.
Running experiments in various data overlap settings, on a large selection of languages and tasks, showed that the hypothesis was supported.
That is to say, providing the model with different data including annotations which correlate highly with the main task yields gains in performance.
However, providing the model with the same data with such highly correlated auxiliary annotations, does not yield any increase at all.
Intuitively, this makes sense if one thinks about it as follows.
The model has already seen sentence $x$ with some annotation.
Giving it the same sentence $x$ with highly correlated annotation does not give the model anything more to learn from – after all, this example has practically already been observed!
However, giving the model a different sentence $y$ with highly correlated annotation essentially entails giving the model an extra training example.

§ PART III - MULTILINGUAL LEARNING

In the second content part of the thesis, the aim was to investigate similar research questions to Part I, focussing on similarities between languages rather than between tasks.
We began by asking the following research question.

'To what extent can multilingual word representations be used to enable zero-shot learning in semantic textual similarity?'

–RQ <ref>

In Chapter <ref> we found that a simple language-agnostic feed-forward neural network using multilingual word representations was able to solve the task of semantic textual similarity assessment to some extent.
Although results were below the current state-of-the-art for this task, some useful insights were gained.
Mainly, we found that languages which are more similar to one another are more suited for this approach, indicating that language similarity is important for the effectivity of model multilinguality.
This is similar to the case in MTL, where task relatedness is an important factor, and raised the following research question which we approached in Chapter <ref>.

'In which way can language similarities be quantified, and what correlations can we find between multilingual model performance and language similarities?'

–RQ <ref>

We looked at correlations between language similarity and multilingual model effectivity in two sequence prediction tasks, namely semantic tagging and PoS tagging, as well as in a sequence-to-sequence task, namely morphological inflection.
The overall results indicate that both measures of language similarity under consideration offer some explanatory value.
One interesting finding in the case of semantic tagging, was the fact that English, Dutch, and German benefitted from having their input representations updated during joint training. Combining these languages with Italian and updated embeddings, however, resulted in a serious drop in performance.
A potential reason for this is that language relatedness plays a large role in maintaining the quality of the multilingual embedding space in such a context.
In future work, it would therefore be interesting to observe the resulting word-space after updating word representations in such a setting.

§ PART IV - COMBINING MULTITASK LEARNING AND MULTILINGUALITY

In the final research part of the thesis, the aim was to probe the possibilities of combining the paradigms of multitask learning and multilingual learning.
Chapter <ref> aimed at providing an answer to the following research question.

'Can a multitask and multilingual approach be combined to generalise across languages and tasks simultaneously?'

–RQ <ref>

We looked at predicting labels for an unseen task–language combination, by taking advantage of other task–language combinations.
In the admittedly somewhat artificial setup, the target task was PoS tagging for three languages offering some typological diversity, namely Finnish, Italian, and Slovene.
In a high-resource scenario, assuming access to parallel text similar to Europarl, sensible tags could be produced for the target languages without seeing any annotated data for that target language.
In the low-resource scenario, assuming access to parallel text similar to the New Testament, similar results have the additional requirement of also having access to target-language annotations of some sort.
Finally, access to the high-resource scenario as well as target-language annotations yielded results on par with a monolingual monotask PoS tagger for the target language – and that without seeing a single PoS tag for the target language.

§ FINAL WORDS

A large part of this thesis was motivated by the intuition that similarities between tasks and languages is one of the most important factors when considering a multitask or a multilingual approach.
Even though some correlations were found in experiments, attempting to correlate measures of task and language similarities with change in model performance, much of the change that is observed is left unaccounted for.
This highlights the case that even if such similarities are important, the situation is more complex than what can be explained purely by measures of correlation.

The successful experiments dealing with the combination of multitask and multilingual learning show the most potential for future research based on this thesis.
A plethora of new studies based on this idea can be imagined.
One could take advantage of morphological similarities by looking at character-level representations, investigating to what extent an architecture such as sluice networks can learn to share parameters for similar task–language combinations.
Another option is to probe into how much annotation is needed in order to bootstrap off of other languages than the target language at hand in order to predict reasonable labels for the target language.

A concrete proposal toward One Model to rule them all at a larger scale, involving more languages and tasks, is to model this in a sluice network <cit.>.
In this recently proposed architecture, the sharing of layers itself is learned by the network.
Combining a large amount of tasks in such a network should therefore allow for taking advantage of relevant similarities between tasks, while not sharing parameters in the case of dissimilarities which may lead to negative transfer.
Taking this one step further, by also involving multilingual learning as in this thesis, could also allow for learning between which languages to share parameters.
For instance, this architecture might be able to learn which parts of a character-RNN to share between which languages, for instance learning to only share these parameters between closely related languages, thus avoiding any negative transfer in this setting.
Further combining this approach with language vectors <cit.> might facilitate exploitation of language similarities.
This approach might therefore alleviate many of the problems with hard parameter sharing, by allowing the model to only utilise parameter sharing for similarities between languages, while learning separate parameters for language-specific features.
As the amount of both unannotated parallel data, and annotated data with various universal annotation schemes increases, it is only a matter of choosing the right approach in order to arrive at One Model to rule them all.

CHAPTER: CORRELATION FIGURES FOR ALL LANGUAGES IN CHAPTER <REF>

Correlations between $\Delta_{acc}$ and entropy. Each data point represents a single experiment run.

Correlations between $\Delta_{acc}$ and mutual information. Each data point represents a single experiment run.

CHAPTER: BIBLIOGRAPHICAL ABBREVIATIONS

* AAAI $\rightarrow$ Conference on Artificial Intelligence

* ACL $\rightarrow$ Annual Meeting of the Association for Computational Linguistics

* COLING $\rightarrow$ International Conference on Computational Linguistics

* CoNLL $\rightarrow$ Conference on Computational Natural Language Learning

* EACL $\rightarrow$ Conference of the European Chapter of the Association for Computational Linguistics

* EMNLP $\rightarrow$ Conference on Empirical Methods in Natural Language Processing

* HLT $\rightarrow$ Conference on Human Language Technology

* ICLR $\rightarrow$ International Conference on Learning Representations

* ICML $\rightarrow$ International Conference on Machine learning

* IJCNLP $\rightarrow$ International Joint Conference on Natural Language Processing

* LREC $\rightarrow$ Language Resources and Evaluation Conference

* NAACL $\rightarrow$ Conference of the North American Chapter of the Association for Computational Linguistics

* NIPS $\rightarrow$ Neural Information Processing Systems Conference

* NoDaLiDa $\rightarrow$ Nordic Conference on Computational Linguistics

* *SEM $\rightarrow$ Joint Conference on Lexical and Computational Semantics

* SemEval $\rightarrow$ International Workshop on Semantic Evaluation

CHAPTER: SUMMARY

When learning a new skill, you take advantage of your preexisting skills and knowledge.
For instance, if you are a skilled violinist, you will likely have an easier time learning to play cello.
Similarly, when learning a new language you take advantage of the languages you already speak.
For instance, if your native language is Norwegian and you decide to learn Dutch, the lexical overlap between these two languages will likely benefit your rate of language acquisition.
This thesis deals with the intersection of learning multiple tasks and learning multiple languages in the context of Natural Language Processing (NLP), which can be defined as the study of computational processing of human language.
Although these two types of learning may seem different on the surface, we will see that they share many similarities.

The traditional approach in NLP is to consider a single task for a single language at a time.
However, recent advances allow for broadening this approach, by considering data for multiple tasks and languages simultaneously.
This is an important approach to explore further as the key to improving the reliability of NLP, especially for low-resource languages, is to take advantage of all relevant data whenever possible.
In Part I of this thesis, we begin with an introduction to neural networks with a focus on NLP (Chapter 2), since such architectures are particularly well suited to combined learning of multiple tasks and languages.
We will then look at some ways in which neural networks can consider multiple tasks and languages at the same time (Chapter 3).
Specifically, we will consider multitask learning (MTL), and several common multilingual approaches.

In Part II of this thesis, I look at exploiting the fact that many NLP tasks are highly related to one another.
This is done by experimenting with MTL using hard parameter sharing, which has proven beneficial for a variety of NLP tasks.
In spite of such successes, however, it is not clear when or why MTL is beneficial in NLP.
Chapter 4 contains a case study in which semantic tagging is shown to be beneficial for POS tagging.
This further highlights the question of when MTL is beneficial in NLP tagging tasks, which is explored using information-theoretic measures in Chapter 5.

Multilingual models can leverage the fact that many languages share commonalities with one another.
These resemblances can occur on various levels, with languages sharing, for instance, syntactic, morphological, or lexical features.
While there are many possibilities for exploiting these commonalities, the focus in this thesis is on using multilingual word representations, as they allow for straight-forward integration in a neural network.
As with MTL, it is not clear in which cases it is an advantage to go multilingual.
In Part III, I begin with presenting a case study on multilingual semantic textual similarity (Chapter 6).
Following this, I explore how similar languages need to be, and in which way, in order to for it to be useful to go multilingual (Chapter 7).

In Part IV of this thesis, I experiment with a combined paradigm, in which a neural network is trained on several languages and tasks simultaneously (Chapter 8).
Finally, the thesis is concluded in Part V (Chapter 9).
The experiments in this thesis are run on a large collection of mainly lexically oriented tasks, both semantic and morphosyntactic in nature, and on a total of 60 languages, representing a relatively wide typological range.

While traditional NLP approaches consider a single task or language at a time, the aim of this thesis was to answer several research questions dealing with pushing past this boundary.
In doing so, the hope is that in the long term, low-resource languages can benefit from the advances made in NLP which are currently to a large extent reserved for high-resource languages.
This, in turn, may then have positive consequences for, e.g., language preservation, as speakers of minority languages will have a lower degree of pressure to using high-resource languages.
In the short term, answering the specific research questions posed should be of use to NLP researchers working towards the same goal.

CHAPTER: SAMENVATTING

Wanneer je een nieuwe vaardigheid leert maak je gebruik van de vaardigheden en kennis die je al bezit.
Als je bijvoorbeeld een ervaren violist bent, is het waarschijnlijk makkelijker om ook cello te leren spelen.
Ook bij het leren van een nieuwe taal maak je gebruik van de talen die je al beheerst.
Bijvoorbeeld, als je moedertaal Noors is en je besluit dat je Nederlands wil gaan leren, dan maakt het grote aantal woorden dat op elkaar lijkt in die twee talen het waarschijnlijk makkelijker om de nieuwe taal te leren.
In dit proefschrift kijk ik naar het leren van meerdere taken, het leren van meerdere talen en de combinatie daarvan, in het kader van natural language processing (NLP): de computationele analyse van menselijke taal.
Hoewel deze twee soorten van leren aan de oppervlakte misschien anders lijken, zullen we zien dat ze meerdere overeenkomsten hebben.

De traditionele aanpak in NLP is om op één taak voor één taal te focussen.
Nieuwe ontwikkelingen maken het echter mogelijk om deze aanpak uit te breiden, door tegelijkertijd meerdere taken én meerdere talen te bekijken.
Dit is een veelbelovende richting voor verder onderzoek, aangezien het gebruiken van zoveel mogelijk relevante data de sleutel is tot het verbeteren van de betrouwbaarheid van NLP, met name voor kleinere talen.
Deel I van dit proefschrift begint met een inleiding over neurale netwerken, met een focus op NLP (Hoofdstuk <ref>). Deze architecturen zijn bijzonder geschikt voor het gecombineerd leren van meerdere taken en talen.
Daarna zien we hoe gecombineerd leren werkt, door een aantal manieren waarop neurale netwerken tegelijkertijd meerdere taken en talen kunnen leren te beschrijven (Hoofdstuk <ref>).
In het bijzonder kijken we naar multitask learning (MTL) en enkele meertalige benaderingen van NLP.

In Deel II van dit proefschrift wordt onderzocht hoe we het feit dat veel taken in NLP sterk met elkaar zijn verbonden kunnen benutten.
Dit wordt gedaan door te experimenteren met MTL met hard parameter sharing, dat voor veel taken in NLP succesvol is gebleken.
Ondanks deze successen is het echter niet helemaal duidelijk wanneer of waarom het gebruik van MTL voordelig is voor NLP.
Hoofdstuk 4 bevat een casestudy van MTL waarin ik laat zien dat het toekennen van semantische labels aan woorden een verbetering oplevert bij het herkennen van woordsoorten.
Deze vinding benadrukt het belang van het beantwoorden van de vraag `Wanneer is het gebruik van MTL voordelig voor NLP?'. In Hoofdstuk 5 wordt dit onderzocht aan de hand van begrippen uit de informatietheorie.

Meertalige modellen kunnen gebruik maken van het feit dat talen vaak overeenkomsten met elkaar hebben.
Deze overeenkomsten kunnen op verschillende niveaus optreden, bijvoorbeeld op syntactisch, morfologisch of lexicaal vlak.
Hoewel er veel mogelijkheden zijn om gebruik te maken van deze overeenkomsten, kijken we in dit proefschrift naar het gebruik van meertalige woordrepresentaties, omdat deze erg goed te combineren zijn met neurale netwerken.
Net als bij MTL is het hier niet duidelijk in welke gevallen het een voordeel is om meertalige modellen te bouwen. Deel III begint met een casestudy over meertalige semantische tekstuele gelijkenis (Hoofdstuk 6).
Daarna onderzoek ik in welke mate talen op elkaar moeten lijken, en op welke manier, om voordeel te kunnen halen uit het bouwen van een meertalig model (Hoofdstuk 7).

In Deel IV van dit proefschrift experimenteer ik met een gecombineerd paradigma waarin een neuraal netwerk verschillende talen én taken tegelijkertijd leert (Hoofdstuk 8).
Tenslotte worden de conclusies uit dit proefschrift gepresenteerd in Deel V (Hoofdstuk 9).

De experimenten in dit proefschrift worden uitgevoerd op een grote verzameling voornamelijk lexicaal georiënteerde taken, zowel semantisch en morfosyntactisch van aard, en op in totaal 60 talen, die een relatief grote typologische diversiteit vertegenwoordigen.

Terwijl de traditionele NLP-benaderingen slechts een enkele taak of taal tegelijk behandelen, is het doel van dit proefschrift juist om onderzoeksvragen die deze grenzen overschrijden te beantwoorden.
Dit heeft als doel dat kleinere talen op de lange termijn kunnen profiteren van de vooruitgang die in NLP wordt geboekt, aangezien deze momenteel grotendeels ten goede komt aan grotere talen.
Dit kan positieve gevolgen hebben voor bijvoorbeeld taalbehoud, omdat, onder andere, sprekers van minderheidstalen een lagere druk zullen voelen om grotere talen te gebruiken.
Op de korte termijn zullen de antwoorden op de specifieke onderzoeksvragen van dit proefschrift nuttig zijn voor NLP-onderzoekers die naar hetzelfde doel streven.