#  Malthusian assumptions, Boserupian response in models of the transitions to
agriculture 111To appear in: “Society, Nature and History: The Legacy of Ester
Boserup”, Springer, Vienna

Carsten Lemmen

(Institut für Küstenforschung, Helmholtz-Zentrum Geesthacht, Max-Planck Straße
1, 21501 Geesthacht, Germany<EMAIL_ADDRESS>)

### Abstract.

In the many transitions from foraging to agropastoralism it is debated whether
the primary drivers are innovations in technology or increases of population.
The driver discussion traditionally separates Malthusian (technology driven)
from Boserupian (population driven) theories. I present a numerical model of
the transitions to agriculture and discuss this model in the light of the
population versus technology debate and in Boserup’s analytical framework in
development theory. Although my model is based on
ecological—Neomalthusian—principles, the coevolutionary positive feedback
relationship between technology and population results in a seemingly
Boserupian response: innovation is greatest when population pressure is
highest. This outcome is not only visible in the theory-driven reduced model,
but is also present in a corresponding “real world” simulator which was tested
against archaeological data, demonstrating the relevance and validity of the
coevolutionary model. The lesson to be learned is that not all that acts
Boserupian needs Boserup at its core.

##  1 Transitions to agriculture

The relationship between humans and their environment underwent a radical
change during the last 10,000 years: from mobile and small groups of foraging
people to sedentary extensive cultivators and on to high-density intensive
agriculture modern society; these transitions fundamentally turned the
formerly predominantly passive human user of the environment into an active
component of the Earth system. The most striking global impact is only visible
and measurable during the last 150 years (Crutzen and Stoermer, 2000; Crutzen,
2002); much earlier, however, the use of forest resources for metal smelting
from early Roman times and the medieval extensive agricultural system had
already changed the landscape (Barker, 2011; Kaplan et al., 2009); global
climate effects of these early extensive cultivation and harvesting practices
are yet under debate (Ruddiman, 2003; Lemmen, 2010; Kaplan et al., 2011;
Stocker et al., 2011).

Transitions to agriculture occurred in almost every region of the world,
earliest in China and the Near East over 9000 years ago (Kuijt and Goring-
morris, 2002; Londo et al., 2006), and latest in Australia and Oceania with
the arrival of Polynesian and European immigrants few hundred years ago
(Diamond and Bellwood, 2003). While each local transition can be considered
revolutionary, the many diverse mechanisms, environments, and cultural
contexts of each agricultural transition make it difficult to speak of the one
’Neolithic revolution’, as the transition to farming and herding was termed by
V. G. Childe almost a century ago (Childe, 1925). The transitions from
foraging to farming were not only one big step, but may have consisted of
intermediary stages: Bogaard (2005) looks at the transition in terms of the
land use system: she sees first inadvertent cultivation then horticulture then
simple and then advanced agriculture, while Boserup (1965) discriminates these
stages by the management practice ranging from forest, bush and short fallow
to annual and multi cropping.

In contemporary hunting-gathering societies much less time has to be devoted
to procuring food from hunting and gathering opposed to agriculture and
herding (e.g., Sahlins, 1972); less labor is required for long fallow systems
compared to intensive multi-cropping agriculture(Boserup, 1965). So why farm?
Different responses from archaeology (Barker, 2011), demography (Turchin and
Nefedov, 2009), historical economy (Weisdorf, 2005), and ecosystem modeling
(Wirtz and Lemmen, 2003) call upon processes such as social reorganization,
the value of leisure, changing resources, or coevolutionary thresholds.

The probably simplest relationship was proposed by Malthus (1798, p.11),
namely that more production sustains larger population. With larger
population, more production is possible, thereby constituting a positive
feedback loop, which ideally results in ever greater (geometric) growth and
productivity. That this is not the case in a world with finite resources was
expressed by Malthus (1798, p. 4) by stating that “Population, when unchecked,
increases at a geometrical ratio. Subsistence increases only in an
arithmetical ratio. A slight acquaintance with numbers will show the immensity
of the first power in comparison with the second”. Malthus identified the need
for positive and preventive checks to balance population increase with the
limited capacity of resources.

How does an increase in productivity come about? First and foremost, the input
of more labor increases productivity (Malthus, 1798, p. 11), subject to the
constraints of finite resources and diminishing returns. Where Malthus,
however, focused on extensive productivity increase, the intensification
component of productivity increase was highlighted by Boserup (1965).
Investments in a more intensive production system would, however, require
large additional labor, and the benefits of such investments were often small.
To stimulate an investment in more intensive agriculture, Boserup requires
population pressure.

Both Malthus (1798, 1826) and Boserup (1965, 1981) concentrate on the role of
labor (and later division of labor and social/family organization) and
innovations which increase area productivity (like storage or tools, requiring
relatively more labor for harvesting, building, and tool processing). Both
authors neglect the role of labor-independent innovation, or innovations which
increase both area and labor productivity; these are innovations in the
resources themselves, such as cultivation of higher-yielding grains or
imported high yield varieties, or their management such as water rights; this
distinction may not be unambiguous for all innovations, it is used here
conceptually. Labor-independent innovation can be stimulated by diversity and
density of a population, both of which are positively related to population
size. Already Darwin (1859, p. 156) wrote “The more diversified [..], by so
much will they be better enabled to seize on many and widely diversified
places in the polity of nature”. Translated into the realm of innovativity,
Darwin’s “seizing of places”, or niche occupation, would be the realization of
technical and scientific opportunities. As for density as a stimulus of
innovation, it is aggregation which constitutes a motor of technological and
cultural change (Smith, 1776; Boyd and Richerson, 1995)222This does not,
however, give the reason for a particular choice of one innovation over
another (Sober, 1992)..

##  2 Models of population, production, and innovation

In 1996, E. Boserup reflected on the problems arising from the differences in
terminology and methodology when comparing different models of development
theories (Boserup, 1996). She suggested a common framework to facilitate
interdisciplinary cooperation based on six structures: Environment ($E$),
Population ($P$), technology, occupational structure, family structure and
culture. In this framework, she then interpreted the major works of Adam
Smith, Thomas Malthus, Max Weber, Karl Marx, David Ricardo, and Neomalthusian
thinking, as well as her own view on different stages of the developmental
process.

For many of the theories and models discussed by Boserup in this framework,
the partitioning in six structures can be simplified by (a) aggregating
technology and occupational structure into a single entity technology ($T$),
and by (b) aggregating culture and family structure into a single entity
culture ($C$). Aggregating technology and occupational structure means that I
assume here that changes in technology are equivalent to changes in
organization and that the location of technological change is the occupational
sector. By aggregating family structure and culture I assume that values and
social conventions penetrate from the society into the family and are governed
by similar dynamics. The reduced framework then consists of the four
compartments population, environment, technology, and culture (PETC, Figure
1).

In this PETC framework, the one referring to Malthus (1798) involves only
population and environment. Population growth exerts pressure on the
environment, and failure to provide adequate resources from the environment
acts as a positive check on population through higher mortality (Figure 1a).
Technology does not play a role in this simplest Malthusian model333Malthus
considered the increase of carrying capacity by autonomously occurring
inventions (Lee, 1986), however, this was not discussed by Boserup (1996) in
her model intercomparison.. Culture in the form of preventive checks—such as
birth control—acts on population only in later versions of his theory
(Malthus, 1826). At its core remains “the dependent role he assigns to
population growth” (Marquette, 1997). D. Ricardo (1821) proposed that the
incentive to intensify and develop technologies comes from a stimulus in
population pressure. The demand for more land ($E$), however, leads to
declining marginal benefits of and a negative feedback on innovation ($T$) due
to high costs of renting the land (Figure 1b). In Ricardo’s work, population
is independent, and technology and environment are the dependent variables.

Population is also the driving factor in Boserup’s (1965; 1981) works. Of the
six transitions considered by Boserup (1996), five can be accommodated within
my PETC framework as a succession of population, environment, technology, and
culture: foraging to crop production, village development, Eastern hemisphere
pastoralism, urbanization, and industrialization (Figure 1c)444The sixth
transition—Western European fertility decline—follows a different path as a
succession of technology, environment, culture, and last population; it is not
considered here.. In all these transitions, population growth leads to
pressure felt from the limited environmental resources, which in turn
stimulates technological and organizational change, and later results in
cultural changes evident in cults, social hierarchies, women’s status, and
status symbols. Within this group of five transitions, her model of village
development, in addition, has a direct population–technology link, and allows
for a feedback of the land resources on occupational structure (dotted lines
in Figure 1c). Furthermore, her model of the foraging to farming transition
includes a feedback from culture to organizational structure (not shown).

Figure 1: Four compartment framework for the interrelationship between
population, environment, technology, and culture. Four economic theories are
contrasted: the essays on the principles of population by T. Malthus (1798,
1826) (panel a, dotted line indicates the revised essay including culture
change); D. Ricardo (1821)’s principles of economy (panel b); E. Boserup
(1965, 1981)’s theories for five transitions explained in Boserup (1996)
(panel c, 1981 refinements shown as dotted lines); and the ecological model
proposed in this chapter. The framework is a simplification of the six
compartment framework originally proposed by Boserup (1996).

##  3 A combined model and ‘real’ world application

I suggest here a different model of population development taking the foraging
to farming transition as an example (Figure 1d). This model is a reduced form
of the Global Land Use and technological Evolution Simulator (GLUES, described
below), which has been operationally applied to a number of problems in
archaeology and climate research (Kaplan et al., 2011; Lemmen, 2010; Lemmen
and Wirtz, 2012; Lemmen et al., 2011). The reduced model shares the functional
characteristics of the full model, but it is not spatially explicit and the
biogeographic and climate background is regarded as constant (see Appendix for
equations).

In terms of the PETC framework, the dynamics between population, environment,
technology and culture is the following (Figure 1d, cmp. Boserup 1996, p.
509).

  1. 1.

P$\rightarrow$ T$\rightarrow$ P Population growth stimulates innovation by
aggregation and diversity. Innovations in, e.g., health care increase
population;

  2. 2.

P$\rightarrow$ E $\rightarrow$ P Population growth uses ever more land for
hunting and exerts pressure on the game stock, higher population densities
damage the environment, and food shortage leads to reduced fertility
(preventive check) or higher mortality (positive check). The rising capacity
of the environment supports higher population;

  3. 3.

T$\rightarrow$ E More intensive foraging or farming strategies damage the
environment, while efficiency gains lead to higher capacity of the
environment;

  4. 4.

T$\rightarrow$ C Adoption of novel technologies induces changes in social
structure where specialists and leaders or cults emerge;

  5. 5.

C$\rightarrow$ P Family and social structure change reproduction rates.

Richerson and Boyd (1998) claim that basically all models which are rooted in
ecology are Neomalthusian in essence, i.e., they can be characterized by a
P$\rightarrow$ T $\rightarrow$ E loop in Boserup’s (1996) framework. This loop
can be detected in my model, as well; in fact, historically it developed from
ecosystem models of tree stands or algal communities(Wirtz and Eckhardt,
1996). Unlike many other models, however, GLUES is based on coevolutionary
dynamics of technologies and population, and as such has no a priori
information on whether there is a (Malthusian) “invention-pull view of
population history” (Lee, 1986, p. 98), or whether population is the
(Boserupian) driver of development555See also Simon (1993) for a detailed
discussion.. Applications of GLUES show that there is an emergent emancipation
of population development from the environment with increasing population and
innovation (Lemmen and Wirtz, 2010, 2012; Lemmen et al., 2011).

GLUES mathematically resolves the dynamics of population density and three
population-averaged characteristic sociocultural traits: technology $T_{A}$,
share of agropastoral activities $C$, and economic diversity $T_{B}$. These
are defined for preindustrial societies as follows:

  1. 1.

Technology $T_{A}$ is a trait which describes the efficiency of food
procurement—related to both foraging and farming—and improvements in health
care. In particular, technology as a model describes the availability of
tools, weapons, and transport or storage facilities. It aggregates over
various relevant characteristics of early societies and also represents social
aspects related to work organization and knowledge management. It quantifies
improved efficiency of subsistence, which is often connected to social and
technological modifications that run in parallel. An example is the technical
and societal skill of writing as a means for cultural storage and
administration, with the latter acting as a organizational lubricant for food
procurement and its optimal allocation in space and among social groups.
$T_{A}$ is labour dependent.

  2. 2.

A second model variable $C$ represents the share of farming and herding
activities, encompassing both animal husbandry and plant cultivation. It
describes the allocation of energy, time, or manpower to agropastoralism with
respect to the total food sector.

  3. 3.

Economic diversity $T_{B}$ resolves the number of different agropastoral
economies available to a regional population. This trait is in the full model
closely tied to regional vegetation resources and climate constraints; in this
reduced model, it denotes a labour-independent technology. A larger economic
diversity offering different niches for agricultural or pastoral practices
enhances the reliability of subsistence and the efficacy in exploiting
heterogeneous landscapes.

The temporal change of each of these characteristic traits follows the
direction of increased benefit for success (i.e. growth) of its associated
population (Appendix equation 2); this concept had been derived for genetic
traits in the works of Fisher (1930), and was recently more stringently
formulated by Metz and colleagues (Metz et al., 1992; Kisdi and Geritz, 2010)
as adaptive dynamics (AD). In AD, the population averaged value of a trait
changes at a rate which is proportional to the gradient of the fitness
function evaluated at the mean trait value. The AD approach was extended to
functional traits of ecological communities (Wirtz and Eckhardt, 1996; Merico
et al., 2009), and was first applied to cultural traits of human communities
by Wirtz and Lemmen (2003).

The adaptive coevolution of the food production system $\\{T_{A},T_{B},C\\}$
and population $P$ (Appendix equations 1–4), which is at the heart of this
model’s implementation, had also been found empirically by Boserup (1981, p.
15): “The close relationship which exists today between population density and
food production system is the result of two long-existing processes of
adaptation. On the one hand, population density has adapted to the natural
conditions for food production []; on the other hand, food supply systems have
adapted to changes in population density.”

##  4 Innovation in the transition to agriculture

Figure 2: Trajectories of population $P$, environment $E$, and technologies
$T_{A},T_{B}$ (panel a) and phase diagram of innovation rate versus population
pressure (panel b) from a simulation with a simplified version of the Global
Land Use and technological Evolution Simulator. The trajectories describe the
temporal evolution of population density, capacity denoted as environment, a
labour dependent technology $T_{A}$, and a labour-independent technology
$T_{B}$. Numbers identify the different stages of development in the both
diagrams. In the phase diagram b), the innovation rate, derived as the
cumulative change in $T_{A}+T_{B}$, is shown in relation to population
pressure, calculated as $1-E+P$.

The outcome of the coevolutionary model simulation with the reduced GLUES is
shown in Figure 2. I divided both the trajectories (temporal evolution of
state variables, panel a) and the the phase space (panel b) into six stages,
which I discuss below.

  1. 1.

Growth phase: Starting from a Malthusian perspective, and looking only at
population and environment (quantified here as the ecosystem capacity, i.e.
the ratio of birth over mortality terms in the growth rate equation 3),
population grows towards its capacity with diminishing returns as $P$
approaches $E$; this first phase spans only a short period of time but covers
a large area in phase space;

  2. 2.

Persistent innovation in technology $T_{A}$ and associated investments in tool
making and administration allow sustained slow growth of population $P$ and
alleviates the built-up population pressure; in contrast to the growth phase,
the phase space coverage is very small while the temporal extent of this phase
is large;

  3. 3.

Transition phase: rapid innovation in a labour-independent technology $T_{B}$
(e.g. domestication successes) leads to

  4. 4.

Pressure relief, but induces also a change in culture (not shown);

  5. 5.

Equilibration: Innovation slows but has led to a wider gap between $P$ and $E$
because of the investments made in manufacturing and organization during the
transition: accordingly, population pressure increases more slowly and up to a
lower value than in the growth phase (1.).

  6. 6.

Persistent innovation: corresponds to phase (2.) and is again characterized by
persistent innovation in technology $T_{A}$ and a slow population pressure
relief.

What can be learned about the relationship between population pressure and
innovativity from Figure 2? (i) Innovation is greatest at high population
pressure. (ii) In this model there is always innovation, at no time is
technology change negative. (iii) The relationship between innovativity and
population pressure changes profoundly during the foraging-farming transition;
three different regimes can be identified: (i) a positive relationship where
acceleration of innovation corresponds to population pressure increases
(phases 1., 2., 6.), (ii) a negative relationship with pressure relief during
accelerating innovation (phase 3., 4.), and (iii) a negative relationship with
deceleration of innovativity at increasing pressure (phase 5.).

A superficial analysis would find that population pressure is the motor of
innovation in this example: population increase seemingly precedes the
stepwise technological change (Figure 2a). Only a detailed look at the phase
space (Figure 2b)—especially at the transition phases 2. and 3.—shows that
innovativity decelerates at very high population pressure and that the largest
innovation occurs slightly below the highest population pressure. In fact, the
driver in the transition depicted here is not population, but
technology666There would be no evolution of $T$ without $P$ due to the
coevolutionary definition of the system. The dynamics of $T$, however, leads
the dynamics of $P$ at the foraging farming transition.. Only the different
coevolutionary time scales of population growth (fast) and innovation (slow)
yield the seemingly Boserupian, i.e., population driven, response.

The same mathematical model—plus spatial and biogeographic aspects—has been
used to successfully simulate the many transitions to agriculture in Neolithic
Europe (Lemmen et al., 2011), with good agreement with the radiocarbon record.
Also there, the transitions appear Boserupian with critical innovations
occurring at high population pressure. If the numerical analysis had not been
available (and proved that this is in fact technology driven), such as it is
in the discretely sampled data from observations of technological change, one
would have to have come to the erroneous conclusion that this type of
innovation was population driven.

##  5 Conclusion

I presented a reduced version of the Global Land Use and technological
Evolution Simulator, a numerical model which is capable of realistically
simulating regional foraging-farming transitions worldwide. The simulated—and
possibly also observed—transitions are seemingly Boserupian, i.e., population
driven: innovation is greatest when population pressure is high. Analytical
examination of the model, however, shows that technological change is the
driver, and that in the context of a simplified version of Boserup’s (1996)
framework in development theory the model should be classified as
Neomalthusian. I thus demonstrated that Boserupian appearance may be based on
Malthusian assumptions; I caution not to infer too quickly a Boserupian
mechanism for an observed real world system when its dynamics appears to be
population pressure driven.

## Appendix: the reduced GLUES model

A coevolutionary system of population $P$ and characteristic traits
$X\in\\{T_{A},T_{B},C\\}$ is defined by the evolution equations

$\displaystyle\frac{\mathrm{d}P}{\mathrm{d}t}$ $\displaystyle=$ $\displaystyle
P\cdot r$ (1) $\displaystyle\frac{\mathrm{d}X}{\mathrm{d}t}$ $\displaystyle=$
$\displaystyle\delta_{X}\cdot\frac{\partial{}r}{\partial{}X},$ (2)

where $r$ denotes the specific growth rate of population $P$, and the
$\delta_{X}$ are variability measures for each $X$. Growth rate $r$ is defined
as

$r=\mu\cdot(1-\omega
T_{A})\cdot(1-\gamma\sqrt{T_{A}}P)\cdot\mathrm{SI}-\rho\cdot T^{-1}_{A}\cdot
P,$ (3)

with coefficients $\mu,\rho,\omega,\gamma$. In this formulation, the positive
term including food production SI is modulated by labour loss for
administration $(-\omega T_{A})$ and by overexploitation of the environment
$(-\gamma\sqrt{T_{A}}P)$. Food production depends on the cultural system $C$
and available technologies as follows:

$SI=(1-C)\cdot\sqrt{(}T_{A})+C\cdot T_{A}\cdot T_{B},$ (4)

where the left summand denotes foraging activities and the right summand
agropastoral practice.

To produce the results for Figure 2, I assumed the following parameter values:
$\mu=\rho=0.004$, $\omega=0.04$, $\gamma=0.12$, $\delta_{T_{A}}=0.025$,
$\delta_{T_{B}}=0.9$; a variable $\delta_{C}=C\cdot(1-C)$; and initial values
for $P_{0}=0.01$, $T_{A,0}=1.0$, $T_{B,0}=0.8$, and $C_{0}=0.04$.

### Acknowledgments.

This study was partly funded by the German National Science Foundation (DFG
priority project 1266 Interdynamik) and by the PACES program of the Helmholtz
Gemeinschaft. The paper received great stimulus from discussions during the
Ester Boserup Conference 2010—A Centennial Tribute: Long-term trajectories in
population, gender relations, land use, and the environment, November 15–17,
2010 in Vienna, Austria. I received helpful comments from two anonymous
reviewers. GLUES is free and open source software and can be obtained from
http://glues.sourceforge.net/.

## References

  * Barker (2011) Barker, G. (2011), Archaeology: The cost of cultivation, Nature, 473, 163–164.
  * Bogaard (2005) Bogaard, A. (2005), Garden agriculture and the nature of early farming in Europe and the Near East, World Archeology, 37(2), 177–196.
  * Boserup (1965) Boserup, E. (1965), The conditions of agricultural growth, 124 pp., Aldine, Chicago.
  * Boserup (1981) Boserup, E. (1981), Population and technological change: A study of long-term trends, University of Chicago Press Chicago.
  * Boserup (1996) Boserup, E. (1996), Development theory: An analytical framework and selected application, Population and Development Review, pp. 505–515.
  * Boyd and Richerson (1995) Boyd, R., and P. J. Richerson (1995), Why does culture increase human adaptibility?, Ethology and Sociobiology, 16, 125–143.
  * Childe (1925) Childe, V. G. (1925), The Dawn of European civilization, Kegan Paul, Trench, Trubner, London.
  * Crutzen (2002) Crutzen, P. J. (2002), Geology of mankind, Nature, 415, 23.
  * Crutzen and Stoermer (2000) Crutzen, P. J., and E. F. Stoermer (2000), The anthropocene, IGBP Newsletter, 41.
  * Darwin (1859) Darwin, C. (1859), The origin of species, 811 pp., Hayes Barton Press, reprinted 1958.
  * Diamond and Bellwood (2003) Diamond, J., and P. Bellwood (2003), Farmers and their languages: The first expansions, Science, 300(5619), 597–603.
  * Fisher (1930) Fisher, R. A. (1930), The Genetical Theory of Natural Selection, Dover, New York.
  * Kaplan et al. (2009) Kaplan, J. O., K. M. Krumhardt, and N. Zimmermann (2009), The prehistoric and preindustrial deforestation of Europe, Quaternary Science Reviews, 28(27-28), 3016–3034, 10.1016/j.quascirev.2009.09.028.
  * Kaplan et al. (2011) Kaplan, J. O., K. M. Krumhardt, E. C. Ellis, W. F. Ruddiman, C. Lemmen, and K. Klein Goldewijk (2011), Holocene carbon emissions as a result of anthropogenic land cover change, The Holocene, 21(5), 775–791, 10.1177/0959683610386983.
  * Kisdi and Geritz (2010) Kisdi, E., and S. A. H. Geritz (2010), Adaptive dynamics: a framework to model evolution in the ecological theatre, Journal of Mathematical Biology, 61(1), 165–9, 10.1007/s00285-009-0300-9.
  * Kuijt and Goring-morris (2002) Kuijt, I., and N. Goring-morris (2002), Foraging , Farming , and Social Complexity in the Pre-Pottery Neolithic of the Southern Levant : A Review and Synthesis, World Archaeology, 16(4).
  * Lee (1986) Lee, R. (1986), Malthus and baserup: A dynamic synthesis, in The state of population theory: Forward from Malthus, edited by D. Coleman and R. Schofield, pp. 96–130, B. Blackwell.
  * Lemmen (2010) Lemmen, C. (2010), World distribution of land cover changes during Pre- and Protohistoric Times and estimation of induced carbon releases, Géomorphologie : relief, processus, environnement, (4), 303–312.
  * Lemmen and Wirtz (2010) Lemmen, C., and K. W. Wirtz (2010), Socio-technological revolutions and migration waves: re-examining early world history with a numerical model, in The Spread of the Neolithic to Central Europe, RGZM Tagungen, vol. 4,1, edited by D. Gronenborn and J. Petrasch, pp. 25–38, Verlag des Römisch-Germanischen Zentralmuseums, Mainz.
  * Lemmen and Wirtz (2012) Lemmen, C., and K. W. Wirtz (2012), On the sensitivity of the simulated European Neolithic transition to climate extremes, Journal of Archaeological Science, submitted.
  * Lemmen et al. (2011) Lemmen, C., D. Gronenborn, and K. W. Wirtz (2011), A simulation of the Neolithic transition in Western Eurasia, Journal of Archaeological Science, 38(12), 3459–3470, 10.1016/j.jas.2011.08.008.
  * Londo et al. (2006) Londo, J. P., Y.-c. Chiang, K.-h. Hung, T.-y. Chiang, and B. A. Schaal (2006), Phylogeography of Asian wild rice, Oryza rufipogon, reveals multiple independent domestications of cultivated rice, Oryza sativa, Proceedings of the National Academy of Sciences of the United States of America.
  * Malthus (1798) Malthus, T. (1798), An essay on the principle of population, in A Commentary on Malthus’ 1798 Essay on the Principle of Population as Social Theory, edited by F. Elwell, pp. 127–294, Mellen Press, Lewiston, commentary printed 2001.
  * Malthus (1826) Malthus, T. (1826), An Essay on the Principle of Population, Cambridge University Press, edited by Patricia James [1989].
  * Marquette (1997) Marquette, C. (1997), Turning but not toppling Malthus: Boserupian theory on population and the environment relationships, Working papers in development studies and human rights, 16.
  * Merico et al. (2009) Merico, A., J. Bruggeman, and K. W. Wirtz (2009), A trait-based approach for downscaling complexity in plankton ecosystem models, Ecological Modelling, 220(21), 3001–3010, 10.1016/j.ecolmodel.2009.05.005.
  * Metz et al. (1992) Metz, J., R. Nisbet, and S. Geritz (1992), How should we define fitness for general ecological scenarios, Trends in Ecology and Evolution, 7, 198–202.
  * Ricardo (1821) Ricardo, D. (1821), Principles of political economy, third ed., John Murray, London.
  * Richerson and Boyd (1998) Richerson, P., and R. Boyd (1998), Homage to Malthus, Ricardo, and Boserup: Toward a general theory of population, economic growth, environmental deterioration, wealth, and poverty, Human Ecology Review, 4, 85–90.
  * Ruddiman (2003) Ruddiman, W. (2003), The anthropogenic greenhouse era began thousands of years ago, Climatic Change, pp. 261–293.
  * Sahlins (1972) Sahlins, M. (1972), Stone Age Economics, 362 pp., Aldine de Gruyter.
  * Simon (1993) Simon, J. (1993), Economic thought about population consequences: Some reflections, Journal of population economics, 6(2), 137–152.
  * Smith (1776) Smith, A. (1776), The Wealth of Nations [reprinted 1937], 740 pp., Modern Library, New York.
  * Sober (1992) Sober, E. R. (1992), Darwin’s nature, in The concept of Nature, edited by J. Torrance, pp. 84–96.
  * Stocker et al. (2011) Stocker, B., K. Strassmann, and F. Joos (2011), Sensitivity of holocene atmospheric CO2 and the modern carbon budget to early human land use: analyses with a process-based model, Biogeosciences, 8, 69–88.
  * Turchin and Nefedov (2009) Turchin, P., and S. Nefedov (2009), Secular cycles, Princeton University Press.
  * Weisdorf (2005) Weisdorf, J. (2005), From foraging to farming: explaining the Neolithic revolution, Journal of Economic Surveys, 19(4).
  * Wirtz and Eckhardt (1996) Wirtz, K. W., and B. Eckhardt (1996), Effective variables in ecosystem models with an application to phytoplankton succession, Ecological Modelling, 92, 33–53.
  * Wirtz and Lemmen (2003) Wirtz, K. W., and C. Lemmen (2003), A global dynamic model for the neolithic transition, Climatic Change, (1998), 333–367.


# Does Typological Blinding Impede Cross-Lingual Sharing?

Johannes Bjerva
Department of Computer Science
Aalborg University
<EMAIL_ADDRESS>
&Isabelle Augenstein
Department of Computer Science
University of Copenhagen
<EMAIL_ADDRESS>

###### Abstract

Bridging the performance gap between high- and low-resource languages has been
the focus of much previous work. Typological features from databases such as
the World Atlas of Language Structures (WALS) are a prime candidate for this,
as such data exists even for very low-resource languages. However, previous
work has only found minor benefits from using typological information. Our
hypothesis is that a model trained in a cross-lingual setting will pick up on
typological cues from the input data, thus overshadowing the utility of
explicitly using such features. We verify this hypothesis by blinding a model
to typological information, and investigate how cross-lingual sharing and
performance is impacted. Our model is based on a cross-lingual architecture in
which the latent weights governing the sharing between languages is learnt
during training. We show that (i) preventing this model from exploiting
typology severely reduces performance, while a control experiment reaffirms
that (ii) encouraging sharing according to typology somewhat improves
performance.

##  1 Introduction

Most languages in the world have little access to NLP technology due to data
scarcity (Joshi et al., 2020). Nonetheless, high-quality multilingual
representations can be obtained using only a raw text signal, e.g. via
multilingual language modelling (Devlin et al., 2019). Furthermore, structural
similarities of languages are to a large extent documented in typological
databases such as the World Atlas of Language Structures (WALS, Dryer and
Haspelmath (2013)). Hence, developing models which can take use typological
similarities of languages is an important direction in order to alleviate
language technology inequalities.

Figure 1: A PoS tagger is exposed (or blinded with gradient reversal,
$-\lambda$) to typological features. Observing $\alpha$ values tells us how
typology affects sharing.

While previous work has attempted to use typological information to inform NLP
models, our work differs significantly from such efforts in that we blind a
model to this information. Most previous work includes language information as
features, by using language IDs, or language embeddings (e.g. Ammar et al.
(2016); O’Horan et al. (2016); Östling and Tiedemann (2017); Ponti et al.
(2019); Oncevay et al. (2020)). Notably, limited effects are usually observed
from including typological features explicitly. For instance, de Lhoneux et
al. (2018) observe positive cross-lingual sharing effects only in a handful of
their settings. We therefore hypothesise that relevant typological information
is learned as a by-product of cross-lingual training. Hence, although models
do benefit from this information, it is not necessary to provide it explicitly
in a high-resource scenario, where there is abundant training data. This is
confirmed by Bjerva and Augenstein (2018a), who find that, e.g., language
embeddings trained on a morphological task can encode morphological features
from WALS.

In contrast with previous work, we blind a model to typological information,
by using adversarial techniques based on gradient reversal (Ganin and
Lempitsky, 2014). We evaluate on the structured prediction and classification
tasks in XTREME (Hu et al., 2020), yielding a total of 40 languages and 4
tasks. We show that when a model is blinded to typological signals relating to
syntax and morphology, performance on related NLP tasks drops significantly.
For instance, the mean accuracy across 40 languages for POS tagging drops by
1.8% when blinding the model to morphological features.

##  2 Model

An overview of the model is shown in Figure 1. We model each task in this
paper using the following steps. First, contextual representations are
extracted using multilingual BERT (m-BERT, Devlin et al. (2019)), a
transformer-based model (Vaswani et al., 2017), trained with shared word-
pieces across languages. We either blind m-BERT to typological features, with
an added adversarial component based on gradient reversal (Ganin and
Lempitsky, 2014), or expose it to them via multi-task learning (MTL, (Caruana,
1997)). Representations from m-BERT are fed to a latent multi-task
architecture learning network (Ruder et al., 2019), which includes $\alpha$
parameters we seek to investigate. The model learns which parameters to share
between languages (e.g. $\alpha_{es,fr}$ denotes sharing between Spanish and
French).

###  2.1 Sharing architecture

Our sharing architecture is based on that of Ruder et al. (2019), which has
latent variables learned during training, governing which layers and subspaces
are shared between tasks, to what extent, as well as the relative weighting of
different task losses. We are most interested in the parameters which control
the sharing between the hidden layers allocated to each task, referred to as
$\alpha$ parameters (Ruder et al., 2019). Consider a setting with two tasks
$A$ and $B$. The outputs $h_{A,k}$ and $h_{B,k}$ of the $k$-th layer for task
$A$ and $B$ interact through the $\alpha$ parameters, for which the output is
defined as:

$\begin{bmatrix}\widetilde{h}_{A,k}\\\
\widetilde{h}_{B,k}\end{bmatrix}=\begin{bmatrix}\alpha_{AA}&\alpha_{AB}\\\
\alpha_{BA}&\alpha_{BB}\end{bmatrix}\begin{bmatrix}{h_{A,k}}^{\top}\>,&{h_{B,k}}^{\top}\end{bmatrix}$
(1)

where $\widetilde{h}_{A,k}$ is a linear combination of the activations for
task $A$ at layer $k$, weighted with the learned $\alpha$s. While their model
is an MTL model, we choose to interpret this differently by considering each
language as a task, yielding $\alpha\in\mathbf{R}^{l\times l}$, where $l$ is
the number of languages for the given task. Each activation
$\widetilde{h}_{A,k}$ is then a linear combination of the language specific
activations $h_{A,k}$. These are used for prediction in the downstream tasks,
as in the baselines from Hu et al. (2020).

Crucially, this model allows us to draw conclusions about parameter sharing
between languages by observing the $\alpha$ parameters under the blinding and
prediction conditions. We will combine this insight with observing downstream
task performance in order to draw conclusions about the effects of typological
feature blinding and prediction.

###  2.2 Blinding/Exposing a Model to Typology

We introduce a component which can either blind or expose the model to
typological features. We implement this as a single task-specific layer per
feature, using the [CLS] token from m-BERT model, without access to any of the
soft sharing between languages from $\alpha$-layers. Each layer optimises a
categorical cross-entropy loss function ($L_{typ}$).

For this task, we predict typological features drawn from WALS (Dryer and
Haspelmath, 2013), inspired by previous work (Bjerva and Augenstein, 2018a).
Unlike previous work, we also blind the model to such features by including a
gradient reversal layer (Ganin and Lempitsky, 2014), which multiplies the
gradient of the typological prediction task with a negative constant
($-\lambda$), inspired by previous work on adversarial learning (Goodfellow et
al., 2014; Zhang et al., 2019; Chen et al., 2019). We hypothesise that using a
gradient reversal layer for typology will yield typology-invariant features,
and that this will perform worse on tasks for which the typological feature at
hand is important. For instance, we expect that blinding a model to syntactic
features will severely reduce performance for tasks which rely heavily on
syntax, such as POS tagging.

##  3 Cross-Lingual Experiments

We investigate the effects of typological blinding, using typological
parameters as presented in WALS (Dryer and Haspelmath, 2013). The experiments
are run on XTREME (Hu et al., 2020), which includes up to 40 languages from 12
language families and two isolates. We experiment on the following languages
(ISO 639-1 codes): af, ar, bg, bn, de, el, en, es, et, eu, fa, fi, fr, he, hi,
hu, id, it, ja, jv, ka, kk, ko, ml, mr, ms, my, nl, pt, ru, sw, ta, te, th,
tl, tr, ur, vi, yo, and zh. We experiment on four tasks: POS (part of speech
tagging), NER (named entity recognition), XNLI (cross-lingual natural language
inference), and PAWS-X (paraphrase identification). Our general setup for the
structured prediction tasks (POS and NER) is that we train on all available
languages, and downsample to 1,000 samples per language. For the
classification tasks XNLI and PAWS-X, we train on the English training data
and fine-tune on the development sets, as no training data is available for
other languages. Hence, typological differences will be the main factor in our
results, rather than differences in dataset sizes.

###  3.1 Typological Prediction and Blinding

We first investigate whether prohibiting or allowing access to typological
features has an effect on model performance using our architecture. We
hypothesise that our multilingual model will leverage signals related to the
linguistic nature of a task when optimising its its sharing parameters
$\alpha$.

There exists a growing body of work on prediction of typological features
(Daumé III and Campbell, 2007; Murawaki, 2017; Bjerva and Augenstein, 2018b;
Bjerva et al., 2019a, b), most notably in a recent shared task on the subject
(Bjerva et al., 2020). While we are inspired by this direction of research,
our contribution is not concerned with the accuracy of the prediction of such
features, and this is therefore not evaluated in detail in the paper.

Moreover, an increasing amount of work measures the correlation of predictive
performance of cross-lingual models with typological features as a way of
probing what a model has learned about typology Malaviya et al. (2017);
Choenni and Shutova (2020); Gerz et al. (2018); Nooralahzadeh et al. (2020);
Zhao et al. (2020). In contrast to such post-hoc approaches, our experimental
setting allows for measuring the impact of typology on cross-lingual sharing
performance in a direct manner as part of the model architecture.

#### Syntactic Features

We first blind/expose the model to syntactic features from WALS (Dryer and
Haspelmath, 2013). We take the set of word order features which are annotated
for all languages in our experiments, resulting in 33 features. This includes
features such as 81A: Order of Subject, Object and Verb, which encodes what
the preferred word ordering is (if any) in a transitive clause. For all
features, we exclude feature values which do not occur for our set of
languages. We hypothesise that performance will drop for all four tasks, as
they all require syntactic understanding.

#### Morphological Features

We next attempt to blind/expose the model to the morphological features in
WALS. We use the same approach as above, resulting in a total of 8
morphological features. This includes features such as 26A: Prefixing vs.
Suffixing in Inflectional Morphology, indicating to what extent a language
uses prefixing or suffixing morphology. We hypothesise that mainly the POS
tagging task will suffer under this condition, whereas other tasks only to
some extent require morphology.

#### Phonological Features

We next consider a control experiment, in which we attempt to blind/expose the
model to phonological features in WALS. We arrive at a total of 15
phonological features, such as 1A: Consonant Inventories which indicates the
size of the consonant inventory of a language. We expect the performance to
remain relatively unaffected by this task, as phonology ought to have little
importance given a textual input.

#### Genealogical Features

Finally, we attempt to use what one might consider to be language meta-data.
We attempt to blind/expose the model to what language family a language
belongs to. This can be seen as a type of proxy to language similarity, and
correlates relatively strongly with structural similarities in languages.
Because of this correlation with structural similarities, we expect blinding
under this condition to only slightly reduce performance for all tasks, as
previous work has shown this type of relationship not to be central in
language representations (Bjerva et al., 2019c).

###  3.2 Results

In general, we observe a drop in performance when blinding the model to
relevant typological information, and an increase in performance when exposing
the model to it (Table 1). For phonological blinding or prediction, none of
the four tasks is noticeably affected. Although, e.g., both the syntactic and
morphological prediction tasks increase performance on POS tagging, it is not
straightforward to draw conclusions on which of these is the most efficient,
as there is a substantial correlation between syntactic and morphological
features. As for XNLI and PAWS-X, performance notably drops under both the
syntactic and genealogical blinding tasks.

Figure 2: PoS tagging results per language family across blinding and
prediction conditions

Figure 2 shows results for PoS tagging under prediction and blinding across
language families, following the same scheme as Hu et al. (2020).
Interestingly, the syntactic and morphological blinding settings are robust
across all language families, yielding a drop in accuracy across the board.
All other conditions yield mixed results. This further strengthens our
argument that preventing a model from learning syntactic and morphological
features can be severely detrimental.

Model | POS | NER | XNLI | PAWS-X
---|---|---|---|---
\+ Syntactic Blind. | 85.3- | 76.4 | 64.2- | 80.6-
\+ Morphological Blind. | 85.0- | 77.2 | 64.9 | 81.4
\+ Phonological Blind. | 86.7 | 77.1 | 65.0 | 81.6
\+ Genealogical Blind. | 86.1 | 77.0 | 64.7 | 81.1
m-BERT baseline | 86.8 | 77.3 | 65.1 | 81.7
\+ Syntactic Pred. | 87.0 | 77.5 | 65.3+ | 81.9+
\+ Morphological Pred. | 87.2+ | 77.3 | 65.2 | 81.7
\+ Phonological Pred. | 86.7 | 77.1 | 65.0 | 81.7
\+ Genealogical Pred. | 87.0 | 77.6 | 65.3+ | 81.8

Table 1: Typological Blinding and Prediction. Mean POS accuracy, NER F1
scores, XNLI accuracy and PAWS-X accuracy across all languages. + and -
indicate significantly better or worse performance respectively, as determined
by a one-tailed t-test ($p<0.01$).

###  3.3 The Effect of Typology on Latent Architecture Learning

The results show that preventing access to typological features hampers
performance, whereas providing access improves performance. We now turn to an
analysis of how the model shares parameters across languages in this setting.
Our hypothesis is that blinding will prevent models from sharing parameters
between similar languages, in spite of typological similarities. Concretely,
we expect that the drop in POS tagging performance under morphological
blinding is caused by lower $\alpha$ weights between languages which are
morphologically similar, and higher $\alpha$ weights between languages which
are dissimilar. Recall that these parameters are latent variables learned by
the model, regulating the amount of sharing between languages (see Eq. 1). We
investigate the correlations between the $\alpha$ sharing parameters, and two
proxies of language similarity. We focus on the POS task, as the results from
the typological blinding and prediction experiments were the most pronounced
here, as both morphological and syntactic blinding affected performance.

Our first measure of language similarity is based on Bjerva et al. (2019c),
who introduce what they refer to as structural similarity. This is based on
dependency statistics from the Universal Dependencies treebank (Zeman et al.,
2020), resulting in vectors which describe how different syntactic relations
are used in each language. Previous work has shown that this measure of
similarity correlates strongly with that learned in embedded language spaces
during multilingual training. In addition to considering these dependency
statistics, we also use language embeddings drawn form Östling and Tiedemann
(2017). For each language similarity measure we calculate its pairwise Pearson
correlation with the $\alpha$ values learned under each condition.

Table 2 shows correlations between $\alpha$ weights and similarities increase
when predicting typological features, and decreases when blinded to such
features. Hence, when the model has indirect access to, e.g., the SVO word
ordering features of languages, sharing also reflects this.

Model | Struct. | Lang. Emb.
---|---|---
Syntactic Blind. | 0.31 | 0.27
Morphological Blind. | 0.34 | 0.29
Phonological Blind. | 0.40 | 0.41
Genealogical Blind. | 0.29 | 0.31
No blind./pred. | 0.43 | 0.40
Syntactic Pred. | 0.52 | 0.53
Morphological Pred. | 0.49 | 0.56
Phonological Pred. | 0.41 | 0.39
Genealogical Pred. | 0.47 | 0.38

Table 2: Pearson correlations between $\alpha$ weights and language similarity
measures.

##  4 Discussion

We have shown that blinding a multilingual model to typological features
severely affects sharing across a relatively large language sample, and for
several NLP tasks. The effects on model performance, as evaluated over 40
languages and 4 tasks from XTREME (Hu et al., 2020), were the largest for POS
tagging. The fact that smaller effects were observed for NER, could be because
this task relies more on memorising NEs rather than using (morpho-)syntactic
cues (Augenstein et al., 2017). Furthermore, the relatively small effects on
XNLI and PAWS-X can also be interpreted as evidence for that typology is less
important in these tasks than in more traditional linguistic analysis.

A potential critique of our approach is that it merely blinds the model to
language identities. This could be the case, if only some latent
representation of, e.g., “SVO” ordering is used to represent a language
identity. However, previous work has shown that morphological information is
encoded by the type of model we investigate. Hence, since we only blind
features in a single category at a time, we expect that the model’s
representation of language identities is unaffected.

Not only do we observe a drop in performance when blinding a model to
syntactic features, but we also observe that the $\alpha$ sharing weights in
our model do not appear to correlate with linguistic similarities in this
setting. Conversely, encouraging a model to consider typology, by jointly
optimising it for typological feature prediction, improves performance in
general. Furthermore, $\alpha$ weights in this scenario converge towards
correlating with structural similarities of languages. This is in line with
recent work which has found that m-BERT uses fine-grained syntactic
distinctions in its cross-lingual representation space (Chi et al., 2020).

We interpret this as evidence for the fact that typology can be a necessity
for modelling in NLP. Our results furthermore corroborate previous work in
that we only find moderate benefits from including typological information
explicitly. We expect that this to a large degree is due to the typological
similarities of languages being encoded implicitly based on correlations
between patterns in the input data. As low-resource languages often do not
even have access to any substantial amount of raw text, but often do have
annotations in WALS, we expect that using typological information can go some
way towards building truly language-universal models.

##  5 Conclusions

We have shown that preventing access to typology can impede the performance of
cross-lingual sharing models. Investigating latent weights governing the
sharing between languages shows that this prevents the model from sharing
between typologically similar languages, which is otherwise learned based on
patterns in the input. We therefore expect that using typological information
can be of particular interest for building truly language-universal models for
low-resource languages.

## Acknowledgements

This research has received funding from the Swedish Research Council (grant No
2019-04129), and the NVIDIA Corporation (Titan Xp GPU).

## References

  * Ammar et al. (2016) Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A Smith. 2016.  Many languages, one parser.  _Transactions of the Association for Computational Linguistics_ , 4:431–444.
  * Augenstein et al. (2017) Isabelle Augenstein, Leon Derczynski, and Kalina Bontcheva. 2017.  Generalisation in named entity recognition: A quantitative analysis.  _Computer Speech & Language_, 44:61–83.
  * Bjerva and Augenstein (2018a) Johannes Bjerva and Isabelle Augenstein. 2018a.  From phonology to syntax: Unsupervised linguistic typology at different levels with language embeddings.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 907–916, New Orleans, Louisiana. Association for Computational Linguistics.
  * Bjerva and Augenstein (2018b) Johannes Bjerva and Isabelle Augenstein. 2018b.  Tracking Typological Traits of Uralic Languages in Distributed Language Representations.  In _Proceedings of the Fourth International Workshop on Computational Linguistics of Uralic Languages_ , pages 76–86, Helsinki, Finland. Association for Computational Linguistics.
  * Bjerva et al. (2019a) Johannes Bjerva, Yova Kementchedjhieva, Ryan Cotterell, and Isabelle Augenstein. 2019a.  A probabilistic generative model of linguistic typology.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1529–1540, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Bjerva et al. (2019b) Johannes Bjerva, Yova Kementchedjhieva, Ryan Cotterell, and Isabelle Augenstein. 2019b.  Uncovering probabilistic implications in typological knowledge bases.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3924–3930, Florence, Italy. Association for Computational Linguistics.
  * Bjerva et al. (2020) Johannes Bjerva, Elizabeth Salesky, Sabrina J. Mielke, Aditi Chaudhary, Celano Giuseppe, Edoardo Maria Ponti, Ekaterina Vylomova, Ryan Cotterell, and Isabelle Augenstein. 2020.  SIGTYP 2020 shared task: Prediction of typological features.  In _Proceedings of the Second Workshop on Computational Research in Linguistic Typology_ , pages 1–11, Online. Association for Computational Linguistics.
  * Bjerva et al. (2019c) Johannes Bjerva, Robert Östling, Maria Han Veiga, Jörg Tiedemann, and Isabelle Augenstein. 2019c.  What Do Language Representations Really Represent? _Computational Linguistics_ , 45(2):381–389.
  * Caruana (1997) Rich Caruana. 1997.  Multitask learning.  _Machine Learning_ , 28 (1):41–75.
  * Chen et al. (2019) Steven Chen, Nicholas Carlini, and David Wagner. 2019.  Stateful detection of black-box adversarial attacks.  _arXiv preprint arXiv:1907.05587_.
  * Chi et al. (2020) Ethan A Chi, John Hewitt, and Christopher D Manning. 2020.  Finding universal grammatical relations in multilingual bert.  _arXiv preprint arXiv:2005.04511_.
  * Choenni and Shutova (2020) Rochelle Choenni and Ekaterina Shutova. 2020.  What does it mean to be language-agnostic? probing multilingual sentence encoders for typological properties. _CoRR_ , abs/2009.12862.
  * Daumé III and Campbell (2007) Hal Daumé III and Lyle Campbell. 2007.  A Bayesian Model for Discovering Typological Implications.  In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_ , pages 65–72, Prague, Czech Republic. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dryer and Haspelmath (2013) Matthew S. Dryer and Martin Haspelmath, editors. 2013.  _WALS Online_.  Max Planck Institute for Evolutionary Anthropology, Leipzig.
  * Ganin and Lempitsky (2014) Yaroslav Ganin and Victor Lempitsky. 2014.  Unsupervised domain adaptation by backpropagation.  _arXiv preprint arXiv:1409.7495_.
  * Gerz et al. (2018) Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018.  On the relation between linguistic typology and (limitations of) multilingual language modeling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 316–327, Brussels, Belgium. Association for Computational Linguistics.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.  _arXiv preprint arXiv:2003.11080_.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the nlp world.  _arXiv preprint arXiv:2004.09095_.
  * de Lhoneux et al. (2018) Miryam de Lhoneux, Johannes Bjerva, Isabelle Augenstein, and Anders Søgaard. 2018\.  Parameter sharing between dependency parsers for related languages.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4992–4997.
  * Malaviya et al. (2017) Chaitanya Malaviya, Graham Neubig, and Patrick Littell. 2017.  Learning language representations for typology prediction.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2529–2535, Copenhagen, Denmark. Association for Computational Linguistics.
  * Murawaki (2017) Yugo Murawaki. 2017.  Diachrony-aware induction of binary latent representations from typological features.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 451–461, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Nooralahzadeh et al. (2020) Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020.  Zero-Shot Cross-Lingual Transfer with Meta Learning.  In _Proceedings of EMNLP_. Association for Computational Linguistics.
  * O’Horan et al. (2016) Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, and Anna Korhonen. 2016\.  Survey on the use of typological information in natural language processing.  _arXiv preprint arXiv:1610.03349_.
  * Oncevay et al. (2020) Arturo Oncevay, Barry Haddow, and Alexandra Birch. 2020.  Bridging linguistic typology and multilingual machine translation with multi-view language representations.  In _Proceedings of EMNLP_. Association for Computational Linguistics.  ArXiv preprint arXiv:2004.14923.
  * Östling and Tiedemann (2017) Robert Östling and Jörg Tiedemann. 2017.  Continuous multilinguality with language vectors.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 644–649, Valencia, Spain. Association for Computational Linguistics.
  * Ponti et al. (2019) Edoardo Maria Ponti, Helen O’horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019.  Modeling language variation and universals: A survey on typological linguistics for natural language processing.  _Computational Linguistics_ , 45(3):559–601.
  * Ruder et al. (2019) Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2019\.  Latent multi-task architecture learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 4822–4829.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Zeman et al. (2020) Daniel Zeman, Joakim Nivre, and Mitchell Abrams et al. 2020.  Universal dependencies 2.6.  LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
  * Zhang et al. (2019) Huan Zhang, Hongge Chen, Zhao Song, Duane Boning, Inderjit S Dhillon, and Cho-Jui Hsieh. 2019.  The limitations of adversarial training and the blind-spot attack.  _arXiv preprint arXiv:1901.04684_.
  * Zhao et al. (2020) Wei Zhao, Steffen Eger, Johannes Bjerva, and Isabelle Augenstein. 2020.  Inducing Language-Agnostic Multilingual Representations.  _arXiv preprint arXiv:2008.09112_.