2022

[3]Daniela Moctezuma These authors contributed equally to this work.

These authors contributed equally to this work.

These authors contributed equally to this work.

These authors contributed equally to this work.

1]Conacyt, Consejo Nacional de Ciencia y Tecnología., Av. Insurgentes Sur
1582, Col. Crédito Constructor., 03940, CDMX, Mexico

2]INFOTEC, Centro de Investigación e Innovación en Tecnologías de la
Información y Comunicación, Circuito Tecnopolo Norte, No.112 Col. Tecnopolo
Pocitos II, Aguascalientes, 20326, Aguascalientes, Mexico

3]CentroGEO, Centro de Investigación en Ciencias de Información Geoespacial.,
Circuito Tecnopolo Norte, No.107 Col. Tecnopolo Pocitos II, Aguascalientes,
20313, Aguascalientes, Mexico

4]UPIITA-IPN, Instituto Politécnico Nacional, Av. Instituto Politécnico
Nacional 2580 Col. Barrio la Laguna Ticomán, Gustavo A. Madero, 07360, Mexico
City, Mexico

# Regionalized models for Spanish language variations based on Twitter

Eric S. Tellez<EMAIL_ADDRESS><EMAIL_ADDRESS>Sabino
Miranda<EMAIL_ADDRESS>Mario Graff<EMAIL_ADDRESS>Guillermo Ruiz<EMAIL_ADDRESS>[  [  [  [

###### Abstract

Spanish is one of the most spoken languages in the world. Its proliferation
comes with variations in written and spoken communication among different
regions. Understanding language variations can help improve model performances
on regional tasks, such as those involving figurative language and local
context information. This manuscript presents and describes a set of
regionalized resources for the Spanish language built on four-year Twitter
public messages geotagged in 26 Spanish-speaking countries. We introduce word
embeddings based on FastText, language models based on BERT, and per-region
sample corpora. We also provide a broad comparison among regions covering
lexical and semantical similarities and examples of using regional resources
on message classification tasks.

###### keywords:

linguistic resources, semantic space, Spanish Twitter

##  1 Introduction

Communication is, at its core, an understanding task. Understanding a message
implies that peers know the vocabulary and structure; i.e., the receiver
obtains what the sender intended to say. Language is a determinant factor in
any communication. Even people who speak the same language can find
difficulties communicating information due to slight language variations due
to regional variations, language evolution, cultural influences, and
informality, to name a few.

A dialect is a language variation that diverges from its origin due to several
circumstances. Dialects can differ regarding their vocabulary, grammar, or
even semantics. The same sentence can be semantically different among
dialects. In contrast, people of different dialects may need help
understanding sentences with the same meaning. This effect is notoriously
complex for figurative language since it contains cultural and ideological
references. Studying these dialects can help us understand the cultural
aspects of each population and the closeness between them. In this sense,
dialectometry studies the regional distribution of dialects. Similarly,
dialectometric analysis has as its objective, through a computational
approach, to analyze this distribution and provide a quantitative linguistic
distance between each dialect or region of it Donoso and Sánchez (2017).
Hence, the research in dialectology tries to understand language differences,
innovations, or variations not only in space but also in time through several
phenomena. Natural Language Processing (NLP) tools can analyze written
communications automatically. However, the support for regional languages is
still in its early stages in NLP, particularly for languages different from
English.

Another important aspect to discuss is about if dialects are directly related
to space or geographical boundaries. As Penny penny2000variation  discuss that
language people who live in different territory do not speak the same way
because the neighboring locality plays a key role. Also, the social and
historical language variations are discussed in penny2000variation . In some
way, with the analysis done we hope to contribute a little to this exciting
discussion. A more linguistic comparison from Spanish variants in America
could be found in cotton1988spanish .

On the other side, social media is a crucial component of our lives; Facebook,
Twitter, Instagram, and Youtube are among the most used social networks that
allow interaction among users in written form and other media. In particular,
Twitter is a micro-blogging platform where most messages are intentionally
publicly available, and developers and researchers can access these messages
through an API (Application Programming Interface). Twitter’s messages are
known as tweets. Each tweet contains text and additional metadata like the
user that wrote it and the geographic location where it was published. In the
case of social media, where the source of the messages is informal, the errors
are another source of variability in the language. This kind of messaging may
impose extra difficulties than in formal written documents.

Nonetheless, Twitter messages’ quality and socio-demographic
representativeness have been continuously questioned Crampton et al. (2013).
Some authors have shown that despite the over-representation of some social
groups, social media usage can still be of enormous usefulness and quality
Huang et al. (2016). Language and geographical information are crucial to
understanding the geographies of this online data and how some information
related to economic, social, political, and environmental trends could be used
Graham, Hale, and Gaffney (2014).

It is easy to observe the relevance of having quality and realistic data for
language variations analysis or model generation with this background.
Sometimes, it is difficult to have a regional-specific corpus. A large corpus
is necessary to learn language models and achieve confident analysis; more
data often imply better models. With this kind of resources acquired, for
instance, from social media platforms, many potential research and
applications have been made in many research areas, such as health paul2011you
, environmental issues mooney2009evaluating , emotion suhasini2020emotion ,
mental health finfgeld2015twitter , gender vashisth2020gender , and misogyny
frenda2019online , among others.

There are several examples of linguistic resources attending specific tasks or
applications, such as the study of Down syndrome in Escudero-Mancebo, David
and Corrales-Astorgano, Mario and Cardeñoso-Payo, Valentín and Aguilar,
Lourdes and González-Ferreras, César and Martínez-Castilla, Pastora and
Flores-Lucas, Valle (2022). Also, on CKennedy, Brendan et al. (2022), a corpus
is proposed for hate-based rhetoric, or hate recognition. On Gruszczyński,
Włodzimierz and Adamiec, Dorota and Bronikowska, Renata and Kieraś, Witold and
Modrzejewski, Emanuel and Wieczorek, Aleksandra and Woliński, Marcin (2022),
the authors provide a corpus of up to 13.5 million tokens of Polish texts
between 1601 and 1772. As can be seen, a variety of corpus has been generated
by the community to deal with general or more specific NLP tasks.

On the other hand, the Spanish language variations are also studied in the NLP
research community. For instance, in Gonçalves and Sánchez (2014), authors
present a crowdsourcing language diatopic variation using Twitter data with
geolocation, employing tweets messages in Spanish for more than two years over
the globe. The analysis was made with a set of pre-established words and
counting each and its variations worldwide. In this sense, to know what
regions are close to each other, the authors used the k-means clustering
algorithm over these words frequencies and PCA (Principal Component Analysis)
to reduce data into two-dimensional space for visualization. The clustering
approach identifies large macro-regions sharing language characteristics.

Unfortunately, for the Spanish language, there are few works, contrary to some
language variations from Europe, e.g., English dialects Hovy et al. (2020),
French dialects lamontagne2022phonological  and Arabic alshutayri2017exploring
, to name some.

One of the possible assumptions using Twitter is that the behaviors of English
users generalize to other language users. In hong2011language  is presented a
study using 62 million tweets over more than 100 different languages over four
weeks. Applying an automatic language detection algorithm, they found that
most of the data were in English (51%), and 39% was for other languages such
as Japanese, Portuguese, Indonesian, and Spanish.

The geographical region of a language helps to know how this language is used
in a particular society. For instance, Spanish is a largely used language;
nevertheless, it is used differently according to the country or even a more
specific geographical location. Hence, the language could analyze at the
regional level Huang et al. (2016); Rodriguez-Diaz et al. (2018). In
Rodriguez-Diaz et al. (2018), the authors study Spanish language variations in
Colombia. The analysis used unigram features, and the authors stated that it
was challenging to compare Spanish variations against regions identified by
other authors using classical dialectometry. Hence, in conclusion, the authors
said that automatic detection of dialectones is an adequate alternative to
classical methods in dialectometry for automated language applications. A more
current effort to deal with the Spanish Language on Twitter was presented by
huertas2022bertuit , which provides a powerful tool to take advantage of
transformers generated with the native language. The authors test their
solution with several classical NLP tasks.

The number of articles related to other languages is also reduced. In
alshutayri2017exploring , the Arabic dialects were classified using the WEKA
tool111https://www.cs.waikato.ac.nz/ml/weka/ reaching an accuracy of 79% on
their classification results. The used dataset contained 210,915K tweets from
some Arabic dialects and classified them considering their geographic
location.

Mocanu et al. Mocanu et al. (2013) survey the linguistic landscape in the
world using Twitter. This landscape includes linguistic homogeneity or
variations over countries that consider the touristic seasons. The method
employed to identify language is the Chromium Compact Language Detector by
Google; the authors also used the location of the devices reported by tweets.
As a result, it was possible to observe distributions of language on Twitter
over several countries by month of the year and where touristic flow is
evident.

Also, emoticons or emojis are effective communication symbols. Their usage has
been studied in the literature. For instance, in park2013emoticon , authors
analyzed the semantic, cultural, and social aspects of their use on Twitter.
Kejriwal et al. Kejriwal et al. (2021) studied the use of emojis in terms of
linguistic use and countries. The authors collected tweets from 30 different
languages and countries, and the authors found that emojis usage strongly
correlates between language and country level, which means that emojis are
used according to language and region. Another example of studying Emojis is
presented in li2019empirical .

Other efforts have been made to exploit the regionalized models for a specific
language variation; for instance, in jimenez2018automatic  where methods to
identify regional words and provide their meaning is studied.

Our contribution is a set of regionalized resources for different variations
of the Spanish language. We created and characterized regional vocabularies
and regional semantic representations of them (i.e., word embeddings). Also,
we learn and test language models based on BERT. We built these resources from
an extensive collection of public tweets from 2016 to 2019, written in 26
countries with a large basis of Spanish-speaking people. Regarding messages,
we provide a sample of Twitter message identifiers divided by region such that
researchers can retrieve them easily. Finally, we show some usage examples of
our resources.

The rest of the manuscript is organized as follows. Section 2 describes our
Twitter Spanish Corpora (TSC), used to generate our regional resources.
Section 3 compare lexical traits among the corpora. Section 4 is dedicated to
presenting our semantic resources and their affinity analysis that includes
visualizations and experimental evidence that support the use of regional word
embedding models on regional tasks. Our resources based on language models are
presented and compared in Section 5. Finally, Section 6 summarizes and
discusses the implications of the TSC.

##  2 Twitter corpora of the Spanish language

With 489 million native speakers in
2020222https://blogs.cervantes.es/londres/2020/10/15/spanish-a-language-
spoken-by-585-million-people-and-489-million-of-them-native, Spanish is one of
the languages with a higher native speaking basis, just ranked behind Chinese
Mandarin in terms of the number of native speakers. Twenty-one countries have
the Spanish language as the official language (by law or de
facto)333https://en.wikipedia.org/wiki/List_of_countries_where_Spanish_is_an_official_language.
Our corpora selected these regions, see Table 1; and we also considered five
additional regions (US, CA, GB, FR, and BR) with well-known migration,
business, and tourism activities of Spanish speakers. The number of Twitter
users varies with each country; since each country has different social,
political, security, health, and economic conditions, we will avoid
generalizations.

As mentioned, we collected publicly published tweets between 2016 and 2019
using the Twitter stream API. Also, we limited our collection to geotagged
messages marked by Twitter as written in Spanish. We decided to let out the
corpus messages from the year 2020 and posteriors to avoid disturbances in
social media regarding the COVID-19 pandemic. Please recall our objective is
to build resources based on the language itself and not analyzing the pandemic
event. Twitter stream API allows tweet retrieval in two ways. The first
consists of using a language marker (lang=es, for Spanish) and a list of
tracking words linked to the specified language. In this case, we can use
Spanish stopwords444Words that are so common in a language, such as articles,
prepositions, interjections, and auxiliary verbs, among other typical words.
to maximize the download process. The second strategy consists in using a
language marker (lang=es, for Spanish) and geographical coordinates, these
kinds of tweets are named geotagged Tweets. We specify worldwide coordinates
to get tweets from everywhere. We use only geotagged tweets from the last
strategy. These geotagged tweets have information such as country code
corresponding to the country where the tweet was published, among other
metadata. We rely on the information provided by Twitter about the country
associated with each tweet.

To ensure a minimum amount of information in each tweet, we discard those
tweets with less than five tokens, i.e., words, emojis, or punctuation
symbols, following the strategy of mikolov2013distributed  for analyzing and
learning from very large collections. We also removed all retweets to avoid
duplication of messages and reduce foreign messages commented on by Spanish
speakers. After this filtering procedure, we retain close to $800$ million
messages.

Table 1: Datasets’ statistics after filtering by retweets and ensuring at
least five words per tweet. We show the origin country, the country code in
ISO 3166-1 alpha-2 format reported by the Twitter API, the number of tweets,
and the number of different users in the collected period.

country |  code |  $\alpha$ |  $\beta$ |  number of users |  number of tweets |  number of tokens
---|---|---|---|---|---|---
Argentina |  AR |  0.7563 |  1.8594 |  1,376K |  234.22M |  2,887.92M
Bolivia |  BO |  0.7509 |  1.8913 |  36K |  1.15M |  20.99M
Chile |  CL |  0.7555 |  1.8874 |  415K |  45.29M |  719.24M
Colombia |  CO |  0.7562 |  1.8993 |  701K |  61.54M |  918.51M
Costa Rica |  CR |  0.7447 |  1.8595 |  79K |  7.51M |  101.67M
Cuba |  CU |  0.7640 |  1.8677 |  32K |  0.37M |  6.30M
Dominican Republic |  DO |  0.7544 |  1.8832 |  112K |  7.65M |  122.06M
Ecuador |  EC |  0.7538 |  1.8968 |  207K |  13.76M |  226.03M
El Salvador |  SV |  0.7494 |  1.9066 |  49K |  2.71M |  44.46M
Equatorial Guinea |  GQ |  - |  - |  1K |  8.93K |  0.14M
Guatemala |  GT |  0.7498 |  1.9175 |  74K |  5.22M |  75.79M
Honduras |  HN |  0.7486 |  1.8941 |  35K |  2.14M |  31.26M
Mexico |  MX |  0.7557 |  1.8895 |  1,517K |  115.53M |  1,635.69M
Nicaragua |  NI |  0.7445 |  1.8535 |  35K |  3.34M |  42.47M
Panama |  PA |  0.7559 |  1.8952 |  83K |  6.62M |  108.74M
Paraguay |  PY |  0.7511 |  1.8815 |  106K |  10.28M |  141.75M
Peru |  PE |  0.7583 |  1.8966 |  271K |  15.38M |  241.60M
Puerto Rico |  PR |  0.7498 |  1.8433 |  18K |  0.58M |  7.64M
Spain |  ES |  0.7648 |  1.9036 |  1,278K |  121.42M |  1,908.07M
Uruguay |  UY |  0.7516 |  1.8346 |  157K |  30.83M |  351.81M
Venezuela |  VE |  0.7614 |  1.8959 |  421K |  35.48M |  556.12M
Brazil |  BR |  0.7681 |  1.9389 |  1,604K |  27.20M |  142.22M
Canada |  CA |  0.7652 |  1.9331 |  149K |  1.55M |  21.58M
France |  FR |  0.9372 |  1.9324 |  292K |  2.43M |  27.73M
Great Britain |  GB |  0.7687 |  1.9129 |  380K |  2.68M |  34.62M
United States of America |  US |  0.7666 |  1.8929 |  2,652K |  40.83M |  501.86M
Total |  |  |  |  12M |  795.74M |  10,876.25M

Table 1 shows statistics about our corpora describing aspects such as country,
number of users, number of tweets, and number of tokens. The table shows that
Spain, the USA, Mexico, and Argentina are countries with more users.
Furthermore, they are also those with more tweets in the Spanish language, but
the USA falls considerably in this aspect. A similar proportion is observed in
the number of tokens column. However, Argentina has the highest number of
tokens, above Mexico and Spain significantly.

The table also lists the coefficients for the expressions behind Heaps’ and
Zipf’s laws. Both laws are broadly surveyed in the literature; for instance,
Gelbukh2001  and (schutze2008introduction, , Chapter 5) describe them and
study their implications from a general perspective. In a nutshell, the laws
describe how the vocabulary grows in text collections written in non-severe-
agglutinated languages. Heaps’ law $n^{\alpha}$ describes the sub-linear
growth of the vocabulary on a growing collection of size $n$. Zipf’s law
represents a power-law distribution where a few terms have very high
frequencies, and many words occur with a shallow frequency in the collection.
The expression that describes Zipf’s law is $1/{r^{\beta}}$, where $r$ is the
rank of the term’s frequency.

Figure 1(a) illustrates the Heaps’ law in a small sample of regions of
interest. One can observe its predicted sub-linearity and that Mexico has the
lowest growth in its vocabulary size according to the number of tokens. On the
contrary, the US corpus shows faster vocabulary growth, possibly explained due
to the mix of languages in many messages.

Figure 1(b) shows Zipf’s law under a log-log scale and its quasi-linear shape.
We can see slight differences among curves, more noticeable on both the left
and right parts of the plot. The left part of the curves corresponds to those
terms with very high frequency, and the right side is dedicated to those terms
being rare in the collection. Notice that all these curves are similar but
slightly different; this is not a surprise since we analyze variations of the
same idiom, i.e., the Spanish language.

(a) Heaps law. Vocabulary size with respect to the number of tokens.

(b) Zipf’s law. Frequency of tokens.

Figure 1: The vocabulary growth and distribution of frequencies of $10^{7}$
tokens over a sample of our Twitter’s Spanish language corpora.

###  2.1 Geographic distribution

Figure 2(a) illustrates the number of collected Spanish language tweets over
the world. The color intensity is on a logarithmic scale, which means that
slight variations in the color imply significant changes in the number of
messages; countries with the darkest blue have the highest number of tweets in
Spanish. This figure shows how American countries (in the south, central, and
north) and the Iberian Peninsula have, as expected, more tweets in the Spanish
language than the rest of the world.

Figure 2(b) shows the distribution of tweeters (users) per country. As in the
previous image, we present a logarithmic scale in the intensity of color to
represent the number of users. The differences between this figure and Figure
2(a) are low, as expected, and follow the same distribution. Note the high
intensity of American countries.

(a) Distribution of tweets tagged as Spanish by Twitter.

(b) Distribution of the number of tweeters with at least one tweet tagged as
Spanish language.

Figure 2: Distribution of tweets and tweeters labeled as Spanish-speaking
users around the world. Colors are related to the logarithmic frequencies in
data collected from 2016 to 2019 with the public Twitter API stream. Darker
colors indicate a high population; the logarithmic scale implies that only
significant frequency differences produce color changes. Figure 3: Ratio
between the number of tweets produces by local tweeters and the total tweets
in the country.

Also, it is essential to know whether the tweets come from persons living in
the country or travelers. It might be impossible to completely answer this by
looking only at the tweets. However, an approach that can provide insight into
the locality of the tweets is to measure the ratio between the tweets produced
by local users and the total number of tweets produced in each country, where
each user is assigned to the country where more tweets have. For example,
suppose a tweeter has 100 tweets in Mexico and 10 in Spain. In that case, that
user is considered Mexican, and those published in Spain (considered there a
tourist) are not considered, which means only the tweets produced in Mexico
are counted.

Figure 3 presents the ratio between local tweets and the total number of
tweets; it can be observed that Puerto Rico (PR) has the lowest ratio, meaning
that most tweets come from foreigners. The second lowest is Equatorial Guinea
(GQ), where more than 60% of its tweets are local. Then, there is a block of
countries where Spanish is not the primary language. After the block comes
Cuba, with 88% of its tweets produced by locals. The United States of America
(US) has a ratio of 90%. From the Dominican Republic (DO) to the right have a
percentage higher than 95%. The median ratio in these countries is 98%. This
effect could indicate most of the tweets are produced by residents.

##  3 Lexical resources

This section describes and analyzes our Spanish Twitter Corpora (STC) in the
lexical aspect, specifically from the vocabulary usage perspective. This
analysis complements that given of the Heaps’ and Zipf’s laws and the
information given in Table 1.

Figure 4 describes the procedure applied to obtain an affinity matrix of our
Spanish corpora.555Under our context, an affinity matrix is a pair-wise matrix
of distances among different regions using a dissimilarity function. The $i$th
row contains the distance of the $i$ region vs. all regions; it has a zero
diagonal. For this purpose, we extracted the vocabulary of each corpus, i.e.,
a matrix that describes the similarities among corpora. The vocabulary was
computed on the entire corpus after text normalizations described in the
diagram. We also removed those terms with less than five occurrences in the
corpus to remove the tail of the term-frequency distribution, similarly to
mikolov2013distributed ; bojanowski2017enriching . The remaining terms are
used to create a vector that represents the regional corpus.

The affinity matrix is computed using the cosine distance described in the
flow diagram. Note that we select the cosine distance as metric due to the
reminiscences of the traditional bag of words with our vocabulary
representation, see schutze2008introduction  for more information about
conventional vector models for information retrieval. The heatmap represents
the actual values in the matrix. This matrix is crucial for the rest of this
analysis since it contains distances (dissimilarities) among all pairs of our
Spanish corpora. Values close to zero (darker colors) imply that those regions
are pretty similar, and lighter ones (close to one) are those regions with
higher differences in their vocabularies. For instance, the affinity matrix
can show us how Mexico (MX) is more similar to Honduras (HN), Nicaragua (NI),
Peru (PE), and the USA (US). This behavior could be the geographical location
of the countries, and therefore, a large migration or cultural interchange is
made. On the other hand, Brazil (BR) and Equatorial Guinea (GQ) are among the
most atypical countries with low similarities with the other countries.

Figure 5 illustrates the similarity between Twitter country vocabularies. Here
we rely on Uniform Manifold Approximation and Projection (UMAP) algorithm
mcinnes2020umap , a non-linear dimension reduction technique that approximates
the $k$ nearest neighbor graph structure of a dataset in the projected low
dimension. We applied UMAP projections (2D for spatial projection and 3D for
colorizing points) using the affinity matrix as input. Please recall that the
affinity matrix represents similarities between regional vocabularies using
the cosine distance as the metric. Please remember that the UMAP algorithm
uses the affinity matrix to generate the low-dimensional projection. To our
best knowledge, this is a novel approach to visualizing similarities among
vocabularies. Still, we can find several uses of dimensional reduction
techniques like tSNE wada2018unsupervised  for visualizing word meaning
similarities in a single multi-language model. UMAP is a more recent non-
linear dimensional reduction technique that typically performs faster than
other non-linear alternatives like tSNE or ISOMAP with remarkable stability on
projections mcinnes2020umap . Interested readers on alternatives are
referenced to the recent literature anowar2021dimension .

The figure shows how close or far each Spanish variation is among the entire
corpora. UMAP is parameterized by the number of nearest neighbors (k$nn$) in
the affinity matrix. The number of neighbors accepts values between $k=2$ and
$n$, i.e., the number of elements in the collection. Small values of $k$
capture local characteristics of the graph’s structure, while large $k$ values
capture global structures.

The figure shows the projection using 3$nn$;666The value $k=3$ was chosen
after several tests, larger values capture more global characteristics, and
$k=2$ produce many sparse clusters (local characteristics). we can see four
well-defined clusters here. For instance, Uruguay (UY) is very close to
Argentina (AR) in three figures, and this is the case in other countries, like
Mexico (MX), Colombia (CO), and the United States (US); or Venezuela (VE), and
Ecuador (EQ).

Figure 4: Affinity matrix among Spanish regions’ vocabularies.

Figure 5: Spanish-language lexical similarity visualization among country’s
vocabularies through a two-dimensional UMAP projection using the Cosine among
vocabularies. The points were colorized using a 3D UMAP projection (normalized
and interpreted as RGB). Both projections use three nearest neighbors, which
emphasizes local features. Figure 6: Regional Vocabulary in RGB representation

While some of these clusters support the idea that geographical similarities
imply language similarities, there are notorious exceptions. Figure 6 shows a
colorized map using the same colors encoding of Figure 5. While it is possible
to observe similarities and divisions among North America, Central and South
America, and European countries, there are essential differences. For
instance, Colombia (CO), a South American country, has more similarities to
Central American language variants.

Regarding our lexical features, Cuba and the Dominican Republic are close to
Venezuela, Bolivia, and Ecuador. It is interesting to recall that these
similarities are present in Twitter and may vary from other data sources.
Still, it could be helpful to take advantage of this knowledge.

Our collections are small for some countries, e.g., GQ, PR, and CU, which can
introduce some possible issues in our analysis. For instance, it is possible
to declare similarities that are non-meaningful. Please recall that the UMAP
projection uses the $k$ nearest neighbor graph (that takes the affinity matrix
of Fig. 4 as input). Even when other regions do not select these regions as
neighbors (please recall we used $k=3$), these regions will have direct
neighbors that can have enough data. For instance, Fig. 4 shows Cuba with a
light row column, which means that most regions are seen relatively far, but
it connects Bolivia. Bolivia has more strong connections that positioned Cuba
on the map. Another effect occurs with Brazil that even when it has a large
corpus, it contains a lot of messages mixing Spanish and Portuguese. It is
pretty different from most regions but visible closer to FR and GB, and
therefore, BR will be placed near them on the 2D and 3D projections, see Fig.
5.

Figure 7: Most popular emojis per Spanish-speaking country.

In addition, emojis are graphical symbols expressing an emotion or popular
concepts. Hence, they are a lexical resource that can also imply an emotional
charge. Emojis compensate for the lack of facial expressions and other
expressive ways of face-to-face conversations. Therefore, emojis are popular
on social networks like Twitter since they are concise and friendly ways to
communicate Dresner and Herring (2010). The use of emojis is also dependent on
the region, as illustrated in Figure 7. The figure shows the 32 most used
emojis in each country; skin tone markers were separated from composed emojis
and counted in an aggregated way. Note that the most popular emojis have
consensus in almost all regions. In top rank, we found the laughing face, the
in love face, and the heart (love). Another symbol that deserves attention is
the color-skin mask, which marks emojis with a skin hue. Regarding
frequencies, lighter color-skin marks are more popular than darker ones; this
information could have different meanings. For example, users identified as
white people, or perhaps it is tricky to select the proper one with Twitter
clients. The real reason behind this finding is beyond the scope of this
manuscript but deserves attention.

##  4 Semantic analysis and regional word embeddings

This section discusses the creation of regional semantic representations (word
embeddings) for our Spanish language corpora and also analyze similarities
between regions using visualization techniques. Word embeddings are vector
representations of a vocabulary that capture the semantics of words learning
how words are used in a large text corpus. Algorithms learns a high
dimensional vector for each token using a distributional hypothesis: words
used in similar contexts have similar semantics. Therefore, if two vectors are
close, both are semantically related; the contrary also becomes true, as two
distant vectors are different semantically. In summary, embeddings are a
popular and effective way to capture semantics from a corpus Yang, Macdonald,
and Ounis (2018). There exist several techniques to learn word embeddings, for
instance, Word2Vec Mikolov et al. (2013a), FastText Joulin et al. (2017);
bojanowski2017enriching , and Glove Pennington, Socher, and Manning (2014).
Our resources are FastText models, to support out-of-vocabulary words, which
are common in social network data. FastText is both a word representation
generator and a text classification tool. It is an open-source library well-
known for its broad language coverage.777https://fasttext.cc/ For instance,
Grave et al. Grave, Edouard and Bojanowski, Piotr and Gupta, Prakhar and
Joulin, Armand and Mikolov, Tomas  trained word embeddings for 157 languages
using Wikipedia (800 million tokens) and Common Crawl (70 billion tokens);
these models include support for the Spanish language. Nonetheless, there is a
lack of country-level language support to our knowledge. Our resources are the
first broad effort on this matter, making it possible to take advantage of
regionalisms and Spanish dialects.

We use our Twitter corpora, divided by country, and apply our preprocessing
step, described in Section 3, as input of the FastText algorithm. Before, we
filtered out several messages: we removed messages with URLs, messages with
less than seven tokens, and those produced by applications that use a template
to write the tweet (e.g., Foursquare). We also removed retweeted messages.
These decisions emphasize that messages contain useful textual information and
do not reference external data. The rationale of retweets and external data
removal is that they may be of a different user, and we cannot be sure that
the linked resource is located in the same region as the original message.
These filters reduced our corpora in half (close to 400 million messages).

Regarding the minimum size of seven tokens, the idea is to preserve some
context for each token in the message; please recall that word embeddings
learn the distributional semantics of each word using its surrounding words.
At the same time, we are unaware of a proper study about the phrase’s minimum
length required to learn word embeddings. However, based on the FastText
implementation that uses sliding windows of size five by default as context,
preserving messages with at least seven tokens is a tradeoff between
maintaining a large dataset and filtering out very short messages.

As commented, we created 26 word-embedding models, one per country, and
learned 300 dimension vectors, which is almost a standard for pre-trained
embeddings. We used the default values for the rest of the hyper-parameters of
FastText. In addition, for comparing purposes, we use the entire corpora as a
single corpus to create global word embeddings; the latter is the strategy of
most pre-trained word embeddings. This embedding is used to show that regional
word embeddings perform differently for regionalized tasks.888These 27
embeddings are available in https://ingeotec.github.io/regional-spanish-
models/

Figure 8: Number of common tokens shared by different countries or regions.

###  4.1 Word-embedding similarity

Our semantic analysis requires an affinity matrix, as the lexical one given in
the previous section. Therefore, we need a representation and a similarity
measure to compare word embeddings. Please note that regular word embeddings
produced with neural networks will generate vectors that cannot be mixed.
Please recall that in the first stage of learning each neural network, its
parameters are randomly initialized. An optimizing algorithm is then used to
minimize a loss function on the dataset, adjusting parameters and iterating
until some objective is achieved. These two procedures, random initialization
and optimizing for different datasets, make that two neural network models
produce no proximal vectors for the same word, i.e., under the cosine
distance. Despite vectors having identical numerical structures, e.g., 300
dimensions, and components showing similar distributions, we cannot evaluate
distances between points predicted in different models.

We propose using an intermediate representation and a distance function that
captures similarities between these embeddings to measure the similarity
between different countries. The core idea is to represent each embedding with
a flattened version of the $k$ nearest neighbor graph under a reduced set of
tokens, i.e., tokens appearing in most word embeddings. Therefore, the
similarity becomes linked to the neighborhood of each word (semantically
similar words). The procedure to create this representation is the following:

  * •

Select a common set of tokens; each token appears in at least five countries.
This filtering reduces the vocabulary from more than a million tokens to
nearly 200 thousand tokens ($vocsize$). This selection corresponds to an
inflection point in the tokens curve, (see Fig. 8). The core idea is to reduce
the final representation dimensionality and increase the similarity between
related words.

  * •

Our representation requires constructing a $k$ nearest neighbor graph for each
country. We use dense vectors of the word embeddings closed to the common
tokens set, and we select $k=33$ after probing several choices. This $k$ value
captures several similar terms and remains specific enough to let out
different tokens. We used the cosine distance on the dense vectors.

  * •

Finally, each country is represented as a high-dimensional vector that uses
$k$ entries per vocabulary word, one per neighbor in the common token set.
Each token is then represented by its $k$ nearest neighbors and weighted
inversely to its distance.999We use the weighting form
$0.5+\frac{1}{1+d(u,v)}$ for embedding vector $u$ and its neighbor’s vector
$v$, the lower the distance, the higher the weight. Note that each word
embedding is represented with a very sparse high-dimensional vector, i.e.,
$vocsize^{2}$ possible entries, and more than 3.8 million non-zero components.

The set of Spanish embeddings is compared with the cosine distance on the
sparse vectors. We computed the affinity matrix shown in Figure 9(a) using the
procedure described above. As in the previous affinity matrix, darker colors
represent a high similarity between the regionalized embeddings and the
contrary with lighter colors.

(a) Affinity matrix of our semantic representations.

(b) Two dimensional UMAP projection of semantic representations.

Figure 9: Semantic similarities of our Spanish regional word embeddings.
Countries are specified in their two letter ISO code. On the left, an affinity
matrix where darker cells indicate higher similarities (small distances). On
the right a two dimensional UMAP projection, near points indicate similarity.
Figure 10: Geographic visualization of regional embeddings. The 3D UMAP
projection is encoded as RGB.

Figure 9(b) shows a two-dimensional UMAP projection of our semantic
representation affinity matrix. The projection uses a $3$ nearest neighbors
graph as input; please recall that few neighbors capture local structures. The
colors are computed by 3D reduction by applying the UMAP dimensional reduction
to the same input; the resulting components create an RGB color set using a
simple translation and scale procedure to compose values between $0$ and $1$.
Both distances and colors describe a few well-defined groups. Please note that
our ALL model is quite different from most points and similar to those regions
with extensive collections. This effect is available in word embedding models
constructed on non-regional corpora; they learn semantic traits of most
represented regions. On the contrary, it is necessary to mention that
countries with few messages (e.g., CU, PR, and BO) could need more data to
support the learning procedure; nonetheless, we decided to maintain them in
the projection to learn about their similarity, yet under this advisor. On the
other hand, we remove GQ since their small vocabulary produces numerical
errors while computing its corresponding $k$ nearest neighbor graph and UMAP
projection.

Figure 10 shows the colormap computed from the previous dimensional reduction
to colorize a world map obtaining a kind of map of semantic similarity of the
Spanish language under our construction characteristics. We can observe how
green colors group non-Spanish speaking countries (CA, FR, UK, and BR), except
for PR, CU, and BO, in Fig. 9(b). Note that they correspond to our corpus with
fewer messages and that Fig. 3 also indicates a high number of foreign
messages. It is necessary to take these results with reservation since can be
issues related to their sizes, as explained in Section 3 for the same set of
collections.

Another large cluster is found with countries of all of Latin America (bottom
of Fig. 9(b)). Here we see at least two subclusters with meaningful
geographical meaning HN, NI, SV, and GT. The other groups include DO, CR, and
PA. Note that EC and PY are also included here. Finally, we found a cluster
containing countries around the world. Note that the ALL word embedding is
also placed in this cluster. This cluster seems to be composed of countries
with larger collections and other countries that are related to them. We found
the AR, ES, CL, CO, UY, PE, US, VE, and MX here. Interestingly, we see the US
here and not in the green cluster that agglutinates countries not having the
Spanish language as an official or de facto language.

The semantic similarities between word embeddings can be of interest and can
be the object of further research, but their practical usage is also of
interest. For instance, it is possible to know what countries can be mixed or
interchanged without affecting the regional semantics significantly. A proper
topic analysis could help clarify some of these clusters, but it is beyond the
scope of this manuscript.

###  4.2 A regional task example: predicting emojis with Emoji-15

Regional information can be used to improve understanding of formal and
informal messages, using typical terms and expressions in some regions but not
necessarily used in others. Our regional models can help improve some NLP
tasks having these characteristics. We introduce the Emoji-15 classification
task, a simple multiclass classification problem that predicts the emoji for
given messages among 15 possible ones. This task involves identifying emotions
and sentiments without a particular topic.

Its creation methodology is simple. We selected 15 popular emojis (see Section
3); we do not select the top 15 emojis per region, but a subset that gives
some diversity in emotions. Also, we explicitly avoided the most popular emoji
and skin tones from our selection. The selected emojis are listed in the first
column of Table 2. We selected the datasets for training and test sets from
2020’s January and February; therefore, the corpus resources, training, and
test sets are disjoint. We ensured that tweets contain at most one of these
emojis (even when they can have other emojis). Messages were also selected to
be geotagged to one of our objective Spanish-speaking countries. We followed
the same filtering procedure and preprocessing as made for the word embedding;
note that we also masked emoji’s occurrences. That emoji was used as a label
for the classification task.

We obtained a number of examples that were divided into a 50-50 holdout
(proportion of label messages remain similar in train and test set); see Table
2 for more details. We removed four countries (BO, CU, GQ, and PR) from this
task due to the low number of retrieved messages. For instance, we kept the
statistics of Cuba in the table to show the lower limit cutting. The idea is
to solve all-region benchmarks with all-region models and quantify their
performance and the pertinence of local models on local tasks.

Table 2: Train distribution of the emoji-15 datasets. Since it is a 50-50
hold-out partition, the test set follows a similar distribution. We removed
countries with a low number of examples.

The train partition was used to create one model per country and one for the
entire set of messages (called ALL). Table 3 shows the accuracy performance
scores of all models vs. all test databases. We can observe that some regions
are more challenging to predict than others, e.g., CA achieves a maximum score
of 0.35 while AR achieves 0.49. One can observe that most accuracy scores are
low but far from a uniform distribution (15 classes). The table shows the
local model’s position in each country benchmark (local rank column). The
best-performing model for a benchmark will rank as 1, the second-best as 2,
and similarly for the rest. Note that small local rank values indicate that
the local model (for that region) is efficient for its corresponding
benchmark. The best five models for each country benchmark are also listed; we
can observe how many geographically near regions perform well in their
geographic neighborhoods. The average rank of the local model is 8.09 while
the median is 6.5; these values support the idea that local models are useful
on tasks where regional information can be used. Also, one can observe that
not always more data (ALL model) is the best, in this case, it could be said
that the geographical aspect is more relevant.

country | min | max | local | top-5
---|---|---|---|---
code | acc. | acc. | rank | 1 | 2 | 3 | 4 | 5
AR | 0.478 | 0.490 | 3 | UY | PY | AR | PE | CO
BR | 0.461 | 0.488 | 1 | BR | ALL | DO | PY | CR
CA | 0.293 | 0.353 | 18 | CL | ALL | CO | MX | US
CL | 0.426 | 0.449 | 1 | CL | US | MX | AR | ES
CO | 0.425 | 0.437 | 2 | US | CO | VE | EC | GT
CR | 0.369 | 0.388 | 9 | US | VE | ALL | MX | CO
DO | 0.338 | 0.381 | 13 | US | CO | VE | CL | ALL
EC | 0.380 | 0.414 | 9 | MX | US | CL | ALL | CO
ES | 0.475 | 0.486 | 1 | ES | AR | MX | US | VE
FR | 0.419 | 0.442 | 4 | ALL | GT | EC | FR | PA
GB | 0.347 | 0.376 | 23 | ALL | AR | ES | VE | MX
GT | 0.349 | 0.388 | 13 | MX | US | ALL | CO | ES
HN | 0.335 | 0.367 | 18 | PE | EC | BR | CR | UY
MX | 0.423 | 0.434 | 1 | MX | GT | CR | US | CO
NI | 0.337 | 0.372 | 18 | VE | CO | CL | MX | US
PA | 0.366 | 0.393 | 10 | US | CL | VE | CO | PE
PE | 0.380 | 0.420 | 9 | MX | ALL | US | AR | CO
PY | 0.424 | 0.442 | 1 | PY | US | BR | PE | UY
SV | 0.323 | 0.395 | 18 | US | CO | MX | CL | VE
US | 0.404 | 0.424 | 1 | US | MX | CO | ES | CL
UY | 0.435 | 0.457 | 1 | UY | US | CO | CL | VE
VE | 0.385 | 0.434 | 4 | MX | CO | ES | VE | US
Table 3: Performance statistics of all benchmarks (countries). Top-5 models are also listed and the rank position of the local model on solving the current benchmark. model | voc. | avg.
---|---|---
| size | rank
US | 292,465 | 4.23
CO | 324,635 | 6.05
MX | 438,136 | 6.27
CL | 282,737 | 6.91
VE | 271,924 | 7.00
ALL | 1,696,232 | 8.45
PE | 178,113 | 8.64
UY | 200,032 | 8.73
EC | 147,560 | 8.95
AR | 673,424 | 9.41
ES | 571,196 | 10.95
PY | 124,162 | 11.14
BR | 127,205 | 11.27
CR | 103,086 | 12.50
PA | 111,635 | 13.36
GT | 95,252 | 13.64
DO | 108,655 | 14.91
GB | 82,418 | 18.00
NI | 68,605 | 18.18
FR | 69,843 | 18.91
CA | 63,161 | 19.00
SV | 73,833 | 19.14
HN | 60,580 | 20.36
Table 4: Average rank of all regional models along all countries datasets.
Models with low average ranks are better.

The average rank of a single model along all benchmarks indicates how well
this model generalizes. Table 4 shows the performance of all models, along
with all benchmarks, as its average rank. We can observe that some country
models are outstanding, like the US model. In this sense, it is remarkable
that models like the US or CO (both using a vocabulary of $300$k tokens)
perform better than huge ones. On the other hand, the ALL model is
competitive; however, it is not the best (global 6th regarding average rank).
Please recall that we created the ALL model by merging the entire corpora into
a single corpus, which is the typical construction; for instance, the ALL
model contains close to $1.7$ million tokens in its vocabulary.101010Note that
our vocabulary in Section 3 has more than $1.2$ million tokens, and here we
mentioned a larger one; this is the vocabulary recognized by the fastText
parser. However, both vocabularies were computed using the same corpus. It is
similar for other word embeddings listed in Table 4.

While these results apply to the regional task of predicting the most popular
emojis, the evidence points out that local models are competitive options for
solving tasks requiring local traits as emoji predictions (see Table 3). Even
more, some regional models perform better than large ones, as shown in Table
4, which is remarkable.

##  5 Language Models

Language Models (LM) are more sophisticated than word embedding models since
they go beyond word semantics to context semantics and text generation. In
more detail, Word2Vec, FastText, and Glove generate fixed embeddings for each
word independently if the word can take different meanings depending on the
context. For example, the word orange can be a fruit or a color, depending on
the context. Language Modeling is the task of predicting the next word given
some context so they perform well in distinguishing homonyms.

In that sense, BERT Devlin et al. (2019) is an LM that has gained considerable
attention lately. It is a model that uses a series of encoders to generate
embeddings for each word depending on its context. BERT differs from
alternatives like ELMo Peters et al.. (2018) because the same pre-trained
model can be fine-tuned for different tasks. The pre-train on BERT uses the
Masked Language Model (MLM) task where each input sentence contains a _mask
token_ on 15% random words. Then, BERT was trained on a second task, the Next
Sentence Prediction (NSP) task, where the input has two sentences, with a
_separation token_ in between, and the task was to predict if the second
sentence followed the first.

Our resources include regional pre-trained BERT-like models using the MLM task
over tweets for the countries AR, CL, CO, MX, ES, UY, VE, and the US, i.e.,
larger ones. First, we applied the same preprocessing as detailed in Section 3
to our corpora. The pre-training was the same as the original BERT, where 15%
of the tokens on each sentence were marked with a [MASK] token, and the model
must predict them. We used the corresponding regional tweets from 2016 to 2019
to pre-train each model. All the models had a series of two encoders with four
attention heads each and output 512-dimensional embedding vectors. This
configuration corresponds with the small-size model following the official
BERT implementation setup. We chose this setup based on the computational
resources we had available. We name our model BILMA, for Bert In Latin
America. We used a learning rate of $10^{-5}$ with the Adam optimizer; the
models for CL, UY, VE, and the US were trained for three epochs and AR, CO,
MX, and ES for just one because of the size of their corpus. All the pre-
trained models are available for
download111111https://ingeotec.github.io/regional-spanish-models/.

Figure 11 shows the loss and accuracy of the MLM task during the training. We
can see that the BILMA model for AR was trained on double the number of
batches; that was because Argentina has double the corpus size. The rest of
the models were trained on a similar number of batches.

(a) Loss function values on training

(b) Accuracy function values on training

Figure 11: Loss and accuracy during training on the Masked Language Model
task. The batch size is of 128 tweets.

In Figure 12(a), we compare the models predicting the masked words on all the
regions over the test set of tweets used in Section 4.2. The test was to
predict the [MASK] tokens correctly. Some interesting points to highlight are
the following. First, the Argentina model got very high scores on all the
regions, even above their corresponding models for UY and CO. This might be
because this model was trained for like double the data. Second, some models
got better results in the ES region than theirs, like CL, CO, UY, VE, and the
US. The US region got the worst outcomes for AR, CL, CO, ES, and UY models.
Finally, CO and UY were the models with lower accuracy.

(a) Accuracy on the MLM task.

(b) Accuracy predicting the emoticon.

Figure 12: Comparison of the accuracy of the trained models on all the regions
on MLM and emoticon prediction tasks.

###  5.1 BILMA’s performance on the Emoji-15 regional task

We applied our BILMA models to our Emoji-15 task (see Section 4.2). For this
matter, we fine-tuned the pre-trained language models to predict the emoticon
by adding two linear layers to the first token of each sentence (the start-of-
sentence token), so the output of the fine-tuned models was a probability
distribution of the assigned emoticon, independent of its position. We split
the tweets into 90% train and 10% validation and trained until the accuracy
stalled. After that, we evaluated the test set; the results are presented in
Figure 12(b). We can conclude that all the models got better results in their
corresponding regions from the results. The AR, MX, and ES models got good
results over all the regions; meanwhile, UY and VE got low scores. The
prediction scores are similar to those found in Table 3; however, our regional
FastText models are slightly better than our fine-tuned BILMA models.
Nonetheless, BILMA models learn how people write in different regions, as it
is exemplified in the rest of this section.

###  5.2 Generating text with BILMA regional language models.

As a qualitative and exemplification exercise, we present how each region
model predicts the masked word for the same example phrase. In Table 5 we show
the predictions for the masked token on a set of selected sentences. The color
intensity indicates the confidence of the model to predict the word. The first
two examples are el/la [MASK] subio de precio (the [MASK] raised in
price),121212the article el indicates the masked word should be singular and
masculine, and for la it should be singular feminine here we can see
differences in how each region name their public transportation, in AR they
use bondi, colectivo, in CL metro, micro, bus, in MX uber and ES metro, bus;
we can also see the differences in how they called the cellphone service, in
CO and MX is celular and in ES is movil. The third example is me gusta tomar
[MASK] en la mañana (I like to drink [MASK] in the morning)131313tomar could
mean to take or to drink, here we can note that in AR and UY people prefer to
drink mates meanwhile in MX, ES, VE, and the US drink coffee. The fourth
phrase is vamos a comer [MASK] (let us eat [MASK]) where we can see the
differences in the cuisine of the countries with dishes like asado, pizza,
ñoquis, empanadas, milanesas, sushi, tacos, oreja, hamburguesa, arepa, torta.
The last sentence is estoy en la ciudad de [MASK] (I am in [MASK] city). The
results include a list of some of the larger cities in each region. This
exercise is a proof of concept to show that the models of different regions
can predict very different words, i.e., regional information. Note that the
diversity of the predictions include dialect differences (celular vs movil)
but also topical (tacos vs asado) and will depend on the input sentence.

el [MASK] subio de precio
---
AR | CL | CO | MX | ES | UY | VE | US
dolar | chofer | que | cel | que | video | internet | que
bondi | metro | tiempo | video | movil | que | video | video
que | 0 | 0 | uber | tiempo | profe | 0 | juego
0 | que | agua | numero | bus | 0 | precio | q
auto | bus | se | celular | dia | horoscopo | telefono | 0
video | internet | ano | tuit | metro | tema | que | pueblo
autoestima | papa | dia | que | coche | nombre | dolar | arbitro
colectivo | mio | man | telefono | agua | pibe | queso | tipo
tiempo | precio | mundo | dinero | cafe | grupo | pan | no
tren | profe | celular | tiempo | pelo | amor | tlf | mundo
la [MASK] subio de precio
AR | CL | CO | MX | ES | UY | VE | US
foto | wea | gente | foto | gente | foto | gente | gente
que | gente | vida | vida | que | vida | plata | que
lluvia | micro | noche | gasolina | foto | gente | semana | foto
luna | foto | que | cancion | vida | historia | caja | prensa
musica | mama | ley | gente | noche | profe | foto | policia
caja | luna | historia | noche | camara | abuela | carne | tipa
gente | mina | semana | ropa | cara | cancion | vida | camara
camara | senora | lluvia | app | bateria | que | cola | ley
coca | profe | luna | pagina | musica | vieja | arepa | cara
heladera | vieja | musica | morra | semana | madre | pasta | 0
me gusta tomar [MASK] en la manana
AR | CL | CO | MX | ES | UY | VE | US
mates | desayuno | fotos | cafe | cafe | mates | cafe | cafe
teres | once | cafe | fotos | algo | mate | fotos | decisiones
cafe | cafe | cerveza | decisiones | nota | algo | ron | ropa
helado | agua | agua | agua | decisiones | agua | clases | clases
sol | clases | peliculas | alcohol | cerveza | cafe | decisiones | manana
fernet | sol | decisiones | cerveza | cola | sol | hoy | fotos
mate | almuerzo | hoy | clases | sol | vino | anis | sol
birra | cerveza | ropa | hoy | todo | alcohol | cartas | agua
algo | hoy | frio | tequila | uno | helado | agua | hoy
birras | helado | musica | comida | alcohol | cerveza | 0 | comida
vamos a comer [MASK]
AR | CL | CO | MX | ES | UY | VE | US
asado | sushi | mierda | tacos | mierda | algo | pizza | _
pizza | todo | jaja | mucho | ya | pizza | mierda | hoy
algo | hoy | ! | jaja | mas | hoy | manana | conmigo
helado | mierda | rico | pizza | esto | jaja | rico | pizza
noquis | algo | . | 0 | mucho | helado | hamburguesa | manana
pizzas | pizza | _ | manana | jaja | oreja | hoy | todo
hoy | ! | helado | hoy | hoy | todo | arepa | mierda
empanadas | jaja | pizza | taquitos | tio | nada | . | bien
facturas | ctm | hoy | emo | usr | uno | todo | playa
milanesas | emo | asi | algo | bien | eso | torta | tacos
estoy en la ciudad de [MASK]
AR | CL | CO | MX | ES | UY | VE | US
mierda | santiago | bogota | mexico | madrid | mierda | venezuela | _
argentina | vina | colombia | monterrey | espana | casa | caracas | mexico
cordoba | conce | cali | guadalajara | verdad | uruguay | valencia | dios
rosario | chile | medellin | puebla | sevilla | hoy | merida | miami
mierdaa | valparaiso | barranquilla | cancun | barcelona | historia | barquisimeto | hoy
tucuman | stgo | dios | veracruz | mierda | nuevo | maracaibo | casa
hoy | concepcion | hoy | mty | hoy | filosofia | maracay | vacaciones
capital | valpo | antioquia | merida | valencia | ingles | margarita | pr
aca | maipu | cartagena | cdmx | futbol | montevideo | aragua | disney
sol | chillan | paz | todos | examenes | clase | carabobo | mi

Table 5: Predictions of the masked words over different regions. The color
intensity indicates the probability of prediction.

##  6 Conclusions

This manuscript proposes a set of regionalized resources for the Spanish
language using Twitter as the data source. We collected messages from
Twitter’s public streaming API from 2016 to 2019; messages must be tagged as
being written in Spanish and geotagged to one of the 26 countries that use
Spanish as one of their primary languages. The vocabulary of each corpus was
extracted, characterized, and compared their similarity, defining a distance
metric between them. We also produce visualizations and insightful information
about lexical and semantical similarities of the Spanish language variations
in Twitter messages.

On the other hand, we created regional semantic models using FastText and
produced some visualizations of the semantic similarities among regions. We
also create regional language models called BILMA, based on the well-known
BERT transformer architecture. We give empirical evidence of the usefulness of
regional models in regionalized text classification tasks (Emoji-15 task) and
how this more careful data segmentation can yield better performances than the
typical more-data-is-better approach.

We provide access to our vocabularies, word embeddings, language models, and
corpora sample through the project site (available in
https://ingeotec.github.io/regional-spanish-models/). The necessary packages
(BILMA) and all scripts used to generate our resources, open-sourced under the
MIT license, are also reachable under the same site.

###  6.1 Limitations and further research

While the regional models seem promising tools for many tasks that require
understanding regionalisms and idiosyncrasies, the use of multiple models can
be cumbersome for real-world systems, not to mention the necessary computing
units needed to handle many models. It is necessary to create models that can
shift their region depending on a regional context. This approach requires
further research.

Our Spanish corpora could be more balanced concerning countries. Some
countries have too many elements, while others like GQ, CU, PR, and BO need to
be bigger to have reliable semantic models (i.e., word embeddings and language
models). More research and data collection are needed to improve resources in
these regions.

Our region similarity comparisons are based on lexical and semantic properties
of vocabularies computed and learned from Twitter messages. While it is not
our goal, the presented projections could compare topics and other internal
knowledge in the resources. Proper topic analysis is beyond the scope of this
study and requires further research. Similarly, it is possible to mine our
resources, i.e., language models, semantic, lexical features, corpus, etc., to
perform data-driven social sciences research, i.e., research about gender
inequality or race perception. All these topics require more research.

The methodology and implementation of this manuscript are open to
improvements. For instance, countries with relatively few examples require
different strategies to be competitive. It is also essential to find ways to
collect better data and discard the bad ones, such that results become more
reliable for small collections. In the same sense, comparing vocabularies and
embeddings created from datasets with such disparate sizes require robust
normalization methods that we barely sketched. Our long-term goal is to update
our resources using more and more data and novel language models as they
appear in the literature.

Acknowledgments This work has been done through CONACYT (National Council of
Science and Technology from Mexico) support with the Ciencia Básica grant with
project ID A1-S-34811. Also, the authors acknowledge the support from CIMAT
and “Laboratorio de Supercómputo del Bajio” through project 300832 from
CONACyT. We also thank the reviewers’ comments and suggestions that help us
improve the manuscript’s quality. The first author thanks CONACyT and CICESE
(México) since this work was partially developed when the first author was in
a fellow year visiting the latter institution.

##  Appendix A BILMA language model usage

In order to use our BILMA models, we need to download one first, we will also
need the vocabulary file.

To clone the repository, download the model and install dependencies, in a
linux terminal just type the following commands:

⬇

git clone https://github.com/msubrayada/bilma

cd bilma

bash download-emoji15-bilma.sh

python3 -m pip install tensorflow==2.4

and now we have the python package and its dependencies, the model and the
vocabulary file (shared to all BILMA models). In particular, this example
downloads the MX model that was trained with one epoch on the MLM task and
fine-tuned on the Emoji-15 task for 13 epochs.

We need to run a Python 3 console and load the BILMA model.

⬇

from bilma import bilma_model

vocab_file = "vocab_file_All.txt"

model_file = "bilma_small_MX_epoch-1_classification_epochs-13.h5"

model = bilma_model.load(model_file)

tokenizer = bilma_model.tokenizer(vocab_file=vocab_file, max_length=280)

this BILMA model has two outputs, the first with shape (bs, 280, 29025) where
bs is the batch size, 280 is the max length and 29025 is the size of the
vocabulary. This output is used to predict the masked words. The second output
has shape (bs, 15) which corresponds to the predicted emoji.

The next step is tokenizing some messages as follows:

⬇

texts = [

"Tenemos tres dias sin internet ni senal de celular en el pueblo.",

"Incomunicados en el siglo XXI tampoco hay servicio de telefonia fija",

"Vamos a comer unos tacos",

"Los del banco no dejan de llamarme"

]

toks = tokenizer.tokenize(texts)

the prediction is made as follows:

⬇

p = model.predict(toks)

finally, the predicted emojis can be displayed with:

⬇

tokenizer.decode_emo(p[1])

this produces the output: , each emoji corresponds to the most probable one
for each message in texts.

##  Appendix B Cut off $N$

This section presents a methodology that addresses the minimum token frequency
to be kept in the analysis.

The idea is to compute the confidence interval of a Bernoulli variable and
select the minimum frequency $f$ (number of times the token appears in the
corpus) that sets the interval in a feasible region. Let $p$ be the
probability of seeing a particular token, assuming $\hat{p}$ is Gaussian
distributed. The confidence interval is
$\hat{p}\pm\alpha\textbf{se}(\hat{p}),$ where se is the standard error of
$\hat{p}$, and $\alpha$ is the percent point function with parameter
$1-\frac{c}{2},$ where $1-c$ represents the confidence, e.g., $\alpha\approx
2$ gives approximately a 95% confidence interval.

The following equations show that under the assumption made, the frequency $f$
(number of times the token appears in the corpus) must be greater or equal to
$\frac{N\alpha^{2}}{N+\alpha^{2}}$ that in the limit when $N$ tends to
infinity corresponds to $\alpha^{2}$.

$\displaystyle\hat{p}-\alpha\textbf{se}(\hat{p})$ $\displaystyle\geq$
$\displaystyle 0$ (1)
$\displaystyle\hat{p}-\alpha\frac{\sqrt{\hat{p}(1-\hat{p})}}{\sqrt{N}}$
$\displaystyle\geq$ $\displaystyle 0$ (2)
$\displaystyle\sqrt{N}\hat{p}-\alpha\sqrt{\hat{p}(1-\hat{p})}$
$\displaystyle\geq$ $\displaystyle 0$ (3) $\displaystyle\sqrt{N}\hat{p}$
$\displaystyle\geq$ $\displaystyle\alpha\sqrt{\hat{p}(1-\hat{p})}$ (4)
$\displaystyle\sqrt{N}\hat{p}$ $\displaystyle\geq$
$\displaystyle\sqrt{\alpha^{2}\hat{p}(1-\hat{p})}$ (5) $\displaystyle
N\hat{p}^{2}$ $\displaystyle\geq$ $\displaystyle\alpha^{2}\hat{p}(1-\hat{p})$
(6) $\displaystyle N\hat{p}^{2}-\alpha^{2}\hat{p}(1-\hat{p})$
$\displaystyle\geq$ $\displaystyle 0$ (7)
$\displaystyle\hat{p}(N\hat{p}-\alpha^{2}(1-\hat{p}))$ $\displaystyle\geq$
$\displaystyle 0$ (8) $\displaystyle N\hat{p}-\alpha^{2}(1-\hat{p})$
$\displaystyle\geq$ $\displaystyle 0$ (9) $\displaystyle
N\hat{p}-\alpha^{2}+\alpha^{2}\hat{p}$ $\displaystyle\geq$ $\displaystyle 0$
(10) $\displaystyle\hat{p}(N+\alpha^{2})$ $\displaystyle\geq$
$\displaystyle\alpha^{2}$ (11) $\displaystyle\hat{p}$ $\displaystyle\geq$
$\displaystyle\frac{\alpha^{2}}{N+\alpha^{2}}$ (12) $\displaystyle f$
$\displaystyle\geq$ $\displaystyle\frac{N\alpha^{2}}{N+\alpha^{2}}$ (13)

## References

  * (1) McInnes, L., Healy, J. & Melville, J. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. (2020)
  * (2) Gelbukh, A. & Sidorov, G. Zipf and Heaps Laws’ Coefficients Depend on Language. Computational Linguistics And Intelligent Text Processing. pp. 332-335 (2001)
  * (3) Schütze, H., Manning, C. & Raghavan, P. Introduction to information retrieval. Cambridge University Press Cambridge. (2008).
  * (4) Jimenez, S., Dueñas, G., Gelbukh, A., Rodriguez-Diaz, C. & Mancera, S. Automatic detection of regional words for pan-hispanic spanish on twitter. Ibero-American Conference On Artificial Intelligence. pp. 404-416 (2018)
  * (5) Li, M., Chng, E., Chong, A. & See, S. An empirical analysis of emoji usage on Twitter. Industrial Management & Data Systems. (2019)
  * (6) Huertas-Tato, J., Martin, A. & Camacho, D. BERTuit: Understanding Spanish language in Twitter through a native transformer. ArXiv Preprint ArXiv:2204.03465. (2022)
  * (7) Hong, L., Convertino, G. & Chi, E. Language matters in twitter: A large scale study. Proceedings Of The International AAAI Conference On Web And Social Media. 5, 518-521 (2011)
  * (8) Park, J., Barash, V., Fink, C. & Cha, M. Emoticon style: Interpreting differences in emoticons across cultures. Proceedings Of The International AAAI Conference On Web And Social Media. 7, 466-475 (2013)
  * (9) Alshutayri, A. & Atwell, E. Exploring Twitter as a source of an Arabic dialect corpus. International Journal Of Computational Linguistics (IJCL). 8, 37-44 (2017)
  * (10) Lamontagne, J. & McCulloch, G. Phonological variation on Twitter: Evidence from letter repetition in three French dialects. Journal Of French Language Studies. pp. 1-32 (2022)
  * Hovy et al. (2020) Hovy, Dirk and Rahimi, Afshin and Baldwin, Timothy and Brooke, Julian. 2020.  Visualizing regional language variation across Europe on Twitter  _Handbook of the Changing World Language Map_ , 3719–3742
  * Hoff (2020) Hoff, Mark. 2020.  Cerca mío/a or cerca de mí? A variationist analysis of Spanish locative+ possessive on Twitter.  _Studies in Hispanic and Lusophone Linguistics_ , 51–78.
  * Crampton et al. (2013) Crampton, Jeremy W., Mark Graham, Ate Poorthuis, Taylor Shelton, Monica Stephens, Matthew W. Wilson, and Matthew Zook. 2013.  Beyond the geotag: situating "big data" and leveraging the potential of the geoweb.  _Cartography and Geographic Information Science_ , 40(2):130–139.
  * Devlin et al. (2019) Devlin, Jacob, Chang, Ming-Wei, Lee, Kenton and Toutanova, Kristina. 2019  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding  _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Association for Computational Linguistics
  * Donoso and Sánchez (2017) Donoso, Gonzalo and David Sánchez. 2017.  Dialectometric analysis of language variation in Twitter.  In _Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)_ , pages 16–25, Association for Computational Linguistics, Valencia, Spain.
  * Dresner and Herring (2010) Dresner, Eli and Susan C Herring. 2010.  Functions of the nonverbal in cmc: Emoticons and illocutionary force.  _Communication theory_ , 20(3):249–268.
  * Gonçalves and Sánchez (2014) Gonçalves, Bruno and David Sánchez. 2014.  Crowdsourcing dialect characterization through twitter.  _PloS one_ , 9(11):e112074.
  * Graham, Hale, and Gaffney (2014) Graham, Mark, Scott A Hale, and Devin Gaffney. 2014.  Where in the world are you? geolocation and language identification in twitter.  _The Professional Geographer_ , 66(4):568–578.
  * Huang et al. (2016) Huang, Yuan, Diansheng Guo, Alice Kasakoff, and Jack Grieve. 2016.  Understanding us regional linguistic variation with twitter data analysis.  _Computers, environment and urban systems_ , 59:244–255.
  * Joulin et al. (2017) Joulin, Armand, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017.  Bag of tricks for efficient text classification.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 427–431, Association for Computational Linguistics.
  * Kejriwal et al. (2021) Kejriwal, Mayank, Qile Wang, Hongyu Li, and Lu Wang. 2021.  An empirical study of emoji usage on twitter in linguistic and national contexts.  _Online Social Networks and Media_ , 24:100149.
  * Mikolov et al. (2013a) Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Distributed representations of words and phrases and their compositionality.  In _Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’13, pages 3111–3119, Curran Associates Inc., USA.
  * Mocanu et al. (2013) Mocanu, Delia, Andrea Baronchelli, Nicola Perra, Bruno Gonçalves, Qian Zhang, and Alessandro Vespignani. 2013.  The twitter of babel: Mapping world languages through microblogging platforms.  _PloS one_ , 8(4):e61981.
  * (24) Wada, T. & Iwata, T. Unsupervised Cross-lingual Word Embedding by Multilingual Neural Language Models. CoRR. abs/1809.02306 (2018)
  * (25) Anowar, F., Sadaoui, S. & Selim, B. Conceptual and empirical comparison of dimensionality reduction algorithms (PCA, KPCA, LDA, MDS, SVD, LLE, ISOMAP, LE, ICA, t-SNE). Computer Science Review. 40 pp. 100378 (2021)
  * (26) Mikolov, T., Sutskever, I., Chen, K., Corrado, G. & Dean, J. Distributed representations of words and phrases and their compositionality. Advances In Neural Information Processing Systems. 26 (2013)
  * (27) Cotton, E. & Sharp, J. Spanish in the Americas. (Georgetown University Press,1988)
  * (28) Bojanowski, P., Grave, E., Joulin, A. & Mikolov, T. Enriching word vectors with subword information. Transactions Of The Association For Computational Linguistics. 5 pp. 135-146 (2017)
  * (29) Penny, R., Penny, R. & Ralph, P. Variation and change in Spanish. (Cambridge University Press,2000)
  * Pennington, Socher, and Manning (2014) Pennington, Jeffrey, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Peters et al.. (2018) Peters, Matthew E. and Neumann, Mark and Iyyer, Mohit and Gardner, Matt and Clark, Christopher and Lee, Kenton and Zettlemoyer, Luke. 2018  Deep Contextualized Word Representations  _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, Association for Computational Linguistics
  * Rodriguez-Diaz et al. (2018) Rodriguez-Diaz, Carlos A, Sergio Jimenez, George Dueñas, Johnatan Estiven Bonilla, and Alexander Gelbukh. 2018.  Dialectones: Finding statistically significant dialectal boundaries using twitter data.  _Computación y Sistemas_ , 22(4):1213–1222.
  * Yang, Macdonald, and Ounis (2018) Yang, Xiao, Craig Macdonald, and Iadh Ounis. 2018.  Using word embeddings in twitter election classification.  _Information Retrieval Journal_ , 21(2):183–207.
  * (34) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018.  Learning Word Vectors for 157 Languages.  _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)_
  * (35) Frenda, S., Ghanem, B., Gómez, M. & Rosso, P. Online hate speech against women: Automatic identification of misogyny and sexism on twitter. Journal Of Intelligent & Fuzzy Systems. 36, 4743-4752 (2019)
  * (36) Vashisth, P. & Meehan, K. Gender classification using twitter text data. 2020 31st Irish Signals And Systems Conference (ISSC). pp. 1-6 (2020)
  * (37) Mooney, P., Winstanley, A. & Corcoran, P. Evaluating Twitter for use in environmental awareness campaigns. (Dept. of Computer Science, National University of Ireland, Maynooth,2009)
  * (38) Suhasini, M. & Srinivasu, B. Emotion detection framework for twitter data using supervised classifiers. Data Engineering And Communication Technology. pp. 565-576 (2020)
  * (39) Paul, M. & Dredze, M. You are what you tweet: Analyzing twitter for public health. Proceedings Of The International AAAI Conference On Web And Social Media. 5, 265-272 (2011)
  * (40) Finfgeld-Connett, D. Twitter and health science research. Western Journal Of Nursing Research. 37, 1269-1283 (2015)
  * CKennedy, Brendan et al. (2022) CKennedy, Brendan and Atari, Mohammad and Davani, Aida Mostafazadeh and Yeh, Leigh and Omrani, Ali and Kim, Yehsong and Coombs, Kris and Havaldar, Shreya and Portillo-Wightman, Gwenyth and Gonzalez, Elaine and others (2022).  Introducing the Gab Hate Corpus: defining and applying hate-based rhetoric to social media posts at scale.  _Language Resources and Evaluation_ , Springer.
  * Escudero-Mancebo, David and Corrales-Astorgano, Mario and Cardeñoso-Payo, Valentín and Aguilar, Lourdes and González-Ferreras, César and Martínez-Castilla, Pastora and Flores-Lucas, Valle (2022) Escudero-Mancebo, David and Corrales-Astorgano, Mario and Cardeñoso-Payo, Valentín and Aguilar, Lourdes and González-Ferreras, César and Martínez-Castilla, Pastora and Flores-Lucas, Valle. 2022.  Prautocal corpus: A corpus for the study of down syndrome prosodic aspects  Language Resources and Evaluation, Springer.
  * Gruszczyński, Włodzimierz and Adamiec, Dorota and Bronikowska, Renata and Kieraś, Witold and Modrzejewski, Emanuel and Wieczorek, Aleksandra and Woliński, Marcin (2022) Gruszczyński, Włodzimierz and Adamiec, Dorota and Bronikowska, Renata and Kieraś, Witold and Modrzejewski, Emanuel and Wieczorek, Aleksandra and Woliński, Marcin, 2022.  The Electronic Corpus of 17th-and 18th-century Polish Texts  Language Resources and Evaluation, Springer.


* [12] Michele Maggiore.  Gravitational wave experiments and early universe cosmology.  Phys. Rept., 331:283–367, 2000.
  * [13] David J. Weir.  Gravitational waves from a first order electroweak phase transition: a brief review.  Phil. Trans. Roy. Soc. Lond. A, 376(2114):20170126, 2018.
  * [14] Nelson Christensen.  Stochastic Gravitational Wave Backgrounds.  Rept. Prog. Phys., 82(1):016903, 2019.
  * [15] Chiara Caprini et al.  Science with the space-based interferometer eLISA. II: Gravitational waves from cosmological phase transitions.  JCAP, 04:001, 2016.
  * [16] Chiara Caprini and Daniel G. Figueroa.  Cosmological Backgrounds of Gravitational Waves.  Class. Quant. Grav., 35(16):163001, 2018.
  * [17] O. Heckmann and E. Schücking.  Bemerkungen zur Newtonschen Kosmologie. I. Mit 3 Textabbildungen in 8 Einzeldarstellungen.  Zeitschrift für Astrophysik, 38:95, January 1955.
  * [18] A. Raychaudhuri.  Relativistic and Newtonian Cosmology.  Zeitschrift für Astrophysik, 43:161, January 1957.
  * [19] J. Ehlers.  Contributions to the relativistic mechanics of continuous media.  Abh. Akad. Wiss. Lit. Mainz. Nat. Kl., 11:793–837, 1961.
  * [20] S. W. Hawking.  Perturbations of an Expanding Universe.  APJ, 145:544, August 1966.
  * [21] G. F. R. Ellis and M. Bruni.  Covariant and gauge-invariant approach to cosmological density fluctuations.  Phys. Rev. D, 40:1804–1818, Sep 1989.
  * [22] G. F. R. Ellis, J. Hwang, and M. Bruni.  Covariant and gauge-independent perfect-fluid robertson-walker perturbations.  Phys. Rev. D, 40:1819–1826, Sep 1989.
  * [23] Max Tegmark and Matias Zaldarriaga.  Separating the early universe from the late universe: Cosmological parameter estimation beyond the black box.  Phys. Rev. D, 66:103508, 2002.
  * [24] Christoph Schmid, Dominik J. Schwarz, and Peter Widerin.  Peaks above the Harrison-Zel’dovich spectrum due to the quark - gluon to hadron transition.  Phys. Rev. Lett., 78:791–794, 1997.
  * [25] Christoph Schmid, Dominik J. Schwarz, and Peter Widerin.  Amplification of cosmological inhomogeneities from the QCD transition.  Phys. Rev. D, 59:043517, 1999.
  * [26] Felix Giese, Thomas Konstandin, Kai Schmitz, and Jorinde Van De Vis.  Model-independent energy budget for LISA.  JCAP, 01:072, 2021.
  * [27] Marco Bruni, Peter K.S. Dunsby, and George F.R. Ellis.  Cosmological perturbations and the physical meaning of gauge invariant variables.  Astrophys. J., 395:34–53, 1992.
  * [28] Ruth Durrer.  The Cosmic Microwave Background.  Cambridge University Press, Cambridge, 2008.
  * [29] J. M. Stewart and M. Walker.  Perturbations of space-times in general relativity.  Proc. R. Soc. Lond., A 341(49), 1974.
  * [30] Christos G. Tsagas, Anthony Challinor, and Roy Maartens.  Relativistic cosmology and large-scale structure.  Phys. Rept., 465:61–147, 2008.
  * [31] Marco Bruni, Sabino Matarrese, Silvia Mollerach, and Sebastiano Sonego.  Perturbations of space-time: Gauge transformations and gauge invariance at second order and beyond.  Class. Quant. Grav., 14:2585–2606, 1997.
  * [32] Roy Maartens.  Linearization instability of gravity waves?  Phys. Rev. D, 55:463–467, 1997.
  * [33] S. W. Hawking and G. F. R. Ellis.  The Large Scale Structure of Space-Time.  Cambridge Monographs on Mathematical Physics. Cambridge University Press, 1973.
  * [34] George F.R. Ellis and Henk van Elst.  Cosmological models: Cargese lectures 1998.  NATO Sci. Ser. C, 541:1–116, 1999.
  * [35] Roy Maartens, Tim Gebbie, and George F. R. Ellis.  Cosmic microwave background anisotropies: Nonlinear dynamics.  Phys. Rev. D, 59:083506, Mar 1999.
  * [36] Peter K.S. Dunsby, Bruce A.C.C. Bassett, and George F.R. Ellis.  Covariant analysis of gravitational waves in a cosmological context.  Class. Quant. Grav., 14:1215–1222, 1997.
  * [37] Despoina Pazouli and Christos G. Tsagas.  Gravitational-wave implications for structure formation: A second-order approach.  Phys. Rev., D93(6):063520, 2016.
  * [38] James M. Bardeen.  Gauge-invariant cosmological perturbations.  Phys. Rev. D, 22:1882–1905, Oct 1980.
  * [39] Anthony Challinor.  Microwave background anisotropies from gravitational waves: the 1 + 3 covariant approach.  Classical and Quantum Gravity, 17(4):871–889, jan 2000.
  * [40] Sidney R. Coleman and Frank De Luccia.  Gravitational Effects on and of Vacuum Decay.  Phys. Rev., D21:3305, 1980.
  * [41] Arthur Kosowsky and Michael S. Turner.  Gravitational radiation from colliding vacuum bubbles: envelope approximation to many bubble collisions.  Phys. Rev. D, 47:4372–4391, 1993.
  * [42] Chiara Caprini, Ruth Durrer, and Geraldine Servant.  Gravitational wave generation from bubble collisions in first-order phase transitions: An analytic approach.  Phys. Rev. D, 77:124015, 2008.
  * [43] Stephan J. Huber and Thomas Konstandin.  Gravitational Wave Production by Collisions: More Bubbles.  JCAP, 09:022, 2008.
  * [44] Chiara Caprini, Ruth Durrer, and Geraldine Servant.  The stochastic gravitational wave background from turbulence and magnetic fields generated by a first-order phase transition.  JCAP, 12:024, 2009.
  * [45] Leonard Kisslinger and Tina Kahniashvili.  Polarized Gravitational Waves from Cosmological Phase Transitions.  Phys. Rev. D, 92(4):043006, 2015.
  * [46] Mark Hindmarsh, Stephan J. Huber, Kari Rummukainen, and David J. Weir.  Gravitational waves from the sound of a first order phase transition.  Phys. Rev. Lett., 112:041301, 2014.
  * [47] John T. Giblin, Jr. and James B. Mertens.  Vacuum Bubbles in the Presence of a Relativistic Fluid.  JHEP, 12:042, 2013.
  * [48] Mark Hindmarsh, Stephan J. Huber, Kari Rummukainen, and David J. Weir.  Numerical simulations of acoustically generated gravitational waves at a first order phase transition.  Phys. Rev. D, 92(12):123009, 2015.
  * [49] Chiara Caprini, Ruth Durrer, Thomas Konstandin, and Geraldine Servant.  General Properties of the Gravitational Wave Spectrum from Phase Transitions.  Phys. Rev. D, 79:083519, 2009.
  * [50] Jose R. Espinosa, Thomas Konstandin, Jose M. No, and Geraldine Servant.  Energy Budget of Cosmological First-order Phase Transitions.  JCAP, 06:028, 2010.
  * [51] Chiara Caprini et al.  Detecting gravitational waves from cosmological phase transitions with LISA: an update.  JCAP, 03:024, 2020.
  * [52] Marc Kamionkowski, Arthur Kosowsky, and Michael S. Turner.  Gravitational radiation from first order phase transitions.  Phys. Rev. D, 49:2837–2851, 1994.
  * [53] Ryusuke Jinno and Masahiro Takimoto.  Gravitational waves from bubble collisions: An analytic derivation.  Phys. Rev., D95(2):024009, 2017.
  * [54] Albert Einstein.  Näherungsweise Integration der Feldgleichungen der Gravitation.  Sitzungsberichte der Königlich Preußischen Akademie der Wissenschaften (Berlin, pages 688–696, January 1916.
  * [55] Albert Einstein.  Über Gravitationswellen.  Sitzungsberichte der Königlich Preußischen Akademie der Wissenschaften (Berlin, pages 154–167, January 1918.
  * [56] Daniel Cutting, Mark Hindmarsh, and David J. Weir.  Gravitational waves from vacuum first-order phase transitions: from the envelope to the lattice.  Phys. Rev. D, 97(12):123513, 2018.
  * [57] Daniel Cutting, Mark Hindmarsh, and David J. Weir.  Vorticity, kinetic energy, and suppressed gravitational wave production in strong first order phase transitions.  Phys. Rev. Lett., 125(2):021302, 2020.
  * [58] Ryusuke Jinno, Thomas Konstandin, and Henrique Rubira.  A hybrid simulation of gravitational wave production in first-order phase transitions, 10 2020.
  * [59] Chiara Caprini and Ruth Durrer.  Gravitational wave production: A Strong constraint on primordial magnetic fields.  Phys. Rev. D, 65:023517, 2001.
  * [60] Christos G. Tsagas.  Gravitational waves and cosmic magnetism: A Cosmological approach.  Class. Quant. Grav., 19:3709–3722, 2002.
  * [61] L. F. Abbott and R. K. Schaefer.  A General, Gauge-invariant Analysis of the Cosmic Microwave Anisotropy.  ApJ, 308:546, September 1986.
  * [62] Kouji Nakamura.  Second-order Gauge-invariant Cosmological Perturbation Theory: Current Status updated in 2019.  1 2020.
  * [63] P. J. E. Peebles and J. T. Yu.  Primeval adiabatic perturbation in an expanding universe.  Astrophys. J., 162:815–836, 1970.
  * [64] Eiichiro Komatsu.  Some basics of the expansion of the universe,cosmic microwave background, and large-scale structure of the universe.  https://wwwmpa.mpa-garching.mpg.de/~komatsu/cmb/lecture_cosmo_iucaa_2011.pdf, 2011\.  [Online; accessed 23-June-2021].
  * [65] Shadab Alam et al.  The clustering of galaxies in the completed SDSS-III Baryon Oscillation Spectroscopic Survey: cosmological analysis of the DR12 galaxy sample.  Mon. Not. Roy. Astron. Soc., 470(3):2617–2652, 2017.
  * [66] N. Aghanim et al.  Planck 2018 results. VI. Cosmological parameters.  Astron. Astrophys., 641:A6, 2020.
  * [67] James M. Bardeen, J. R. Bond, Nick Kaiser, and A. S. Szalay.  The Statistics of Peaks of Gaussian Random Fields.  Astrophys. J., 304:15–61, 1986.
  * [68] Roman Scoccimarro, Matias Zaldarriaga, and Lam Hui.  Power spectrum correlations induced by nonlinear clustering.  Astrophys. J., 527:1, 1999.
  * [69] Irshad Mohammed and Uros Seljak.  Analytic model for the matter power spectrum, its covariance matrix, and baryonic effects.  Mon. Not. Roy. Astron. Soc., 445(4):3382–3400, 2014.
  * [70] Stephan J. Huber, Thomas Konstandin, Germano Nardini, and Ingo Rues.  Detectable Gravitational Waves from Very Strong Phase Transitions in the General NMSSM.  JCAP, 03:036, 2016.
  * [71] Sarah Burke-Spolaor et al.  The Astrophysics of Nanohertz Gravitational Waves.  Astron. Astrophys. Rev., 27(1):5, 2019.
  * [72] Maura A. McLaughlin.  The North American Nanohertz Observatory for Gravitational Waves.  Class. Quant. Grav., 30:224008, 2013.
  * [73] Zaven Arzoumanian et al.  The NANOGrav 12.5 yr Data Set: Search for an Isotropic Stochastic Gravitational-wave Background.  Astrophys. J. Lett., 905(2):L34, 2020.
  * [74] R. N. Manchester ed al.  The Parkes Pulsar Timing Array Project.  Publications of the Astronomical Society of Australia, 30:e017, January 2013.
  * [75] L. Lentati et al.  European Pulsar Timing Array Limits On An Isotropic Stochastic Gravitational-Wave Background.  Mon. Not. Roy. Astron. Soc., 453(3):2576–2598, 2015.
  * [76] Tristan L. Smith, Elena Pierpaoli, and Marc Kamionkowski.  A new cosmic microwave background constraint to primordial gravitational waves.  Phys. Rev. Lett., 97:021301, 2006.
  * [77] Irene Sendra and Tristan L. Smith.  Improved limits on short-wavelength gravitational waves from the cosmic microwave background.  Phys. Rev. D, 85:123002, 2012.
  * [78] Luca Pagano, Laura Salvati, and Alessandro Melchiorri.  New constraints on primordial gravitational waves from Planck 2015.  Phys. Lett. B, 760:823–825, 2016.
  * [79] Paul D. Lasky et al.  Gravitational-wave cosmology across 29 decades in frequency.  Phys. Rev. X, 6(1):011035, 2016.
  * [80] Sophie Henrot-Versille et al.  Improved constraint on the primordial gravitational-wave density using recent cosmological data and its impact on cosmic string models.  Class. Quant. Grav., 32(4):045003, 2015.
  * [81] Richard H. Cyburt, Brian D. Fields, Keith A. Olive, and Evan Skillman.  New BBN limits on physics beyond the standard model from ${}^{4}He$.  Astropart. Phys., 23:313–323, 2005.
  * [82] I. Wasserman.  Late Phase Transitions and the Spontaneous Generation of Cosmological Density Perturbations.  Phys. Rev. Lett., 57:2234–2236, 1986.
  * [83] Xiao-chun Luo and David N. Schramm.  The Phenomenological status of late time phase transition models after COBE.  Astrophys. J., 421:393–399, 1994.
  * [84] Amol V. Patwardhan and George M. Fuller.  Late-time vacuum phase transitions: Connecting sub-eV scale physics with cosmological structure formation.  Phys. Rev. D, 90(6):063009, 2014.
  * [85] Joshua A. Frieman, Christopher T. Hill, and Richard Watkins.  Late time cosmological phase transitions. 1. Particle physics models and cosmic evolution.  Phys. Rev. D, 46:1226–1238, 1992.
  * [86] Sourish Dutta, Stephen D.H. Hsu, David Reeb, and Robert J. Scherrer.  Dark radiation as a signature of dark energy.  Phys. Rev. D, 79:103504, 2009.
  * [87] Kazuhiko Kojima, Toshitaka Kajino, and Grant J. Mathews.  Generation of curvature perturbations with extra anisotropic stress.  JCAP, 2010(2):018, February 2010.
  * [88] Oliver F. Piattella.  Lecture Notes in Cosmology.  UNITEXT for Physics. Springer, Cham, 2018.