# A Large and Diverse Arabic Corpus for Language Modeling

Abbas Raza Ali  Data Science and AI Lab
New York University
Abu Dhabi, UAE
<EMAIL_ADDRESS>
###### Abstract

Language models (LMs) have introduced a major paradigm shift in Natural
Language Processing (NLP) modeling where large pre-trained LMs became integral
to most of the NLP tasks. The LMs are intelligent enough to find useful and
relevant representations of the language without any supervision. Perhaps,
these models are used to fine-tune typical NLP tasks with significantly high
accuracy as compared to the traditional approaches. Conversely, the training
of these models requires a massively large corpus that is a good
representation of the language. English LMs generally perform better than
their other language counterparts, due to the availability of massive English
corpora.

This work elaborates on the design and development of a large Arabic corpus.
It consists of over 500 GB of Arabic cleaned text targeted at improving cross-
domain knowledge and downstream generalization capability of large-scale
language models. Moreover, the corpus is utilized in the training of a large
Arabic LM. In order to evaluate the effectiveness of the LM, a number of
typical NLP tasks are fine-tuned. The tasks demonstrate a significant boost
from 4.5 to 8.5% when compared to tasks fine-tuned on multi-lingual BERT
(mBERT). To the best of my knowledge, this is currently the largest clean and
diverse Arabic corpus ever collected.

###### Index Terms:

Arabic Corpus, GPT-3, Language Model, Transformers, NLP

utf8

##  I Introduction

Large Language models (LMs) have introduced a major shift in the Natural
Language Processing (NLP) paradigm which makes the fine-tuning of typical NLP
tasks effective and at the same time reduces their complexity [41, 17]. The
effectiveness of NLP tasks is dependent on the scale of the language model
training which in turn depends on the availability of massively large corpus
from diverse domains [32]. Moreover, diversity of the training data is
critical to improving the generalization of the large models from where the
downstream task, such as listed in [5], can effectively acquire knowledge
given a small amount of training data [8, 6, 17].

Arabic is the $5^{th}$ most widely spoken language in the world despite being
a low-resource language. Perhaps, the most commonly used Arabic language model
is Bidirectional Encoder Representations from Transformers (BERT) [21] which
is trained on a relatively small corpus, Arabic Wikipedia, compare to the LMs
trained for other languages. Thus, it is insufficient to represent a rich and
complex language like Arabic.

In order to address the issue, this work presents the design and development
of a 500 GB high-quality Arabic corpus which can be used to train the large
LMs. The corpus consists of 22 diverse data-sources, including both existing
and newly developed. The LMs trained on this corpus drastically elevated the
accuracy of a number of downstream tasks which is demonstrated in the coming
sections.

TABLE I: Arabic character-set used to standardize different data-sources Grapheme |  Arpabet | Description |  Grapheme |  Arpabet | Description |  Grapheme |  Arpabet | Description
---|---|---|---|---|---|---|---|---
ء¿ |  E | hamza |  خ¿ |  KH | kha |  ل¿ |  L | lam
ا¿ |  AE/E | alif |  د¿ |  D | dal |  م¿ |  M | meem
آ¿ |  AE: | alif maddah |  ذ¿ |  DH | Thal |  ن¿ |  N | noon
أ¿ |  E | alef + hamza above |  ر¿ |  R | ra |  ه¿ |  HH | ha
ؤ¿ |  E | waw + hamza above |  ز¿ |  ZH | zay |  و¿ |  W | waw
إ¿ |  E | alef + hamza below |  س¿ |  S | seen |  ى¿ |  AE | alif maksura
ئ¿ |  E | ya+ hamza above |  ش¿ |  SH | sheen |  ي¿ |  Y | ya
ؤ¿ |  E | waw + hamza above |  ص¿ |  SS | sad |  ¿ |  AE | fathah
إ¿ |  E | alef + hamza below |  ض¿ |  DD | dad |  ¿ |  IH | kasra
ئ¿ |  E | ya + hamza above |  ط¿ |  TT | ta |  ¿ |  UH | damma
ب¿ |  B | ba |  ظ¿ |  DH2 | za |  ¿ |  AE N | fathathan
ة¿ |  T/H | ta marbuta |  ع¿ |  AI | ayn |  ¿ |  IH N | kasrathan
ت¿ |  T/H | ta |  غ¿ |  GH | ghayn |  ¿ |  UH N | dammathan
ث¿ |  TH | tha |  ف¿ |  F | fa |  ¿ |  - | tashdeed
ج¿ |  ZH | jeem |  ق¿ |  Q | qaf |  ¿ |  - | sakun
ح¿ |  HH | ha |  ك¿ |  KH | kaf | -

The recent paradigm shift is triggered by the introduction of BERT in 2018 by
Google [21]. BERT outperformed the previous state-of-the-art Long-Short Term
Memory (LSTM) based Recurrent Neural Network (RNN) approaches with a
significant improvement of upto 7.7% on typical NLP tasks. Indeed the major
credit goes to a powerful state-of-the-art neural architecture known as
Transformer with an attention mechanism [49]. BERT has another variant which
is trained on 104 different languages, including Arabic, known as multi-
lingual BERT (mBERT). Besides, separate BERT models are only available for the
English and Chinese languages. The Arabic model has trained on a small corpus
and validated only on the _sentence contradiction_ task. Though, Arabic mBERT
was able to achieve state-of-the-art results in 6 text classification tasks
with a few-short task-specific fine-tuning.

A year later, [25] proposed the first Arabic _Universal Language Model_ ,
known as _hULMonA_ , and demonstrated its use for Arabic Text classification
tasks. The experimental results show that _hULMonA_ generalizes well to
multiple Arabic tasks and particularly achieved state-of-the-art on Arabic
Sentiment Analysis. Accordingly, AraBERT [13] and GigaBERT [36] are two LMs
that are trained on relatively large Arabic corpus which mostly consists of
Modern Standard Arabic (MSA) dialect. Both offer comparatively better
performance than mBERT on most of the downstream tasks.

An effort similar to this work has been made for the English language with the
introduction of Pile in 2020 [30]. The Pile is an 825 GB large and diverse
corpus that is composed of 22 smaller and high-quality sources. The diversity
in data-sources improves general cross-domain knowledge of both the LM and the
downstream generalization capability. Pile covers the diverse knowledge and
reasoning ability in a number of different domains which makes it a robust
benchmark for general text modeling ability for large LMs.

The notable contributions of this work are listed as follows:

  1. 1.

The design and development of over 500 GB Arabic corpus by combining new and
existing 22 diverse sources.

  2. 2.

Training an Arabic LM on the proposed corpus using Generative Pre-trained
Transformer 3 (GPT-3) 2.7 billion parameter architecture.

  3. 3.

A thorough evaluation of the LM demonstrating its effectiveness on 5 typical
supervised NLP tasks.

##  II Corpora

The proposed Arabic corpus is composed of 22 constituent data-sources which
are listed in Table VIII and their statistics are shown in Table IX. The
sources belong to news, academic, social, religious, cultural and other
related domains. Those sources represent a good proportion of most of the
commonly used Arabic dialects including MSA, Egyptian, Gulf, Levant & North
African (MEGLN) along with Classical Arabic (CA).

The raw corpus is gathered from different sources that possess different
character-sets in their transcripts. Hence, a common character-set has been
defined to standardize all the sources which are shown in Table I. The
character-set includes 3 short vowels including fathah, kasra and damma; 3
long vowels called alef, waw and ya; 5 Arabic diacritical marks comprising of
fathatan, kasratan, dammatan, shadda and sukun along with 36 standard
consonants of Arabic. The diacritical marks are integral to the Arabic
pronunciation. The presence of diacritical marks in the corpus could lead to
the zero-short diacritization generation task which is considered crucial for
almost all speech processing tasks [9, 4, 7]. The following sections briefly
summarize all the sources.

###  II-A ArabicWeb16

ArabicWeb16 is a web crawl of a large number of Arabic web-pages with
significant coverage of both MSA and Dialectal Arabic (DA) [47]. The corpus
comprises over 150 million web-pages crawled over the month of January 2016,
and is covering a wide variety of domains and dialects. It contains many
forums, including question-answer sites given as seeds, as well as many
informational pages such as Wikipedia, which can usefully support question-
answering research. Table II shows the distribution of dialects and domains
used in this corpus.

TABLE II: Distribution of the dialects and domains of ArabicWeb16 corpus Dialect (in millions) |  Category (in thousands)
---|---
MSA | 119 |  78.86% |  Informational | 113 |  12.80%
Egyptian | 9.2 |  6.10% |  Discussion/opinion | 295 |  33.41%
Gulf | 7.6 |  5.04% |  Media & News | 93 |  10.53%
Levantine | 7.2 |  4.77% |  Online services | 36 |  4.08%
Meghrebi | 5.1 |  3.38% |  Entertainment | 28 |  3.17%
Others | 2.8 |  1.86% |  Others | 318 |  36.01%
Total | 150.9 |  100% |  Total | 883 |  100%

###  II-B OSCAR

The Open Super-large Crawled ALMAnaCH coRpus (OSCAR) is a multi-lingual
corpus, extracted from Common Crawl (CC) by performing language
classification, filtering and cleaning. CC ’s complete archive consists of
petabytes of monthly snapshots collected since 2011 [46].

###  II-C Common Crawl (CC)

CC is a collection of 9 billion website crawls from the year 2012 till 2017,
including raw web-pages, metadata and text extractions [18]. Due to the raw
nature of the corpus, CC covers content from diverse domains at the cost of
varying quality data. Therefore, well-designed extraction and filtering
routines have been applied on the entire corpus, elaborated in Algorithm 1,
which yields higher quality output.

###  II-D 1.5 Billion Words Corpus

The corpus is an attempt to build a contemporary linguistic corpus [23]. It
contains over 1.5 billion words, out of which, there are 3 million unique
words. The corpus is collected from 5 million newspaper articles from 10
different news sources belonging to 8 Arab countries, over a period of 14
years. Table III, indicates the selected sources for the corpus, its origin,
the time period for each one of them, country of origin and statistics.

TABLE III: Statistics of 1.5 billion words corpus Source |  Country |  Period | Documents | Terms
---|---|---|---|---
Alittihad |  Emirates |  Jan’08-Jun’14 | 349,342 | 932,628
Echorouk Online |  Algeria |  Feb’8-May’14 | 139,732 | 543,799
Alriyadh |  KSA |  Oct’00-Dec’13 | 858,188 | 1,451,320
Alyaum |  KSA |  Jul’00-Dec’13 | 888,068 | 1,319,996
Tishreen |  Syria |  Jan’4-May’14 | 314,597 | 905,169
Alqabas |  Kuwait |  Jan’06-Apr’14 | 817,274 | 1,260,511
Almustaqbal |  Lebanon |  Sep’03-Apr’14 | 446,873 | 982,765
Almasry-alyoum |  Egypt |  Dec’15-Jan’14 | 291,723 | 760,511
Youm-7 |  Egypt |  Jan’8-May’13 | 1,025,027 | 1,020,444
Saba News |  Yemen |  Dec’9-May’14 | 92,149 | 255,098
Total | 5,222,973 | 9,432,241

###  II-E OSIAN

Open Source International Arabic News (OSIAN) is an open-source corpus
compiled from 31 international Arabic news broadcasting platforms. The clean
and filtered data consists of about 3.5 million articles which are extracted
from 6 million web-pages. These articles comprise of over 37 million sentences
and roughly 1 billion tokens [52]. This effort aimed to build a balanced
corpus in which the data must be drawn from a wide range of reliable and
freely available sources. The crawling was conducted in the month of March
2018. Table IV lists the demography and domain-wise statistics of the corpus.

TABLE IV: List of OSIAN crawled web-domains Demography |  Web-domain | Articles
---|---|---
International |  un.org, euronews.com, reuters.com, namnewsnetwork.org, sputniknews.com | 693,629
Middle-east |  aljazeera.net, alarabiya.net | 366,211
Algeria |  djazairess.com | 588,514
Australia |  eltelegraph.com | 4,614
Canada |  arabnews24.ca, halacanada.ca | 30,135
China |  arabic.cctv.com | 1,365
Egypt |  alwatanalarabi.com | 85,351
France |  france24.com | 74,718
Iran |  alalam.ir | 344,011
Iraq |  iraqakhbar.com | 28,248
Germany |  dw.com | 117,261
Jordon |  sarayanews.com | 49,461
Morocco |  www.marocpress.com | 188,045
Palestine |  al-ayyam.ps | 81,495
Qatar |  raya.com | 8,986
Russia |  arabic.rt.com | 57,238
Saudi Arabia |  alwatan.com.sa | 1,512
Sweden |  alkompis.se | 33,790
Syria |  syria.news | 36542
Tunisia |  www.turess.com | 495,674
Turkey |  turkey-post.net, aa.com.tr | 76,638
UAE |  emaratalyoum.com | 25,081
UK |  bbc.com | 10,686
USA |  arabic.cnn.com | 113,557
Total | 3,512,762

###  II-F Arabic Wikipedia

The Arabic Wikipedia articles of January 2021 Wikimedia format data dump is
used. The content is mostly in MSA and Egyptian DA [28].

###  II-G English Wikipedia (translated)

The English Wikipedia articles of October 2021 Wikimedia format data dump is
translated into Arabic using automatic Machine Translation (MT). The content
is mostly in MSA dialect.

###  II-H MGB-2

The Multi-Genre Broadcast-2 (MGB-2) corpus is taken from the second edition of
the MGB-2 Challenge [34]. The corpus is composed of the manually captioned
recorded programs of Al-Jazeera Satellite Channel (Al-Jazeera) TV spanning
over 10 years. The quality of the transcription varies significantly including
multiple dialects and overlapping talkers, which is the typical scenario for
political debate and talk shows. Based on a rough estimation, more than 70% of
the speech is MSA, and the rest is spoken in different DA.

###  II-I Arabic Wiki Books

Arabic Wiki Books consists of sharded files from Arabic Wikipedia that are
cleaned and formatted [33]. This source is merged with another Arabic books
source known as Al-Shamela [15]. Moreover, several books were excluded from
the dataset due to bad formatting.

###  II-J Arabic News and Ajdir Corpus

The Arabic newspaper articles corpus is collected from British Broadcasting
Corporation (BBC) Arabic, EuroNews, Al-Jazeera, Cable News Network (CNN)
Arabic and Russia Today (RT) Arabic. These news articles were collected in
April 2019 [43]. Table V is reporting the statistics of the 5 different
sources.

Ajdir is another multi-source news corpus comprising of 113 million tokens
[1].

TABLE V: Statistics of Arabic News corpus Source | Documents | Tokens | Terms
---|---|---|---
BBC | 212,271 | 1,764,796 | 1,076,526
RT | 368,920 | 3,411,451 | 2,080,985
Al-Jazeera | 249,106 | 1,525,372 | 930,477
EuroNews | 46,468 | 517,227 | 315,508
CNN | 30,338 | 317,260 | 193,529
Total | 907,103 | 7,536,106 | 4,597,025

###  II-K OSAC

Open-Source Arabic Corpora (OSAC) has a collection of journalistic text corpus
which is freely accessible [44]. The corpus consists of web documents
extracted from over 25 Arabic websites, comprising of roughly 113 million
tokens, which are categorized into 3 different sources listed in Table VI. The
BBC and CNN news articles belong to 7 categories including Middle East (ME),
world news, business & economy, sports, international press, science &
technology and art & culture. Besides, it is collected from multiple websites
belonging to 10 categories including economics, history, education & family,
religion & fatwas, sports, health, astronomy, law, stories and cooking
recipes.

TABLE VI: OSAC corpus statistics Source | Documents | Tokens | Terms
---|---|---|---
BBC | 4,763 | 1,860,786 | 106,733
CNN | 5,070 | 2,241,348 | 144,460
OSAC | 22,429 | 18,183,511 | 449,600
Total | 32,262 | 22,285,645 | 700,793

###  II-L Common Crawl 100 (CC-100)

It is a collection of sources comprising mono-lingual data for 100+ languages
and also includes data for romanized languages [50]. It was constructed using
the URLs and paragraph indices provided by the CC-Net repository of 2018 CC
snapshots. Each file comprises documents separated by double-newlines and
paragraphs within the same document separated by a single newline.

###  II-M Daypop News

Daypop is an Arabic news articles corpus gathered from February to October
2020. The content is mostly in MSA DA [20].

###  II-N Tashkeela

Tashkeela is a fully-diacritized Arabic corpus that is commonly used to train
NLP tasks such as automatic diacritization and disambiguity resolution [48].
The corpus contains 75.6 million fully-vocalized words mainly collected from
97 books belong to MSA and CA dialects. The sources of the corpus along with
their statistics are listed in Table VII. Additionally, a number of small
fully-diacritized sources belonging to MEGLN dialects have been collected
[51].

TABLE VII: Tashkeela corpus statistics Source | Dialect | Tokens
---|---|---
Shamela Library - 97 Books | CA | 74,762,008
Modern Books - 20 | MSA | 398,911
Crawl from learning.aljazeera.net, al-kalema.org, enfal.de | MSA | 461,283
Manually diacritized | MSA | 7,701
Total | 75,629,903

###  II-O KSUCCA

King Saud University Corpus of Classical Arabic (KSUCCA) is a pioneering 50
million tokens annotated corpus representing CA dialect belonging to the
period of the pre-Islamic era until the $4^{th}$ Hijri century [11]. This
corpus aims at studying the distributional lexical semantics of _The Holy
Quran_ words. However, it can also be used for other research purposes, such
as Information Retrieval (IR), MT, and Question Answering (QA). Additionally,
an automatically annotated information of the corpus comprising lemma, stem,
Part-of-speech (POS) tags, gender and number annotations is also available.

###  II-P NQ and SQuAD

Natural Questions (NQ) is a question-answering corpus where the questions are
anonymized and aggregated queries are made through the Google search engine
[35]. The corpus consists of 307,373 training examples in the English language
with single annotations. This corpus is translated from English to Arabic
followed by linear parse by the human. NQ corpus is combined with an Arabic
translated version of Stanford Question Answering Dataset (SQuAD) [42]. SQuAD
is the most widely used corpus to train Machine Reading Comprehension (MRC)
models. It has 100,000 question-answer pairs.

###  II-Q EAPCOUNT and AMARA

English–Arabic Parallel Corpus of the United Nations Texts (EAPCOUNT) is a
well-known English-Arabic parallel corpus comprising of 5,392,491 words
compiled using two sources including the English originals and their Arabic
translations. The content of the corpus mainly includes resolutions and annual
reports issued by different United Nations’ organizations, along with the text
taken from publications of other international institutions [53].

AMARA is another source of online educational video subtitles which provides
multi-lingual alignment for 20 languages [2].

###  II-R ArSAS and Arabic Tweets

Arabic Speech-Act and Sentiment (ArSAS) is an Arabic corpus of tweets
annotated for sentiment analysis task [26]. A large set of 21,000 Arabic
tweets were collected from a number of different domains. In addition to this,
the same set of tweets were categorized into 4 classes including positive,
negative, neutral and mixed. This corpus is combined with another large corpus
of sentiment analysis having 4,418,128 tweets known as Arabic Tweets Corpus
[45].

###  II-S OpenITI

Open Islamicate Texts Initiative (OpenITI) is a large-scale, historical corpus
of Arabic comprising 1 billion words from diverse periods of time. The corpus
is cleaned and processed with a morphological analyzer. This corpus is
combined with the second version of Al-Shamela which contains 6,111 historical
Islamic books, available until the beginning of Rajab 1433 AH.

TABLE VIII: Overview of the data-sources used to develop the proposed corpus Corpus Name | Language | Dialect | Domain | Duration | Reference
---|---|---|---|---|---
ArabicWeb16 | Arabic & English | Multi-dialect | Cross | Jan 2016 | [47]
OSCAR | Arabic |  MSA & Egyptian | Cross | 2011 | [46]
CC | Arabic | MEGLN | Cross | 2012-2017 | [18]
1.5 Billion Words Corpus | Arabic | MEGLN | News | 2000-2014 | [23]
OSIAN Corpus | Arabic | Multi-dialect | News | Mar 2018 | [52]
Arabic Wikipedia | Arabic |  MSA & Egyptian | Cross | Feb 2021 | [29]
English Wikipedia (translated) | Arabic & English | MSA | Cross | Oct 2020 | [28]
MGB-2 | Arabic | MEGLN | News | 2005-2015 | [34]
Arabic Wiki Books | Arabic | MSA | Education | Feb 2020 | [33]
Arabic News and Ajdir Corpus | Arabic | MSA | News | Apr 2019 | [43]
OSAC | Arabic | Multi-dialect | News | 2010 | [44]
CC-100 | Arabic | MSA | Cross | Jan-Dec 2018 | [50]
Daypop News | Arabic | MSA | News | Feb-Oct 2020 | [20]
Tashkeela | Arabic |  MEGLN & CA | Education | - | [51, 48]
KSUCCA | Arabic | CA | Education | - | [11]
NQ and SQuAD (translated) | Arabic | MSA | Cross | - | [35, 42]
EAPCOUNT and Amara | Arabic & English | MSA | Cross | - | [53, 10]
ArSAS and Arabic Tweets Corpus | Arabic | MSA | Social | - | [26]
OpenITI | Arabic | CA | Religion | - | [15, 14]
Arabic Books Corpus | Arabic | Multi-dialect | Education | - | [16]
Hadith Books | Arabic | CA | Religion |  9th century | [31]
Arabic Poetry | Arabic | Multi-dialect | Culture | till 2019 |  [3] [22]

###  II-T Arabic Books Corpus

Books Corpus is a large Arabic corpus developed as part of a research project
named _A New Approach of Semi-Indexing of Text Documents_ [16]. The corpus
consists of more than 460 Arabic books. It can be used for the development of
language engineering applications, IR and information extraction. The total
corpus size is 137 MB It contains 23,264,785 tokens.

###  II-U Hadith Books

Hadith Books contains Hadith from the 9 sahih books scraped from _islambook_
website. There are total 62,169 Hadith where Sahih Bukhari has 7,008, Sahih
Muslim has 5,362, Sunan Al-Tirmidhi has 3,891, Sunan al-Nasai has 5,662, Sunan
Abu-Dawud has 4,590, Sunan Ibn Maja has 4,332, Musnad Ahmad ibn Hanbal has
26,363, Maliks Muwatta has 1,594 and Sunan al Darami has 3,367 Hadiths.

###  II-V Arabic Poetry

The Arabic Poetry corpus is the oldest and the most prominent form of Ancient
Arabic poetry available today [3]. The corpus contains over 58,000 poems that
extend from the 6th century to the present day. This is combined with a song
lyrics corpus, known as _Habibi_ , which is the first Arabic song lyrics
corpus collected by [22]. This source comprises more than 30,000 Arabic song
lyrics in 6 Arabic dialects for singers from 18 Arabic countries. The lyrics
are segmented into more than 500,000 song verses with more than 3.5 million
words.

TABLE IX: The statistics of the raw and clean sources where the text size is in GB Corpus Name | Documents | Raw Text | Clean Text
---|---|---|---
ArabicWeb16 | 3,005 | 12,000.00 | 99.80
OSCAR | 46 | 80.07 | 76.07
CC | 25 | 1,200.00 | 250.00
1.5 Billion Words Corpus | 10 | 16.50 | 15.00
OSIAN Corpus | 24 | 4.70 | 4.50
Arabic Wikipedia | 12 | 21.20 | 6.60
English Wikipedia (translated) | 112 | 22.00 | 13.00
MGB-2 | 1 | 1.30 | 1.20
Arabic Wiki Books | 512 | 5.70 | 5.50
Arabic News and Ajdir Corpus | 203 | 1.816 | 1.80
OSAC | 32,262 | 0.29 | 0.27
CC-100 | 1 | 28.00 | 26.00
Daypop News | 1 | 0.89 | 0.86
Tashkeela | 397 | 1.493 | 1.34
KSUCCA | 410 | 0.44 | 0.44
NQ and SQuAD | 58 | 0.333 | 0.32
EAPCOUNT and Amara | 17,455 | 9.55 | 9.09
ArSAS and Arabic Tweets Corpus | 3 | 0.770 | 0.76
OpenITI | 7,145 | 13.85 | 13.00
Arabic Books Corpus | 462 | 0.234 | 0.22
Hadith Books | 18 | 0.10 | 0.90
Arabic Poetry | 30,073 | 0.209 | 0.09
Total | 92,218 | 13,000 | 501.42

##  III Data Cleansing

The corpus is collected from multiple sources spanning from high-quality to
very noisy corpus. In addition to that, every source comes up with its own
character-set, diacritical marks and other normalization issues. Hence, data
cleansing and standardization become critical when developing a corpus from
multiple diverse sources that possess different rules. The text of the data-
sources has been standardized on the character-sets defined in Table I.
Moreover, any character which is not part of this character-set is considered
as a _noisy character_. Accordingly, the alphabets are normalized using the
same character-set where _hamza above_ diacritical mark is normalized by
combining with long vowels instead of separately putting hamza diacritical
mark on above or below of the alphabets. Furthermore, both English and Arabic
punctuation marks have been found in the raw text which are normalized to
Arabic punctuation marks. The punctuation marks other than “! ? .” have been
removed from the text. For instance, the hyphen is used as a sentence marker
and also Arabic version of the comma, semi-colon and question mark have been
used instead of their English versions. Conversely, the Arabic digits are
normalized to their corresponding English digits. Since the English digits are
frequently occurring in the Arabic text.

In order to further reduce lexical sparsity, all the sources have been cleaned
by masking the web-links, email addresses, phone numbers and similar kinds of
identification-related information with their corresponding tags. All the
pairs of a slash followed by a hyphen are removed from the text. Sometimes in
Arabic, the characters are repeated more than twice in a word either to
emphasize or make it the title of the document, known as _tatweel_. Tatweel is
also stripped during the cleansing process. The tokenization is done using
Farasa word segmentor [19] and the emojis are preserved in the text. The
Arabic word segmentation is applied before byte-pair encoding (BPE)
tokenization which is found to be effective for the downstream tasks. The data
cleansing pseudocode is outlined in Algorithm 1.

// replace URLs with tag

$sentence$.replace(URLs, [link]);

// replace emails with tag

$sentence$.replace(Emails, [mail]);

// remove html markup

$sentence$.replace(html_markup, ‘’);

// remove tatweel

$sentence$.replace(ـ¿, ‘’);

$sentence$ = normalize hamza above أؤئ¿;

$sentence$ = remove parentheses & content between them;

$sentence$ = standardize Arabic characters;

// remove punctuation

$sentence$.replace(punctuation, ‘’);

// remove noisy characters

$sentence$.replace(noisy_characters, ‘’);

Algorithm 1 Raw data pre-processing

##  IV Language Modeling

The cleaned text has been used to train a large Arabic LM. The model is
trained without any supervision as the next word prediction problem. In order
to train the model, GPT-3 architecture having 2.7 billion parameters has been
used. The model has 32 transformer decoder layers, each of which consists of
self-attention and feed-forward sub-layers where each layer has 32 attention
heads and each head has 80 dimensions. Moreover, a large batch size of 1
million instances has been used along with a learning rate of 1.6x10-4.

A mesh-Tensorflow implementation has been used to train the model since the
GPT-3 code implementation is not available [24]. Furthermore, [40]
implementation was used for BPE which converts the text into a series of
integers. The input was transformed into BPE before feeding into the model.
The model was trained on 128 x Tensor Processing Unit (TPU) v3 for 320 hours.

TABLE X: List of tasks data-sources and their evaluation on the proposed LM vs mBERT Pre-trained Networks | Pre-trained Networks
---|---
Task | Corpus Name | Size | Proposed LM | BERT
MRC |  SQuAD (translated) | 48,344 pairs | 76.68 | 69.38
Text Summarization | KALIMAT | 20,291 articles | 41.99 | 32.08
Diacritics Restoration |  King Saud University (KSU) transcript | 10,000 sentences | 98.21 | 95.39
Sentiment Analysis | Arabic Hotel Reviews | 24,028 comments | 95.97 | 91.58
POS Tagger |  Arabic Penn Tree Bank (PTB) | 738,845 tokens | 15.52 | 23.57

##  V Evaluation

The LM is evaluated by fine-tuning it on a number of typical Arabic NLP tasks,
such as MRC, Text Summarization, Diacritics Restoration, Sentiment Analysis
and POS Tagger. Though the proposed LM is a Generative Pre-trained Transformer
(GPT) style model, however, it is compared with the most frequently used LM,
mBERT. Perhaps Both of them are based on the transformer architecture,
however, they are fundamentally different where BERT has just the encoder
blocks conversely GPT has just the decoder blocks from the transformer.

The MRC task is fine-tuned on the Arabic translated version of SQuAD
comprising of 48,344 question-answer pairs [39]. The evaluation was done using
the Sentence Match (SM) metric which measures the percentage of predictions
that fall in the same sentence in the paragraph as the ground truth answer.

Text Summarization is trained on a multi-purpose Arabic corpus containing
20,291 articles with their extractive summaries. The dataset belongs to 6
broad categories including culture, economy, local news, international news,
religion and sports. The quality of summarization is evaluated and compared
with the BERT fine-tuned system using Rough measure [37].

The diacritics restoration task is fine-tuned on the transcript of the widely
used Arabic speech recognition corpus, known as KSU [12]. The corpus has
10,000 rich and balanced sentences manually diacritized on MSA dialect. Word
Error Rate (WER) is computed at the word-level to evaluate this task.

For the sentiment analysis task, Hotel Arabic Reviews Dataset (HARD) comprises
24,028 hotel reviews, representing both MSA in DA, has been used for the fine-
tuning [27]. The reviews are categorized into positive, negative and neutral
reviews. In order to determine the effectiveness of the fine-tuning, the
accuracy metric was adopted.

Penn Arabic Treebank (PATB) is the most commonly used dataset for POS Tagging
[38]. It contains approximately 300,000 pairs of Arabic words which are
categorized into 33 POS tags. The performance of the POS tagger was evaluated
using the WER measure.

The LM trained on the proposed corpus has significantly outperformed the
multi-lingual BERT on all the downstream NLP tasks. The statistics of the few-
shot learning on the proposed LM along with their comparison with the tasks
fine-tuned on BERT are reported in Table X.

##  VI Conclusion

This work explains the design and development of the largest clean and diverse
Arabic Corpus, particularly, to train large-scale LMs. Although Arabic is the
$5^{th}$ most widely spoken language, however, the Arabic LMs are trained on a
relatively small corpus which is not a good representation of this kind of
rich language. Therefore, the Arabic NLP tasks persistently struggled to
perform better than English tasks which raised the need of a diverse Arabic
corpus that can be enough to train a large Arabic LM.

The size of the proposed corpus is over 500 GB which aims at improving cross-
domain knowledge and downstream generalization capability of large LMs.
Further, the corpus is utilized to train a GPT-3 2.7 billion parameters model.
This model is evaluated by fine-tuning different supervised learning NLP
tasks. The evaluation demonstrated the effectiveness of the diverse corpus and
a large LM trained using it. The tasks which were fined-tuned using the
proposed LM, significantly outperformed the ones downstream on multi-lingual
BERT.

## Acknowledgments

The author wishes to thank TensorFlow Research Cloud (TFRC) for the free
access to Cloud TPUs, couldn’t have done it without this program. Also a
special thanks to Dr. Marcin Budka, Professor of Data Science at Bournemouth
University, UK for taking part in the subject discussion and review of this
work.

## References

  * [1] Ahmed Abdel-Ali.  Aracorpus-multi-source news corpus, 2005.
  * [2] Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and et al.  The amara corpus: Building parallel language resources for the educational domain.  In Proceedings of the $9^{th}$ International Conference on Language Resources and Evaluation (LREC’14), 2014.
  * [3] Fahd Alhazmi.  Arabic poetry dataset (6$5^{th}$ \- 21$5^{st}$ century), 2018.
  * [4] Abbas Raza Ali.  Automatic urdu diacritization.  In MS (CS) Thesis, Department of Computer Science, National University of Computer and Emerging Sciences, 2009.
  * [5] Abbas Raza Ali.  Cognitive computing to optimize it services.  In IEEE 17$5^{th}$ International Conference on Cognitive Informatics and Cognitive Computing (ICCI*CC), 2018.
  * [6] Abbas Raza Ali.  Meta-level learning for the effective reduction of model search space.  In PhD Thesis, Faculty of Science and Technology, Bournemouth University, 2020.
  * [7] Abbas Raza Ali.  Multi-dialect arabic speech recognition.  In International Joint Conference on Neural Networks (IJCNN), pages 1–7, 2020.
  * [8] Abbas Raza Ali, Marcin Budka, and Bogdan Gabrys.  Towards meta-learning of deep architectures for efficient domain adaptation.  Pacific Rim International Conference on Artificial Intelligence (PRICAI): Trends in Artificial Intelligence, pages 66–79, aug 2019.
  * [9] Abbas Raza Ali and Sarmad Hussain.  Automatic diacritization for urdu.  In Proceedings of the Conference on Language and Technology (CLT), 2010.
  * [10] Hind M. Alotaibi.  Arabic-english parallel corpus: A new resource for translation training and language teaching.  Arab World English Journal, 8, 2017.
  * [11] Maha Alrabiah, Abdulmalik Alsalman, and Eric Atwell.  The design and construction of the 50 million words ksucca king saud university corpus of classical arabic.  In In Proceedings of the Second Workshop on Arabic Corpus Linguistics (WACL-2), 2013.
  * [12] Mansour Alsulaiman, Zulfiqar Ali, Ghulam Muhammed, Mohamed Bencherif, and Awais Mahmood.  Ksu speech database: Text selection, recording and verification.  In In Proceedings of the European Modelling Symposium, 2013.
  * [13] Wissam Antoun, Fady Baly, and Hazem Hajj.  Aragpt2: Pre-trained transformer for arabic language generation.  Computing Research Repository (CoRR), 2012.15520, 2020.
  * [14] Yonatan Belinkov, Alexander Magidow, Alberto Barron-Cedeno, Avi Shmidman, and Maxim Romanov.  Studying the history of the arabic language: language technology and a large-scale historical corpus.  Proceedings of the $11^{th}$ Language Resources and Evaluation Conference, pages 771–805, 2019.
  * [15] Yonatan Belinkov, Alexander Magidow, Maxim Romanov, and et al.  Shamela: A large-scale historical arabic corpus.  Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH), pages 45–53, Dec 2016.
  * [16] Nagoudi El Moatez Bellah.  Arabic books corpus, 2014.
  * [17] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, and et al.  Language models are few-shot learners.  Computing Research Repository (CoRR), 2005.14165, 2020.
  * [18] Christian Buck, Kenneth Heafield, and Bas van Ooyen.  N-gram counts and language models from the common crawl.  In Proceedings of the Language Resources and Evaluation Conference, May 2014.
  * [19] Kareem Darwish and Hamdy Mubarak.  Farasa: A new fast and accurate arabic word segmenter.  In Proceedings of the $10^{th}$ International Conference on Language Resources and Evaluation (LREC’16), pages 1070–1074, May 2016.
  * [20] Daypop.  Arabic news, 2020.
  * [21] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and et al.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1:4171–4186, 2019.
  * [22] Mahmoud El-Haj.  Habibi - a multi dialect multi national Arabic song lyrics corpus.  In Proceedings of the $12^{th}$ Language Resources and Evaluation Conference, pages 1318–1326. European Language Resources Association, May 2020.
  * [23] Ibrahim Abu El-Khair.  1.5 billion words arabic corpus.  Computing Research Repository (CoRR), 1611.04033, 2016.
  * [24] EleutherAI.  Gpt neo, gpt neox, 2020.
  * [25] Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj, and Khaled Shaban.  hulmona: The universal language model in arabic.  In In Proceedings of the $4^{th}$ Arabic Natural Language Processing Workshop, ACL, pages 68–77, 2019.
  * [26] AbdelRahim Elmadany, Hamdy Mubarak, and Walid Magdy.  Arsas: An arabic speech-act and sentiment corpus of tweets.  In The $3^{rd}$ Workshop on Open-Source Arabic Corpora and Processing Tools, LREC, 2018.
  * [27] Ashraf Elnagar, Yasmin S. Khalifa, and Anas Einea.  Hotel Arabic-Reviews Dataset Construction for Sentiment Analysis Applications, pages 35–52.  Springer, 2018.
  * [28] Wikimedia Foundation.  English wikipedia dump, 2020.
  * [29] Wikimedia Foundation.  Arabic wikipedia dump, 2021.
  * [30] Leo Gao, Stella Biderman, Sid Black, and et al.  The Pile: An 800gb dataset of diverse text for language modeling.  Computing Research Repository (CoRR), 2101.00027, 2020.
  * [31] Abdulrahman Kamr.  Arabic hadith-nine books, 2020.
  * [32] Jared Kaplan, Sam McCandlish, Tom Henighan, and et al.  Scaling laws for neural language models.  Computing Research Repository (CoRR), 2001.08361, 2020.
  * [33] Abed Khooli.  Arabic wiki books, 2020.
  * [34] Sameer Khurana and Ahmed Ali.  Qcri advanced transcription system (qats) for the arabic multi-dialect broadcast media recognition: Mgb-2 challenge.  In In IEEE Spoken Language Technology Workshop (SLT), 2016.
  * [35] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, and et al.  Natural questions: a benchmark for question answering research.  Transactions of the Association of Computational Linguistics, 2019\.
  * [36] Wuwei Lan, Yang Chen, Wei Xu, and et al.  An empirical study of pre-trained transformers for arabic information extraction.  CoRR, abs/2004.14519, 2020.
  * [37] Chin-Yew Lin.  Looking for a few good metrics: Rouge and its evaluation.  In Working Notes of NTCIR-4, 2004.
  * [38] Mohamed Maamouri, Ann Bies, Tim Buckwalter, and et al.  Penn arabic treebank, part-3 v1.0.  In Linguistic Data Consortium (LDC), 2004.
  * [39] Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj.  Neural arabic question answering.  In Proceedings of the $4^{th}$ Arabic Natural Language Processing Workshop, pages 108–118. Association for Computational Linguistics, Aug 2019.
  * [40] OpenAI.  Gpt-3 encoder/decoder, 2020.
  * [41] Alec Radford, Jeff Wu, Rewon Child, and et al.  Language models are unsupervised multitask learners.  OpenAI, 2019.
  * [42] Pranav Rajpurkar, Robin Jia, and Percy Liang.  Know what you don’t know: Unanswerable questions for squad.  Computing Research Repository (CoRR), 1806.03822, 2018.
  * [43] Motaz K. Saad.  Arabic news, 2018.
  * [44] Motaz K. Saad and Wesam Ashour.  Osac: Open source arabic corpus.  In Proceedings of the $6^{th}$ International Conference on Electrical and Computer Systems (EECS’10), pages 1–6, Nov 2010.
  * [45] Abu Bakr Soliman.  Arabic tweets corpus, 2017.
  * [46] Pedro Javier Ortiz Suarez, Laurent Romary, and Benoit Sagot.  A monolingual approach to contextualized word embeddings for mid-resource languages.  In In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Jan 2020.
  * [47] Reem Suwaileh, Mucahid Kutlu, Nihal Fathima, and et al.  Arabicweb16: A new crawl for today’s arabic web.  In In Proceedings of the 39$5^{th}$ International ACM SIGIR conference on Research and Development in Information Retrieval, pages 673–676, 2018.
  * [48] Amar Balla Taha Zerrouki.  Tashkeela: Novel corpus of arabic vocalized texts, data for auto-diacritization systems.  In Data in Brief, 11:147–151, 2017.
  * [49] Ashish Vaswani, Noam Shazeer, Niki Parmar, and et al.  Attention is all you need.  Computing Research Repository (CoRR), 1706.03762, 2017.
  * [50] MGuillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, and et al.  Ccnet: Extracting high quality monolingual datasets from web crawl data.  In Proceedings of the $12^{th}$ Language Resources and Evaluation Conference (LREC), pages 4003–4012, 2020.
  * [51] Hamid Zarrabi-Zadeh.  Tanzil, 2007.
  * [52] Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and et al.  OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure.  In Proceedings of the 4$5^{th}$ Arabic Natural Language Processing Workshop, pages 175–182. Association for Computational Linguistics, Aug 2019.
  * [53] Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen.  The united nations parallel corpus v1.0.  In Proceedings of the $10^{th}$ International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534, 2016.