# ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic

Muhammad Abdul-Mageed† AbdelRahim Elmadany† El Moatez Billah Nagoudi†
Natural Language Processing Lab
The University of British Columbia
<EMAIL_ADDRESS>
###### Abstract

Masked language models (MLM) have become an integral part of many natural
language processing systems. Although multilingual MLMs have been introduced
to serve many languages, these have limitations as to their capacity and the
size and diversity of non-English data they are pre-trained on. In this work,
we remedy these issues for Arabic by introducing two powerful deep
bidirectional transformer-based models, ARBERT and MARBERT, that have superior
performance to all existing models. To evaluate our models, we propose
ArBench, a new benchmark for multi-dialectal Arabic language understanding.
ArBench is built using $41$ datasets targeting $5$ different tasks/task
clusters, allowing us to offer a series of standardized experiments under rich
conditions. When fine-tuned on ArBench, ARBERT and MARBERT collectively
achieve new SOTA with sizeable margins compared to all existing models such as
mBERT, XLM-R (Base and Large), and AraBERT on $37$ out of $45$ classification
tasks on the $41$ datasets (%82.22). Our models are publicly available for
research. †† † All authors contributed equally; order is alphabetical.

##  1 Introduction

Language models (LMs) exploiting self-supervised learning such as BERT Devlin
et al. (2019) and RoBERTa Liu et al. (2019a) have recently emerged as powerful
transfer learning tools that help improve a very wide range of natural
language processing (NLP) tasks such as part of speech tagging, named entity
recognition, text classification, natural language inference, and question
answering. Given the success of monolingual LMs, researchers have developed
multilingual versions for application on $100+$ languages. Examples are
multilingual BERT (mBERT) Devlin et al. (2019) and XLM-RoBERTa (XLM-R) Conneau
et al. (2020). Although multilingual models are effective, they are usually
outperformed by monolingual models trained using larger vocabulary and more
sizable language-specific datasets.

Many dedicated language models have been developed. These include AraBERT for
Arabic Antoun et al. (2020), Bertje for Dutch de Vries et al. (2019),
CamemBERT Martin et al. (2020) and FlauBERT Le et al. (2020) for French,
PhoBERT for Vietnamese Nguyen and Nguyen (2020), and the models presented by
Virtanen et al. (2019) for Finnish, Dadas et al. (2020) for Polish, and
Malmsten et al. (2020) for Swedish. Pyysalo et al. (2020) also create
monolingual LMs for 42 languages exploiting Wikipedia data.

The model developed for Arabic, AraBERT Antoun et al. (2020), yields promising
performance. However, it suffers from a number of issues some of which are
serious. First, in spite of the prominently positive impact of data size on
model performance, AraBERT does not make full use of available and/or easily
accessible data. Second, it is pre-trained with only Modern Standard Arabic
(MSA) data. This largely limits AraBERT’s applicability to tasks involving
dialects. Arabic dialects vary from MSA at various linguistic levels, thus
rendering a model that only serves MSA significantly sub-optimal. The critical
need for an Arabic LM that serves dialects also arise from the explosive use
of these dialects on social media. Third, AraBERT was evaluated on only three
tasks: named entity recognition, question answering, and named entity
recognition. Hence it is not clear how much it would perform on more diverse
datasets and a wider range of tasks. Fourth, AraBERT has not been compared to
the current SOTA multilingual model XLM-R Conneau et al. (2020), making it
hard to decide which model to choose when working on Arabic.

In this work, we introduce ARBERT and MARBERT, two Arabic-focused MLMs
exploiting large-to-massive scale datasets. For evaluation, we also introduce
a novel Arabic natural language understanding Benchmark (ArBench) comprising a
very wide range of NLP tasks/task clusters. Namely, we evaluate on sentiment
analysis (SA), a cluster of social meaning prediction tasks (SM), topic
classification (TC), dialect identification (DI), and named entity recognition
(NER). For each of these tasks/task clusters, we collect multiple datasets
from published research and provide standard splits for comparisons whenever
appropriate. ArBench comprises $41$ datasets from MSA and Arabic dialects,
making it by far the largest and most diverse Arabic NLP benchmark. Our goal
is for ArBench to serve the existing critical need for measuring progress on
Arabic, making it easier to compare models across tasks.

To summarize, our contributions are as follows:

  1. 1.

We develop ARBERT and MARBERT, two novel Arabic-specific Transformer-based MLS
pre-trained on very large diverse datasets to facilitate transfer learning on
MSA as well as Arabic dialects.111Our models are available for research at:
https://github.com/UBC-NLP/marbert.

  2. 2.

We introduce ArBench, a new benchmark developed by collecting and
standardizing splits on a large number of datasets across $5$ different NLU
tasks/task clusters, thereby facilitating measuring progress on Arabic NLU
across models and tasks.

  3. 3.

We fine-tune our new powerful models on our new benchmark and provide an
extensive set of comparisons to available models. Our models achieve new SOTA
on all tasks/task clusters on the majority of the datasets.

The rest of the paper is organized as follows: In Section 2, we provide an
overview of related work. Section 3 describes our Arabic pre-tained models. In
Section 4, we present the datasets comprising our new benchmark, ArBench,
across $5$ different tasks/task clusters, and evaluate our models against
other models. We conclude in Section 5.

##  2 Related Work

Distributed representations of words, as in Word2Vec Mikolov et al. (2013),
GloVe Pennington et al. (2014) and FastText Mikolov et al. (2017), have
brought significant improvements to NLP. Contextualized word embeddings such
as ELMo Peters et al. (2018) and Flair Akbik et al. (2018) have made it
possible to provide more context-sensitive (hence more accurate)
representations of words, and a growing list of embeddings Akbik et al. (2019)
followed. More recently, models exploiting a self-supervised objective with
masking such as BERT Devlin et al. (2019) have brought even more powerful
representations. Better hyper-parameter optimization, such as in RoBERTa Liu
et al. (2019b), have resulted in further improvements. Multilingual versions
of MLMs such as mBERT and XLM-Roberta Conneau et al. (2019) were also trained.
Other models with different objectives and/or architectures such as ALBERT Lan
et al. (2019), T5 Raffel et al. (2019) and GPT3 Brown et al. (2020) were also
introduced. We briefly describe each of these models next.

###  2.1 BERT

BERT Devlin et al. (2019) stands for bidirectional encoder representations
from Transformers, and is trained with a masked language modeling as well as a
next sentence prediction objective. Authors use WordPiece embeddings Wu et al.
(2016), with learned positional embeddings and a sequence length up to $512$
tokens. Devlin et al. (2019) present two architectures. Denoting the number of
layers as L, the hidden size as H, the number of self-attention heads as A,
vocabulary size as V, these two models are as follows:

  * •

BERTBase: L=$12$, H=$768$, A=$12$, V=$30$K, Total Parameters=$110$M.

  * •

BERTLarge: L=$24$, H=$1,024$, A=$16$, V=$30$K, Total Parameters=$340$M.

BERT models are trained on English Wikipedia 222https://www.wikipedia.org/
($2.5$B words) and BooksCorpus Zhu et al. (2015). The mBERT model is the
multilingual version of BERT and is trained on Wikipedia for $104$ languages
(including Arabic), with L=$12$ layers, A=$12$, H=$768$, V=$110$K, making up
$110$M parameters.

###  2.2 XLM-RoBERTa

Liu et al. (2019a) introduce RoBERTa, a model similar to BERT but with better
optimized hyper-parameters. Conneau et al. (2019) extend RoBERTA to the
multilingual setting by creating XLM-RoBERTa (XLM-R). XLM-R is trained on more
than $2$TB of filtered CommonCrawl data in $100$ languages. and comes in two
sizes and architectures:

  * •

XLM-RBase: L=$12$, H=$768$, A=$12$, V=$250$K, Total Parameters=$270$M.

  * •

XLM-RLarge: L=$24$, H=$1,024$, A=$16$, V=$250$K, Total Parameters=$550$M.

While both XLM-R models use the same masking objective as BERT, they do not
include the next sentence prediction objective used in BERT. Conneau et al.
(2019) report SOTA performance on several tasks, including sequence labeling,
cross-lingual classification, and question answering.

###  2.3 ALBERT

Lan et al. (2019) present ALBERT, A Lite Bert for self-supervised learning of
language representations. ALBERT is a Transformer-based neural network
architecture (similar to BERT, RoBERTa and XLM-R) with two parameter reduction
techniques proposed to increase the training speed and lower memory
consumption of the BERT model Devlin et al. (2019). The first one is the
factorization of the embedding parameterization by decomposing the vocabulary
embedding matrix into two small matrices. ALBERT uses an input-level
embeddings with a relatively-low dimensionality (e.g., $128$) and hidden-layer
embeddings with higher dimensions ($768$ as in the BERT case, or more). The
second technique is cross-layer parameter sharing (i.e., the same layer is
applied such that similar layers are used on top of one another). Implementing
these two techniques significantly reduces the number of parameters for BERT
without considerably impacting performance. For instance, ALBERTBase model has
only $12$M parameters (i.e., $89$% parameter reduction compared to the
ALBERTLarge model, which is as large as BERTBase, yet still achieves good
performance across $3$ natural language understanding benchmarks: GLUE Wang et
al. (2018), SQuAD Rajpurkar et al. (2016), and RACE Lai et al. (2017).
Moreover, Lan et al. (2019) show that ALBERTLarge can be trained $\sim 1.7$x
faster with $\sim 18$x fewer parameters.

###  2.4 T5

Raffel et al. (2019) propose Text-to-Text Transfer Transformer model (T5). The
main idea of T5 is to treat every text-based language task as a “text-to-text”
problem. That is, T5 takes text format as input and produces new text format
as output. T5 applies multi-task learning with the same model, objective,
training procedure, and decoding process to several English NLP tasks. Tasks
included are question answering, document summarization, machine translation,
and sentiment classification. T5 is essentially an encoder-decoder Transformer
Vaswani et al. (2017) with some architectural modifications including applying
a normalization layer before a sub-block and adding a pre-norm (i.e., initial
input to the sub-block output). The encoder and the decoder are similar in
configuration and size to a BERTBase Devlin et al. (2019). T5 is trained on
the “Colossal Clean Crawled Corpus” (or C4 for short), which is a $\sim 750$
GB clean and natural English text extracted from CommonCrawl web data. T5
achieves SOTA on many benchmarks.

###  2.5 Arabic LMs

A few Arabic LMs have been developed. The most notable among these is AraBERT
Antoun et al. (2020). AraBERT is trained with the same BERT architecture
Devlin et al. (2019) and uses the same BERTBase configuration. The second
version of AraBERT, AraBERTv1, employs the Farasa tool Abdelali et al. (2016)
to segment input text before training. AraBERT is trained on $23$GB of Arabic
text, comprising more than $70$M sentences and $3$B words. To train AraBERT,
Antoun et al. (2020) use Arabic Wikipedia, the Open Source International
dataset (OSIAN) Zeroual et al. (2019) ($3.5$M news articles from $24$ Arab
countries), and $1.5$B words Corpus from El-Khair (2016) ($5$M articles
extracted from $10$ news sources). Antoun et al. (2020) evaluate AraBERT on
$3$ Arabic downstream tasks. These are (1) sentiment analysis from $6$
different datasets: HARD Elnagar et al. (2018), ASTD Nabil et al. (2015),
ArsenTD-Lev Baly et al. (2019), LABR Aly and Atiya (2013), and ArSaS Elmadany
et al. (2018). (2) NER, with the ANERcorp Columbia University (2016), and (3)
Arabic QA, on Arabic-SQuAD Mozannar et al. (2019) and ARCD Mozannar et al.
(2019) datasets. We now introduce our models.

##  3 Our Models

###  3.1 ARBERT

ARBERT is a large scale pre-training masked language model focused on Modern
Standard Arabic (MSA). To train ARBERT, we use the same architecture as
BERTBase: $12$ attention layers, each has $12$ attention heads and $768$
hidden dimensions, a vocabulary of $100$K WordPieces, making $\sim 163$M
parameters. We now describe ARBERT’s pre-train dataset, vocabulary, and pre-
training setup.

####  3.1.1 Training Data

We train ARBERT on a collection of Arabic datasets comprising $61$GB of text.
We list each of these datasets next, and provide data size and the number of
tokens per dataset in Table 1.

Source | Size |  | #Tokens
---|---|---|---
Books (Hindawi) |  $650$MB |  |  $72.5$M
El-Khair |  $16$GB |  |  $1.6$B
Gigawords |  $10$GB |  |  $1.1$B
OSIAN |  $2.8$GB |  |  $292.6$M
OSCAR-MSA |  $31$GB |  |  $3.4$B
OSCAR-Egyptian |  $32$MB |  |  $3.8$M
Wiki |  $1.4$GB |  |  $156.5$M
Total |  $\bf 61$GB |  |  $\bf 6.5$B

Table 1: ARBERT ’s pre-train resources.

  * •

Books (Hindawi). We collect and pre-process $1,800$ Arabic books from the
public Arabic bookstore Hindawi.333https://www.hindawi.org/books/

  * •

El-Khair. This is a $5$M news articles dataset from $10$ major news sources
covering $8$ Arab countries from El-Khair (2016).

  * •

Gigaword. We use Arabic Gigaword 5th Edition (LDC2011T11) from the Linguistic
Data Consortium (LDC).444 https://catalog.ldc.upenn.edu/LDC2009T30 The dataset
is a comprehensive archive of newswire text from multiple Arabic news sources.

  * •

OSCAR. This is the MSA and Egyptian Arabic portion of the Open Super-large
Crawled Aalanach coRpus Suárez et al. (2019),555https://oscar-corpus.com/ a
huge multilingual subset from Common Crawl666https://commoncrawl.org obtained
using language classification and filtering.

  * •

OSIAN. The Open Source International Arabic News Corpus (OSIAN) Zeroual et al.
(2019) consists of $3.5$ million articles from $31$ news sources in $24$ Arab
countries.

  * •

Wikipedia Arabic. We download and use the December $2019$ dump of Arabic
Wikipedia. We use WikiExtractor 777https://github.com/attardi/wikiextractor to
extract articles and remove markup from the dumb.

####  3.1.2 Training Procedure

Pre-processing. To prepare the raw data for pre-training, we perform light
pre-processing. This helps retain a faithful representation of the naturally
occurring text. We only remove diacritics and replace URLs, user mentions, and
hashtags that may exist in any of the collections with the generic string
tokens URL, USER, and HASHTAG. We do not perform any further pre-processing of
the data before splitting the text off to wordPieces (which we explain
next).888For example, we do not normalize variants of Arabic letters such as
“Alif”, “Yeh”, “TaMarbota” to generic forms as is usually undertaken in order
to reduce sparsity. We also are not concerned with sparsity since our datasets
are large. In addition, this normalization could increases ambiguity and so we
avoid it.

Multilingual models such as mBERT and XLM-R have $5$K (out of $110$K) and
$14$K (out of $250$K) Arabic subwords, respectively, in their vocabularies.
AraBERT employs a larger vocabulary of $60$K (out of $64$K)999The additional
$4$K vocabulary bin is reserved to allow for additional wordPieces if needed..
For ARBERT , we use a larger vocabulary of $100$K WordPieces Schuster and
Nakajima (2012). For subword tokenization, we use the WordPiece tokenizer Wu
et al. (2016) provided by Devlin et al. (2019).

Pre-training. To pre-train ARBERT, we follow Devlin et al. (2019)’s pre-
training setup. To generate each training input sequence, we use the whole
word masking, where $15\%$ of the $N$ input tokens are selected for
replacement. These tokens are replaced $80\%$ of the time with the [MASK]
token, $10\%$ with a random token, and $10\%$ with the original token. We use
the original implementation of BERT in the TensorFlow framework. 101010
https://github.com/google-research/bert. As mentioned, we use the same network
architecture as BERTBase: $12$ layers, $768$ hidden units, $12$ heads, for a
total of $110$M parameters. We train ARBERT with a batch size of $256$
sequences and a maximum sequence length of $128$ tokens ($256$ sequences
$\times$ $128$ tokens = $32,768$ tokens/batch) for $8$M steps, which is
approximately $42$ epochs over the $6.5$B tokens. We train the model on $1$
Google Cloud TPU with $8$ cores (v$2.8$) from TensorFlow Research Cloud (TFRC)
111111https://www.tensorflow.org/tfrc. Training took $16$ days, for $42$
epochs over all the tokens. Table 2 shows a comparison of ARBERT with mBERT,
XLM-R, AraBERT, and MARBERT (see next section) in terms of data sources and
size, vocabulary size, and model parameter size.

###  3.2 MARBERT

Models | Training Data | Vocabulary | Configuration
---|---|---|---
Source |  Tokens (ar/all) | Tokanization |  Size (ar/all) | Cased | Arch. | #Param.
mBERT | Wikipedia |  $153$M/$1.5$B | WordPiece |  $5$K/$110$K | yes | base |  $110$M
XLM-RB | CommonCrawl |  $2.9$B/$295$B | SentencePiece |  $14$K/$250$K | yes | base |  $270$M
XLM-RL | CommonCrawl |  $2.9$B/$295$B | SentencePiece |  $14$K/$250$K | yes | large |  $550$M
AraBERT | Several (3 sources) |  $2.5$B/$2.5$B | SentencePiece |  $60$K/$64$K | no | base |  $135$M
ARBERT | Several (6 sources) |  $6.2$B/$6.2$B | WordPiece |  $100$K/$100$K | no | base |  $163$M
MARBERT | Arabic Twitter |  $15.6$B/$15.6$B | WordPiece |  $100$K/$100$K | no | base |  $163$M

Table 2: Configuration comparisons for AraBERT Antoun et al. (2020), mBERT
Devlin et al. (2019), XLM-R Conneau et al. (2020), ARBERT, and MARBERT.

Arabic has multiple varieties. Many of these varieties are understudied due to
rarity of resources. Multilingual models such as mBERT and XLM-R are trained
almost exclusively on MSA data, which is also the case for AraBERT and
ARBERT121212We note that, as explained earlier, ARBERT is trained on a small
portion of Egyptian Arabic of text ($32$MB, making up $3.8$M tokens).. As
such, these models are not best suited for downstream tasks involving
dialectal Arabic. To treat this issue, we use a large Twitter dataset to pre-
train a new model, MARBERT , from scratch. For this new model, we also use the
BERTBase architecture listed in Section 3.1. We now describe MARBERT ’s pre-
training dataset, vocabulary, and settings.

####  3.2.1 Training data

To train MARBERT, we randomly sample $1$B Arabic tweets from a large in-house
dataset of about $6$B tweets. We only include tweets with at least $3$ Arabic
words, based on character string matching, regardless whether the tweet has
non-Arabic string or not. That is, we do not remove non-Arabic so long as the
tweet meets the $3$ Arabic word criterion. The dataset makes up $128$GB of
text ($15.6$B tokens).

####  3.2.2 Training Procedure

Pre-processing. Again, we remove diacritics and replace URLs, user mentions,
and hashtags with the generic string tokens URL, USER, and HASHTAG.

Pre-training. We use the same network architecture as BERTBase, but without
the next sentence prediction (NSP) objective since tweets are short. NSP were
also shown not to be crucial for model performance Conneau et al. (2019). We
use the same vocabulary size ($100$K wordPiece) as ARBERT. We train MARBERT
for $36$ epochs with a batch size of $256$ and a maximum sequence length of
$128$. Training took $40$ days on $8$ Google Cloud TPUs. We now present a
comparison between our models and popular multilingual models as well as
AraBERT.

###  3.3 Model Comparison

ARBERT and MARBERT compare to multilingual MLMs and AraBERT across $3$ primary
dimensions: (1) training data size, (2) vocabulary size, and (3) language
varieties. Table 2 shows a comparison between all theses models across these
$3$ dimensions.

Data size. mBERT Devlin et al. (2019) and XLM-R Conneau et al. (2020) use a
small Arabic text collection from Wikipedia ($153$M tokens) and CommonCrawl
($2.9$B tokens), respectively. AraBERT Antoun et al. (2020) is BERTBase
pretrained model for Arabic. As explained, Antoun et al. (2020) use two
newswire datasets to train AraBERT (Zeroual et al. (2019) and El-Khair (2016))
comprising more than $70$M sentences, corresponding to $24$GB of text ($2.5$B
tokens). Our ARBERT and MARBERT are trained on $61$GB of Arabic text ($6.2$B
tokens) and $128$GB of Arabic tweets ($15.6$B tokens), respectively.

Arabic Vocabulary size. Each of ARBERT and MARBERTis trained with $100$K
Arabic subwords. The mBERT, XLM-R, and AraBERT models have $5$K, $14$K, and
$60$K subwords, respectively

Language Variety. Devlin et al. (2019) train mBERT with Wikipedia Arabic data,
which is MSA. XLM-R Conneau et al. (2020) is trained on Common Crawl data,
hence conceivably comprises some dialectal Arabic data. However, it is not
clear how much dialects are part of the XLM-R Arabic data. The datasets
AraBERT is trained on are also only MSA. Our ARBERT model is trained on a
collection of large MSA datasets and small amount of Egyptian data. The data
we train MARBERTon is from Twitter, and comprises both MSA and diverse
dialects. We now introduce our performance on downstream tasks.

##  4 Downstream Tasks

We evaluate our models on a wide range of natural language understanding
tasks. To facilitate presentation, we thematically organize these tasks into
the following $5$ categories: (1) sentiment analysis, (2) social meaning
(e.g., age and gender, dangerous and hateful speech, emotion, irony, sarcasm),
(3) topic classification, (4) dialect identification, and (5) named entity
recognition (NER).In each case, we fine-tune our models and all other models
we compare to on the training data for each task, and evaluate on both
development and test data. We typically use the exact data splits provided by
original author(s) of each dataset. Whenever no clear splits are available, or
in cases where expensive cross-validation was used by original authors, we
divide the data following a standard $80\%$ TRAIN, $10\%$ DEV and $10\%$ TEST
split. Similar to our pre-processing for model pre-training, we slightly
normalize the text as remove Arabic diacritics, replace hyperlinks and user
mentions with URL and USER, respectively. For Twitter hashtags, we remove the
hash sign and replace any underscore with a space. For all tasks, we fine-tune
mBERT, XLM-RBase, and AraBERT. Although XLM-RLarge is much bigger (in terms of
architecture) than our models, we also compare to it. For all models,
including ARBERT and MARBERT, we use the same settings. We typically identify
the best checkpoint for each model on the development set, and report its
performance on both development and test data.

Baselines. Whenever applicable, we consider SOTA results for each task as our
baseline. Otherwise, we consider AraBERT results as a strong baseline. We now
introduce the different tasks.

###  4.1 Sentiment Analysis

We evaluate our models on all publicly available sentiment analysis datasets
in addition to datasets we could acquire directly from authors. In total, we
fine-tune our models on the following $17$ MSA and DA sentiment datasets:

####  4.1.1 Datasets

Dataset (#classes) | Classes | TRAIN | DEV | TEST
---|---|---|---|---
AJGT (2) | {neg, pos} |  $1.4$K | - | $361$
AraNETSent (2) | {neg, pos} |  $100.5$K |  $14.3$K |  $11.8$K
AraSenTi-Tweet (3) | {neg, neut, pos} |  $11.1$K |  $1.4$K |  $1.4$K
ArSarcasmSent (3) | {neg, neut, pos} |  $8.4$K | - |  $2.1$K
ArSAS (3) | {neg, neut, pos} |  $24.8$K | - |  $3.7$K
ArSenD-LEV (5) | {neg, neut, pos, neg+, pos+} |  $3.2$K | - | $801$
ASTD (3) | {neg, neut, pos} |  $24.8$K | - | $664$
ASTD-B (2) | {neg, pos} |  $1$K | - | $267$
AWATIF (4) | {neg, neut, obj, pos } |  $2.3$K | $288$ | $284$
BBN (3) | {neg, neut, pos} | $960$ | $125$ | $116$
HARD (2) | {neg, pos} |  $75$K | - |  $18.7$K
LABR (2) | {neg, pos} | $13.2$ | - |  $3,3$K
SAMAR (5) | {mix, neg, neut, obj, pos} | $2.5$ | $310$ | $316$
SemEval (3) | {neg, neut, pos} |  $24.8$K | - |  $6.1$K
SYTS (3) | {neg, neut, pos} | $960$ | $202$ | $199$
TwitterAbdullah (2) | {neg, pos} |  $1.6$K | $202$ | $190$
TwitterSaad (2) | {neg, pos} |  $47$K |  $5.8$K | $5.8$

Table 3: Sentiment analysis datasets. neg+: “very negative”; pos+: “very
positive”.

  * •

AJGT. The Arabic Jordanian General Tweets (AJGT) dataset Alomari et al. (2017)
involves MSA and Jordanian tweets, with a balanced split of $900$ positive
posts and $900$ negative posts (total=$1,800$).

  * •

AraNETSent. Abdul-Mageed et al. (2020b) collect $15$ datasets in both MSA and
dialects from Abdul-Mageed and Diab (2012) (AWATIF), Abdul-Mageed et al.
(2014) (SAMAR), Abdulla et al. (2013); Nabil et al. (2015); Kiritchenko et al.
(2016); Aly and Atiya (2013); Salameh et al. (2015); Rosenthal et al. (2017);
Alomari et al. (2017); Mohammad et al. (2018), and Baly et al. (2019). These
datasets involve varying numbers of sentiment categories: binary (negative and
positive), 3-way (negative, neutral, and positive). Some of the datasets also
have a subjective language label. Abdul-Mageed et al. (2020b) combine the
datasets for binary sentiment classification by reducing the different tags
into the binary case (i.e., negative and positive) and removing data from all
other labels, acquiring $126,766$ samples, which they split into 80%
($100,599$), 10% ($14,344$) and 10% ($11,823$) for training, development, and
test respectively.

  * •

AraSenTi-Tweet. This is a collection of MSA and Saudi Arabic tweets by Al-
Twairesh et al. (2017). The $17,573$ dataset is manually annotated. The
distribution of labels is as follows: $4,957$ positive, $6,155$ negative,
$4,639$ neutral, and $1,822$ mixed sentences.

  * •

ArSarcasmSent Farha and Magdy (2020) introduce ArSarcasm, a sarcasm dataset
labeled with sentiment tags. ArSarcasm has $10,547$ tweets, of which $2,472$
taken from ASTD Nabil et al. (2015) and $8,075$ from SemEval-2017 Task 4
Rosenthal et al. (2017). Farha and Magdy (2020) evaluate on the $3$ sentiment
labels negative, neutral, and positive.

  * •

ArSAS.The Arabic Speech Act and Sentiment (ArSAS) corpus Elmadany et al.
(2018) consists of $21,064$ tweets annotated with four labels: positive,
negative, neutral, and mixed.

  * •

ArSenD-Lev. The Arabic Sentiment Twitter Dataset for LEVantine dialect
(ArSenD-Lev) Baly et al. (2019) has $4,000$ tweets retrieved from the Levant
region (i.e., Jordan, Lebanon, Palestine, and Syria). Sentiment is labeled on
a 5-point scale, with $835$ positive, $397$ very positive, $885$ neutral,
$1,253$ negative, and $630$ very negative.

  * •

ASTD. This is a collection of $10,006$ Egyptian tweets annotated with the
labels positive (n=$799$), negative (n=$1,684$), objective (n=$832$), and
neutral (n=$6,691$) by Nabil et al. (2015).

  * •

AWATIF. A multigenre MSA dataset Abdul-Mageed and Diab (2012) consisting of
$2,855$ sentences of newswire stories from Part 1, V3 of the Penn Arabic
TreeBank,131313https://catalog.ldc.upenn.edu/LDC2005T20 $5,342$ sentences from
$30$ Wikipedia talk pages, and $2,532$ threaded conversations from $7$ web
forums.

  * •

BBNS & SYTS. The BBN blog posts Sentiment (BBNS) and Syria Tweets Sentiment
(SYTS) are introduced by Salameh et al. (2015). BBNS comprises $1,200$
Levantine dialectal sentences extracted from the BBN Arabic-Dialect/English
Parallel Text.141414 https://catalog.ldc.upenn.edu/LDC2012T09 SYTS consist of
$2,000$ Levantine Arabic tweets. Both BBNS and SYTS are manually annotated
with tags from the set {negative, positive, neutral}.

  * •

CAMelSent. Obeid et al. (2020) merge training and development data from ArSAS
Elmadany et al. (2018), ASTD Nabil et al. (2015), SemEval Rosenthal et al.
(2017), and ArSenTD Al-Twairesh et al. (2017) to create a new training dataset
($\sim 24$K). For evaluation, they report performance on the independent test
set from each of these sources.

  * •

HARD. The Hotel Arabic Reviews Dataset (HARD) Elnagar et al. (2018) consists
of $93,700$ hotel reviews, each with a rating of $1$-$5$ stars. The reviews
are written in MSA and DA. Reviews with $1$ and $2$ stars are treated as
negative and those with $4$ and $5$ stars as positive. Reviews with $3$ stars
reviews are removed from the dataset. Overall, there are $46.8$K for each of
the positive and negative classes.

  * •

LABR. The Large Arabic Book Review Corpus Aly and Atiya (2013) has $63,257$
book reviews from Goodreads, the book readers social
network.151515www.goodreads.com The reviews are rated with a $1$-$5$ stars
scale. Posts with $1$-$2$ stars are treated as negative, those with $3$ starts
as neutral, and those with $3$-$5$ as positive.

  * •

TwitterAbdullah.161616For ease of reference, we assign a name to this and
other unnamed datasets. This is a dataset of $2,000$ MSA and Jordanian Arabic
tweets manually labeled by Abdulla et al. (2013). The dataset is balanced,
with $1,000$ positive and $1,000$ negative tweets.

  * •

TwitterSaad. This dataset is released by Moatez Saad in 2019 and is available
on Kaggle.171717www.kaggle.com/mksaad/arabic-sentiment-twitter-corpus The
dataset is collected using an emoji lexicon as a proxy for sentiment. In
total, it has $29.8$K positive and $28.7$K negative tweets.

  * •

SemEval-2017. The SemEval-2017 sentiment analysis in Arabic Twitter task
Rosenthal et al. (2017) provides data for $3$ sentiment analysis sub-tasks.
These included on $2$ classes (negative and positive) with $9,500$ tweets, $3$
classes (negative, positive, and neutral) with $3,400$ tweets, and $5$ classes
(strongly negative, negative, neutral, positive and strongly positive) with
$9,450$ tweets.

####  4.1.2 Baselines

For sentiment analysis, we compare our results to the following STOA:

  * •

Antoun et al. (2020). The authors fine-tune two versions of AraBERT:
AraBERTv0.1, which employs no Arabic segmentation nor pre-possessing, and
AraBERTv1, which is based on segmentation nor pre-possessing. They fine-tune
these two versions on $5$ sentiment datasets. These are HARD Elnagar et al.
(2018), the balanced data for ASTD (which we will refer to as ASTD-B) Nabil et
al. (2015), ArSenTD-Lev Baly et al. (2019), AJGT Alomari et al. (2017), and
the unbalanced positive and negative classes for LABR Aly and Atiya (2013).
Antoun et al. (2020) split the data into 80/20 for training/test, respectively
and report results in accuracy using the best epoch identified on test data.
For a valid comparison, we use the same data splits and evaluation set up as
Antoun et al. (2020). Their best results are obtained by fine-tuning
AraBERTv1.

  * •

Obeid et al. (2020). The authors fine-tune mBERT and AraBERT on the merged
CAMelsent datasets and compared their results with Mazajak Farha and Magdy
(2019). Obeid et al. (2020) use $F_{1}$PN score for evaluation, which was
defined by SemEval-2017 and also used by Majazak Farha and Magdy (2019).
$F_{1}$PN is the macro $F_{1}$ score over the positive and negative classes
only, while neglecting the neutral class.

  * •

Abdul-Mageed et al. (2020b). The authors fine-tune mBERT on the AraNETSent
data and report results in $F_{1}$ score on test data.

####  4.1.3 Sentiment Analysis Results

To facilitate comparison to previous works with the appropriate evaluation
metric, we split our results into two tables: We show results in accuracy and
F${}_{1}^{PN}$ in Table 5 and accuracy and F1 in Table 4. We typically bold
results acquired with the best model in the metric adopted in previous work on
each dataset. As Table 5 shows, our models achieve best results in $10$ out of
$12$ datasets. More specifically, MARBERT achieves best results on $9$
datasets, ARBERT is best on one dataset, and XLM-RLarge is best on $2$
datasets. We also note that XLM-RLarge outperforms AraBERT on $10$ datasets
out of the $12$ datasets. Our models achieve new SOTA on all the datasets,
outperforming cases where there is previous published SOTA (as by to Obeid et
al. (2020) and Abdul-Mageed et al. (2020b), both marked in Table 5). On
average, our models are $2.8$ F1 better than any next best model on the $12$
datasets.

Of the $5$ datasets in Table 4, MARBERT and ARBERT achieve best results on $4$
datasets. MARBERT, again, compares favorably to ARBERT. In all except one case
(ArsenTF-LEV), ARBERT is better than XLM-RLarge. Across all datasets, our
models are consistently better than our own fine-tuned AraBERT baseline and
AraBERT results published by Antoun et al. (2020), which we consider
SOTA.181818Our own fine-tuned AraBERT results are all lower than those
reported in Antoun et al. (2020) in spite of us using a wide set of hyper-
parameters including those described in Antoun et al. (2020). However, as
noted, we also compare to AraBERT results published by Antoun et al. (2020).

To summarize, compared to SOTA on the $5$ datasets HARD, ArsenTD-LEV, LABER,
and ASTD-B reported by Antoun et al. (2020), ARBERT and MARBERT achieve new
SOTA (exceeding AraBERT, mBERT and XLM-R models). Compared to SOTA reported by
Obeid et al. (2020) on the $3$ datasets ArSAS CAMel, ASTD CAMel, and SemEval
CAMel datasets, our models achieve new SOTA (on all $3$ datasets) with an
average improvement of $4\%$ F${}_{1}^{PN}$. Compared to Abdul-Mageed et al.
(2020b) on AraNETSent data, both of our models outperform AraNETSent but our
new SOTA is achieved by XLM-RLarge (which is $\sim 1\%$ higher than MARBERT on
AraNETSent data).

For all other datasets ($n=8$), we consider published AraBERT results in
Antoun et al. (2020) a strong baseline (calling it SOTA, although we find that
XLM-RLarge outperforms it in most cases). Both of our models outperform
AraBERT on all datasets except on AraSenTi-Tweet (where both XLM-RLarge and
AraBERT obtain better results than our models). Overall, experiments on the
the sentiment datasets show our fine-tuned ARBERT and MARBERT models are
better in $14$ out of the $17$ datasets ($\%=\sim 82.4)$. These results also
clearly show that MARBERT is superior to ARBERT, outperforming it on $16$ out
of the $17$ datasets.

Data (#classes) SOTA LM TEST Acc. F1 AJGT (2) 93.8 mBERT 86.67 86.59 XLM-RB
89.44 89.43 XLM-RL 91.94 91.87 AraBERT 92.22 92.19 3-6 ARBERT 94.44 94.43
MARBERT 96.11 96.10 HARD (2) 96.2 mBERT 95.54 95.54 XLM-RB 95.74 95.74 XLM-RL
95.96 95.96 AraBERT 95.89 95.89 3-6 ARBERT 96.12 96.12 MARBERT 96.17 96.17
ArsenTD-LEV (5) 59.4 mBERT 50.50 47.98 XLM-RB 55.25 49.89 XLM-RL 62.00 59.99
AraBERT 56.13 54.30 3-6 ARBERT 61.38 58.06 MARBERT 60.38 59.04 LABR (2) 86.7
mBERT 91.20 81.98 XLM-RB 91.23 82.70 XLM-RL 92.20 83.82 AraBERT 91.97 84.89
3-6 ARBERT 92.51 84.98 MARBERT 92.49 85.20 ASTD-B (2) 92.6 mBERT 79.32 79.30
XLM-RB 87.59 87.55 XLM-RL 77.44 77.41 AraBERT 83.08 83.07 3-6 ARBERT 93.23
93.21 MARBERT 96.24 96.24 SOTA are results reported in Antoun et al. (2020)
based on Acc.

Table 4: Performance of models on SA tasks (I).

Dataset (#classes) SOTA LM TEST Acc. F1PN ArSAS (3) 92$\star$ mBERT 89.65
87.50 XLM-RB 91.76 90.00 XLM-RL 92.85 91.50 AraBERT 91.76 91.00 3-7 ARBERT
93.12 92.00 MARBERT 93.85 93.00 ASTD (3) 73$\star$ mBERT 59.54 67.00 XLM-RB
63.96 60.67 XLM-RL 71.38 67.67 AraBERT 63.78 72.00 3-7 ARBERT 68.73 76.50
MARBERT 71.02 78.00 SemEval (3) 69$\star$ mBERT 58.02 57.00 XLM-RB 64.62 64.00
XLM-RL 68.01 67.00 AraBERT 62.37 62.00 3-7 ARBERT 68.08 69.00 MARBERT 69.35
71.00 AraNETSent (2) 76.2$\star\star$ mBERT 84.80 84.00 XLM-RB 92.40 91.93
XLM-RL 93.50 93.11 AraBERT 87.14 86.44 3-7 ARBERT 89.55 88.95 MARBERT 92.77
92.31 ArSarcasmSent (3) - mBERT 70.43 60.50 XLM-RB 72.89 63.50 XLM-RL 75.92
70.00 AraBERT 70.71 63.50 3-7 ARBERT 75.12 68.00 MARBERT 77.35 71.50 AraSenTi
(3) - mBERT 89.93 89.50 XLM-RB 92.40 92.00 XLM-RL 93.34 93.50 AraBERT 90.95
91.00 3-7 ARBERT 90.80 90.00 MARBERT 90.59 90.00 BBN (3) - mBERT 52.17 55.50
XLM-RB 66.96 69.50 XLM-RL 47.83 46.50 AraBERT 66.96 70.00 3-7 ARBERT 71.30
76.50 MARBERT 73.91 79.00 SYTS (3) - mBERT 73.30 67.00 XLM-RB 78.01 78.00 XLM-
RL 68.59 40.50 AraBERT 78.53 75.50 3-7 ARBERT 79.06 79.00 MARBERT 78.53 76.50
TwitterSaad (2) - mBERT 79.08 79.00 XLM-RB 95.19 95.00 XLM-RL 95.29 95.00
AraBERT 80.90 81.00 3-7 ARBERT 89.85 90.00 MARBERT 95.72 96.00 SAMAR (5) -
mBERT 42.54 22.50 XLM-RB 54.92 54.00 XLM-RL 56.83 57.00 AraBERT 49.21 36.50
3-7 ARBERT 48.57 43.50 MARBERT 57.46 55.50 AWATIF (4) - mBERT 66.16 60.50 XLM-
RB 67.68 63.50 XLM-RL 71.10 68.50 AraBERT 71.10 66.50 3-7 ARBERT 73.00 71.50
MARBERT 75.67 72.50 TwitterAbdullah (2) - mBERT 82.01 81.50 XLM-RB 91.01 91.00
XLM-RL 92.06 92.00 AraBERT 89.42 89.50 3-7 ARBERT 91.53 91.50 MARBERT 94.71
95.00 $\star$ Obeid et al. (2020), $\star\star$ Abdul-Mageed et al. (2020b)
based on F1, otherwise we consider AraBERT Antoun et al. (2020) as SOTA.

Table 5: Performance of models on SA tasks (II).

###  4.2 Social Meaning Tasks

We evaluate on a host of tasks to which we collectively refer to as social
meaning. These are age and gender detection; dangerous, hateful, and offensive
speech detection; emotion detection; irony detection; and sarcasm detection.
We now describe datasets we use for each of these tasks.

####  4.2.1 Tasks & Datasets

Table 6: Social Meaning datasets.

Task | Dataset (#classes) | Classes | TRAIN | DEV | TEST
---|---|---|---|---|---
Age | Arap-Tweet (3) | { $\leq$ $24$ yrs, $25-34$ yrs, $\geq$ $35$ yrs } |  $1.3$M |  $160.7$K |  $160.7$K
Dangerous | AraDang (2) | {dangerous, not-dangerous} |  $3.5$K | $616$ | $664$
Emotion | AraNETEmo (8) | {ang, anticip, disg, fear, joy, sad, surp, trust} |  $190$K | $911$ | $942$
Gender | Arap-Tweet (2) | {female, male} |  $1.3$M |  $160.7$K |  $160.7$K
Hate Speech | HS@OSACT (2) | {hate, not-hate} |  $10$K |  $1$K |  $2$K
Irony | FIRE2019 (2) | {irony, not-irony} |  $3.6$K | - | $404$
Offensive | OFF@OSACT (2) | {offensive, not-offensive} |  $10$K |  $1$K |  $2$K
Sarcasm | AraSarcasm (2) | {false, true} |  $8.4$K | - |  $2.1$K

Age and Gender. For both age and gender, we use the Arap-Tweet dataset
Zaghouani and Charfi (2018). This dataset covers $17$ different countries from
$11$ Arab regions. For each region, authors crawl data from $100$ Twitter
users based on an initial list of seed words characteristic of each region.
Zaghouani and Charfi (2018) assign gender from the set {male, female} and age
group labels from the set {under-25, 25-to-34, above-35}. We follow the 80%
($1,285,795$), 10% ($160,724$), and 10% ($160,725$) data split of AraNet
Abdul-Mageed et al. (2020b).

Dangerous Speech. We use the dangerous speech AraDang dataset from Alshehri et
al. (2020a). This data is extracted from Arabic Twitter and consists of
$4,445$ manually annotated tweets. Of these, $3,225$ are labelled as safe and
$1,220$ as dangerous.

Offensive Language and Hate Speech. We use the dataset released in the shared
task of offensive speech by Mubarak et al.
(2020).191919http://edinburghnlp.inf.ed.ac.uk/workshops/OSACT4/ The shared
task is divided into two sub-tasks: sub-task A: detecting if a tweet is
offensive or not-offensive, and sub-task A: detecting if a tweet is hate-
speech or not-hate-speech. The offensive content dataset consists of $10,000$
manually annotated tweets, of which $1,991$ are offensive and rest (n=$8,009$)
is not-offensive. The same tweets are also labeled with hate speech tags
($9,494$ hate-speech and $8,009$ not-hate-speech).

Emotion. For this task we use the AraNeTemo dataset from Abdul-Mageed et al.
(2020b). To create this data, the authors merge the two datasets LAMA-DINA and
LAMA-DIST from Alhuzali et al. (2018). AraNeTemo consist of $192$K tweets
(split into $190$K TRAIN, $911$ DEV, and $942$ TEST), and labeled with the
Plutchik $8$ primary emotions classes in the set {anger, anticipation,
disgust, fear, joy, sadness, surprise, and trust}.

Irony. We use the irony identification dataset for Arabic tweets released by
IDAT@FIRE2019 shared task Ghanem et al. (2019). This dataset contains $5,030$
tweets and involves both MSA and Egyptian, Gulf, and Levantine dialects.
IDAT@FIRE2019 Ghanem et al. (2019) is set up as a binary classification task
with ironic and non-ironic labels. A total of $4,024$ and $1,006$ tweets were
released by organizers as training and test data. In our experiments, we use
the same split of Abdul-Mageed et al. (2020b), where they split the $4,024$
released training data into 90% TRAIN (n$=3,621$ tweets; ironic$=1,882$ and
non-ironic$=1,739$) and 10% DEV (n$=403$ tweets; ironic$=209$ and non-
ironic$=194$).

Sarcasm. We use the ArSarcasm dataset developed by Farha and Magdy (2020).
ArSarcasm contains $10,547$ tweets, $2,472$ of these are from ASTD Nabil et
al. (2015) while the rest (n=$8,075$) are from SemEval2017 Rosenthal et al.
(2017). The tweets are labeled with sarcasm and not-sarcasm tags.

####  4.2.2 Baselines

We compare our results with the STOA on each task, as follows:

  * •

Age and Gender. We compare to AraNET Abdul-Mageed et al. (2020b) age and
gender models, trained by fine-tuning mBERT. The authors report $51.42$ and
$65.30$ $F_{1}$ on age and gender, respectively.

  * •

Dangerous Speech. We compare to Alshehri et al. (2020b), who report best
results with an mBERT model fined-tuned on emotion data on $8$ emotion classes
from Twitter. They report $59.60$ F1 on test data.

  * •

Emotion. We compare to Abdul-Mageed et al. (2020b), who acquire $60.32$
$F_{1}$ on test with a fine-tuned mBERT.

  * •

Hate Speech. The best results on the offensive and hate speech shared task
Mubarak et al. (2020) are reported by Husain (2020). The authors employ heavy
feature engineering with SVMs exploiting emoticons, emojis, and hashtags. They
also convert dialectal tokens to MSA and perform word clustering. They report
$95$ F1 score. Since our goal is to advance methods exploiting language models
without need for heavy hand-crafted features, we do not compare to Husain
(2020). Rather, we compare our models to Djandji et al. (2020) who rank
$2^{n}d$ in the shared task since they use AraBERT in a multi-task setting.
They report $83.41$ F1 score on test data.

  * •

Irony. We compare to Zhang and Abdul-Mageed (2019a) who fine-tune mBERT on the
irony task. The authors use multi-task learning, where they employ an
auxiliary author profiling task along with the main irony task. They report
$82.4$ F1 score on the test set.

  * •

Offensive Language. The best results on the offensive shared task Mubarak et
al. (2020) are reported by Hassan et al. (2020). They propose an ensemble of
SVMs, CNN-BiLSTM, and mBERT with majority voting. They report $90.51$ F1. We
compare to their results.

  * •

Sarcasm. We compare to Farha and Magdy (2020) who train a BiLSTM model using
the AraSarcasm dataset, reporting $46.00$ F1 score.

####  4.2.3 Social Meaning Results

Results of social meaning tasks are reported in Table 7. As the table shows,
our models acquire best results on all $8$ tasks. MARBERT acquires best
performance on $7$ tasks, while ARBERT is best on on one task
(irony@FIRE2019). We also note that XLM-R outperforms AraBERT on $6$ out of
the $8$ tasks. On average, our models are $12.5$ F1 better than all previous
SOTA.

TEST Task Dataset (#classes) SOTA LM Acc. F1 mBERT 56.19 56.35 XLM-RB 59.73
59.73 XLM-RL 53.39 53.60 AraBERT 57.55 57.72 5-10 ARBERT 58.82 58.95 Age
ArapTweet (3) 51.42$\ddagger\ddagger$ MARBERT 62.10 62.27 mBERT 80.09 62.66
XLM-RB 80.84 62.76 XLM-RL 80.84 65.01 AraBERT 81.30 64.37 5-10 ARBERT 81.00
63.21 Dangerous AraDange (2) 59.60$\dagger$ MARBERT 81.00 67.53 mBERT 67.41
65.79 XLM-RB 72.20 70.67 XLM-RL 75.72 74.89 AraBERT 67.20 65.68 5-10 ARBERT
69.86 67.73 Emotion AraNETEmo (8) 60.32$\ddagger\ddagger$ MARBERT 77.42 75.83
mBERT 68.59 68.06 XLM-RB 71.15 71.00 XLM-RL 71.46 71.14 AraBERT 68.58 67.75
5-10 ARBERT 70.18 69.86 Gender Arab_Tweet (2) 65.3$\ddagger\ddagger$ MARBERT
72.90 72.62 mBERT 96.15 72.81 XLM-RB 95.15 71.33 XLM-RL 96.35 79.31 AraBERT
95.80 78.89 5-10 ARBERT 96.85 83.02 Hate Speech OSACT-B (2) 82.28$\star\star$
MARBERT 97.00 84.79 mBERT 81.12 80.96 XLM-RB 82.11 81.97 XLM-RL 82.86 82.52
AraBERT 83.23 83.01 5-10 ARBERT 85.84 85.59 Irony FIRE2019 (2) 60.32$\ddagger$
MARBERT 85.59 85.33 mBERT 90.70 84.25 XLM-RB 91.00 85.26 XLM-RL 92.80 88.28
AraBERT 91.80 86.57 5-10 ARBERT 93.85 90.38 Offensive OSACT-A (2) 90.51$\star$
MARBERT 95.20 92.41 mBERT 81.80 68.20 XLM-RB 86.26 66.76 XLM-RL 86.68 69.23
AraBERT 85.31 72.23 5-10 ARBERT 86.73 75.04 Sarcasm AraSarcasm (2)
46.00$\dagger\dagger$ MARBERT 87.63 76.30 Dataset has only TRAIN and TEST
splits or SOTA system only reported on TEST. $\star$ Hassan et al. (2020),
$\star\star$ Djandji et al. (2020), $\ddagger$ Zhang and Abdul-Mageed (2019a)
$\dagger$ Alshehri et al. (2020b), $\dagger\dagger$ Farha and Magdy (2020),
$\ddagger\ddagger$ Abdul-Mageed et al. (2020b) $F_{1}$ score is the evaluation
metric

Table 7: Results on social meaning tasks.

###  4.3 Topic Classification

Classifying documents by topic is a classical task. Work in this area
generally pre-dates the more recent focus on teasing apart texts by their more
nuanced semantic and pragmatic attributes. Most of the topic classification
work we identified in the Arabic NLP literature involves some focus on pre-
processing techniques, dimensionality reduction, data transformation, etc.
Altınel and Ganiz (2018). We include this task since categorizing documents by
topic is still a useful task of practical value. We now describe each of the
topic classification datasets we collected.

Dataset (#classes) | Classes | TRAIN | DEV | TEST
---|---|---|---|---
ANT (5) | {culture, economy, inter news, middle east, sports, technology} |  $25.2$K |  $3.2$K |  $3.2$K
OSAC (10) | {economy, family, health, history, law, recipes, religious, space, sports, stories} |  $18$K |  $2.2$K |  $2.2$K
Khallej (4) | {economy, inter news, local news, sports} |  $4.6$K | $570~{}$ | $570~{}$

Table 8: Topic Classification datasets.

####  4.3.1 Datasets

We fine-tune on the following datasets:

  * •

Arabic News Text. Chouigui et al. (2017) build the Arabic news text (ANT)
dataset from transcribed radio broadcasts of the Tunisian radio channel
Jawhara FM. ANT has $10$K documents divided into $5$ categories: Culture,
economy, international news, technology, and sports. ANT comes with a title
field and so we run our experiments on this dataset under $3$ settings
pertaining input on which we fine-tune the language models: Title only, text
only, and title+text.

  * •

Khaleej. Abbas et al. (2011) created the Khaleej dataset from Gulf Arabic
websites. Khaleej comprises $5,690$ documents from $4$ topical categories:
economy, international news, local news, and sports.

  * •

OSAC. Saad and Ashour (2010) collect the OSAC corpus, a total of $22,429$
articles covering $10$ categories: economy, family, health, history, law,
recipes, religion, space, sports, and stories.

####  4.3.2 Baselines

For topic classification, to the best of our knowledge, there are no published
results exploiting deep learning models. Therefore, we consider fine-tune
AraBERT a strong baseline.

###  4.4 Topic Classification Results

Table 9 shows our results on the topic classification datasets. As the table
shows, ARBERTacquires best results on both OSAC and Khaleej, and the title-
only setting of ANT. AraBERT slightly outperforms our models on the text-only
and title+text settings of ANT. We also note that, on average, MARBERT is
$\sim 1\%$ F1 less than both ARBERT and AraBERT.

TEST Data (#classes) LM Acc. F1 mBERT 98.17 96.84 XLM-RB 98.31 97.15 XLM-RL
98.84 98.20 AraBERT 98.35 97.03 5-9 ARBERT 98.44 97.50 OSAC (10) MARBERT 98.48
97.23 mBERT 94.20 92.81 XLM-RB 93.15 91.87 XLM-RL 94.73 93.56 AraBERT 95.25
93.83 5-9 ARBERT 95.43 94.53 Khallej (4) MARBERT 94.73 93.63 mBERT 85.66 84.89
XLM-RB 86.42 85.77 XLM-RL 86.90 86.72 AraBERT 88.58 88.17 5-9 ARBERT 87.25
86.87 ANTText (5) MARBERT 85.12 85.27 mBERT 79.57 78.29 XLM-RB 81.00 79.96
XLM-RL 81.82 81.25 AraBERT 82.20 81.03 5-9 ARBERT 81.54 81.70 ANTTitle (5)
MARBERT 80.87 81.19 mBERT 85.53 84.67 XLM-RB 86.64 86.21 XLM-RL 87.50 86.96
AraBERT 87.56 87.22 5-9 ARBERT 87.02 87.21 ANTTitle+Text (5) MARBERT 85.72
85.60 SOTA (F1) are results reported in Antoun et al. (2020).

Table 9: Performance of models on TC tasks.

###  4.5 Dialect Identification

Task (#classes) | Dataset | Classes | TRAIN | DEV | TEST
---|---|---|---|---|---
AOC (2) | Binary | {DA, MSA} |  $86.5$K |  $10.8$K |  $10.8$K
AOC (3) | Region | {Egypt, Gulf, Levnt} |  $35.7$K |  $4.5$K |  $4.5$K
AOC (4) | Region | {Egypt, Gulf, Levnt, MSA} |  $86.5$K |  $10.8$K |  $10.8$K
ArSarcasmDia (5) | Regoin | {Egypt, Gulf, Lev, Magreb, MSA} |  $8.4$K | - |  $2.1$K
MADAR-TL (21) | Country |  | {Algeria, Bahrain, Djibouti, Egypt, Iraq, Jordan, Kuwait,
---
Lebanon, Libya, Mauritania, Morocco, Oman, Palestine,
Qatar, KSA, Somalia, Sudan, Syria, Tunisia, UAE, and Yemen}
$193.1$K |  $26.6$K |  $44$K
NADI (21) | Country |  | {Algeria, Bahrain, Djibouti, Egypt, Iraq, Jordan, Kuwait, Lebanon,
---
Libya, Mauritania, Morocco, Oman, Palestine, Qatar, KSA,
Somalia, Sudan, Syria, Tunisia, UAE, and Yemen}
$2.1$K |  $5$K |  $5$K
QADI (18) | Country |  | {Algeria, Bahrain, Egypt, Iraq, Jordan, Kuwait, Lebanon,
---
Libya, Morocco, Oman, Palestine, Qatar, KSA,
Sudan, Syria, Tunisia, UAE, and Yemen}
$497.8$K | - |  $3.5$K

Table 10: Dialect datasets.

Available Arabic dialect identification datasets carry labels of varying
granularity, from binary (i.e., MSA-DA) Elaraby and Abdul-Mageed (2018), AOC
Zaidan and Callison-Burch (2014) to regional Farha and Magdy (2020), country-
level Bouamor et al. (2018); Abdul-Mageed et al. (2020a); Abdelali et al.
(2020), and recently province level Abdul-Mageed et al. (2020a). We introduce
these next.

####  4.5.1 Datasets

We introduce each dataset briefly here, and provide a description summary of
all datasets in Table 10.

  * •

Arabic Online Commentary (AOC). This is a repository of $3$M Arabic comments
on online news sites developed by Zaidan and Callison-Burch (2014). AOC is
labeled with MSA and the $3$ regional dialects Egyptian, Gulf, and Levantine.

  * •

ArSarcasmDia. This dataset is developed by Farha and Magdy (2020) for sarcasm
detection but also carries regional dialect labels from the set {Egyptian,
Gulf, Levantine, and Maghrebi}. ArSarcasmDia comprises $10,547$ tweets,
$2,472$ of which are taken from ASTD Nabil et al. (2015) while the rest
(n=$8,075$) come from SemEval 2017 Rosenthal et al. (2017).

  * •

MADAR. Sub-task 2 of the MADAR shared task Bouamor et al.
(2019)202020https://sites.google.com/view/madar-shared-task/home. is focused
on user-level dialect identification with country labels. The dataset is
composed of $\sim 265$K manually-curated Tweets. In this paper, we evaluate
our models on MADAR at the tweet level and compare to the shared task top
ranking system’s Zhang and Abdul-Mageed (2019b) tweet-level results.

  * •

NADI. The first Nuanced Arabic Dialect Identification shared task (NADI 2020)
Abdul-Mageed et al. (2020a)212121https://sites.google.com/view/nadi-shared-
task/home. targets country level as well as province level dialects. The
dataset is composed of $21,000$ tweets, covering $21$ Arab countries and $100$
respective provinces.

  * •

QADI. The QCRI Arabic Dialect Identification (QADI) dataset Abdelali et al.
(2020) is labeled at the country level and has $540$K tweets from $2,525$
Twitter users.

####  4.5.2 Baselines

We compare our models to the following dialect identification SOTA:

  * •

Elaraby and Abdul-Mageed (2018) report $3$ levels of classification on the AOC
dataset Zaidan and Callison-Burch (2014). These are MSA vs. DA, where they
acquire $87.23$ accuracy, regional (i.e., Egyptian, Gulf, and Levantine),
where they report $87.81$ accuracy, and MSA, Egyptian, Gulf, and Levantine
accuracy of $82.45$. Elaraby and Abdul-Mageed (2018) acquire best models with
either BiLSTMs or a simple Naive Bayes classifier.

  * •

Abdelali et al. (2020) fine-tune AraBERT on the QADI dataset. They report
$60.6$ F1.

  * •

Zhang and Abdul-Mageed (2019b) developed the top ranked system in the MADAR
sub-task $2$ shared task Bouamor et al. (2019). Although the sub-task is
focused on user level dialect identification, their system is based on tweet
level modeling where they first identify the dialect of a tweet then port the
label at the user level. They report $48.76$ accuracy and $34.87$ F1 on the
tweet-level task.

  * •

Talafha et al. (2020). We compare to the NADI sub-task 1 (country level)
winning team, Mawdoo3 AI Talafha et al. (2020). They further pre-train AraBERT
on $10$M unlabeled tweets released by the NADI 2020 organizers Abdul-Mageed et
al. (2020a) for $3$ epochs. They report $26.78$ F1 for this country level sub-
task.

  * •

El Mekki et al. (2020). NADI sub-task 2 (province level) and was won by El
Mekki et al. (2020). They clean and normalized the data by remove
diacritization, URLs, hashtags, retweets, emojis, Latin letters, and numbers.
They then augment the original data (i.e., before normalization) with the
normalized data and use a combination of word and character n-grams TF_IDF
vectors to fine-tune AraBERT. El Mekki et al. (2020) report $6.08$ F1 on this
province level sub-task.

  * •

AraBERT. For ArSarcasmDia, where there is no dialect identification system
previously reported, we fine-tune AraBERT on the dataset and consider it a
strong baseline. We report results in both accuracy and macro F1, but treat F1
as the metric for comparison to align with results on other datasets.

###  4.6 Dialect Identification Results

Our models on the dialect datasets (Section 4.5) can be viewed as performing
identification on $5$ different levels: binary (MSA vs. DA), regional (e.g.,
Egyptian, Gulf, Levantine, and Maghrebi), country (e.g., Algeria, Egypt,
Morocco, Saudi Arabia), and province levels (e.g., the Egyptian provinces of
Cairo and Alexandria, the Saudi Arabia provinces of Al-Madinah and Ar-Riyad).
As Table 11 shows, MARBERT outperforms all SOTA as well as AraBERT with an
average of $5\%$ F1 across all classification levels. These results reflect
the powerful and diverse dialectal representation capacity of MARBERT.
Although ARBERTis mainly developed for MSA, it outperforms AraBERT, mBERT, and
XLM-R with an average of $3\%$ F1 across the different dialect classification
levels. This is a function of overlap (e.g., lexical) between MSA and dialects
and demonstratesARBERT’s ability to transfer knowledge on what may be referred
to as a “zero-shot” setting (i.e., going from MSA to dialects).222222This
applies only in the wider sense of the term, due to cited overlap between MSA
and dialects.

TEST Dataset (#classes) Task SOTA LM Acc. F1 mBERT 76.30 43.81 XLM-RB 77.44
41.71 XLM-RL 77.96 41.83 AraBERT 79.00 47.54 6-10 ARBERT 76.87 51.27
ArSarcasmDia (5) Regoin 47.54 MARBERT 78.29 54.70 mBERT 48.89 34.92 XLM-RB
49.48 35.91 XLM-RL 50.17 35.14 AraBERT 49.26 34.87 6-10 ARBERT 51.78 37.90
MADAR-TL (21) Country – MARBERT 53.77 40.40 mBERT 49.11 36.02 XLM-RB 50.02
34.59 XLM-RL 50.72 35.77 AraBERT 50.22 36.44 6-10 ARBERT 51.38 37.84 AOC (4)
Region 82.45$\star$ MARBERT 86.19 82.37 mBERT 86.17 85.76 XLM-RB 86.87 86.39
XLM-RL 88.02 87.56 AraBERT 88.15 87.68 6-10 ARBERT 89.51 89.06 AOC (3) Region
78.81$\star$ MARBERT 91.27 90.85 mBERT 86.81 86.19 XLM-RB 87.34 86.85 XLM-RL
87.87 87.30 AraBERT 88.25 87.76 6-10 ARBERT 88.94 88.46 AOC (4) Binary
87.23$\star$ MARBERT 89.08 88.59 mBERT 66.92 66.57 XLM-RB 76.61 77.00 XLM-RL
82.58 82.73 AraBERT 72.00 72.23 6-10 ARBERT 88.60 88.63 QADI (18) Country
60.6$\star\star$ MARBERT 90.88 90.89 mBERT 32.38 13.32 XLM-RB 34.18 16.36 XLM-
RL 35.16 17.17 AraBERT 36.06 17.46 6-10 ARBERT 41.02 22.56 NADI (21) Country
26.78$\dagger\dagger$ MARBERT 48.40 29.14 mBERT 3.32 2.13 XLM-RB 4.76 4.12
XLM-RL 2.14 0.32 AraBERT 3.86 3.13 6-10 ARBERT 6.94 6.10 Province (100 )
Province 6.08$\dagger\dagger$ MARBERT 8.48 6.28 $\star$ Elaraby and Abdul-
Mageed (2018), $\star\star$ Abdelali et al. (2020), $\dagger\dagger$ Abdul-
Mageed et al. (2020a), $\dagger$ Zhang and Abdul-Mageed (2019b). SOTA
evaluation metric is $F_{1}$ score

Table 11: Dialect Identification Tasks.

###  4.7 Named Entity Recognition

NER is usually carried out as a sequence labeling task with the objective of
predicting which words refer to categories of real-world objects such persons,
locations, and organisations. We fine-tune our models on $5$ datasets, which
we now introduce.

####  4.7.1 Datasets

We evaluate on the Automatic Context Extraction (ACE)’s 2003 Broadcast News
(ACE-BN 2003) and Newswire (ACE-NW 2003) Mitchell et al. (2004), ACE’s 2004
Newswire datasets Mitchell et al. (2005), ANERcorp Benajiba and Rosso (2007),
and Twitter Social Media NER (TW-NER) Darwish (2013). Table 12 shows the
distribution of named entity classes and the total number of tokens across the
$5$ datasets as presented by Khalifa and Shaalan (2019).

Dataset | #Tokens | #PER | #LOC | #ORG
---|---|---|---|---
ANERCorp | $150K$ | $6,504$ | $5,018$ | $3,437$
ACE-2003_BN |  $15$K | $832$ | $1,223$ | $288$
ACE-2003_NW |  $27$K | $1,146$ | $1,147$ | $893$
ACE-2004_BN | $70K$ | $3201$ | $3,921$ | $2,239$
TW-NER | $81K$ | $1,252$ | $1,300$ | $765$
Table 12: Distribution of classes and token size of the Arabic NER datasets
Khalifa and Shaalan (2019).

####  4.7.2 Baseline

We compare our results with SOTA presented by Khalifa and Shaalan (2019) and
follow them in focusing on person (PER), location (LOC) and organization (ORG)
named entity tags, and setting other tags to the unnamed entity (O). Khalifa
and Shaalan (2019) apply a character Convolutional Neural Networks (WC-CNN)
and character-level bidirectional Long Short-Term Memory (WC-BiLSTM) on the
$5$ listed NER datasets. They use an $80-10-10$ train-dev-test split and
report F1 scores of $88.77$, $91.47$, $94.92$, $91.20$, and $65.34$ on test on
the ANERcorp, ACE-NW 2004, ACE-BN 2003, ACE-NW 2003, and TW-NER datasets,
respectively. We use their exact data splits.

TEST Dataset (#classes) SOTA LM Acc. F1 mBERT 97.69 86.78 XLM-RB 88.06 87.24
XLM-RL 98.73 89.94 AraBERT 98.41 89.13 5-9 ARBERT 97.85 84.38 ANERcorp. 88.77
MARBERT 9756 80.64 mBERT 96.82 86.37 XLM-RB 97.94 89.93 XLM-RL 98.16 89.89
AraBERT 97.74 89.03 5-9 ARBERT 97.65 88.24 ACE-NW 2004 91.47 MARBERT 97.45
85.02 mBERT 98.08 91.23 XLM-RB 92.97 83.97 XLM-RL 97.77 85.41 AraBERT 98.64
91.94 5-9 ARBERT 99.21 96.18 ACE-BN 2003 94.92 MARBERT 97.02 79.05 mBERT 96.03
81.40 XLM-RB 97.31 87.24 XLM-RL 98.10 90.62 AraBERT 97.88 88.09 5-9 ARBERT
97.90 90.09 ACE-NW 2003 91.2 MARBERT 97.67 87.76 mBERT 90.80 36.83 XLM-RB
92.22 49.16 XLM-RL 92.80 54.44 AraBERT 88.61 41.26 5-9 ARBERT 95.50 59.17
Twitter 65.34 MARBERT 96.57 67.39 SOTA are results reported in Khalifa and
Shaalan (2019) based on F1.

Table 13: Performance of our models on NER task.

###  4.8 NER Results

As Table 13 shows, we outperform SOTA on $3$ out of the $5$ datasets for NER.
Namely, our MARBERT model outperforms SOTA on ACE-NW 2004 and Twitter datasets
with $+1.26$% and $+2.05$% F1, respectively. Our model ARBERTalso outperforms
SOTA on ACE-BN 2003, with $+1.26$% F1. XML-RLarge acquires best F1 ($89.94$)
on ArERcop. We also note that our models outperform AraBERT on ACE-NW 2003,
ACE-BN 2003, and Twitter datasets. Again, these reults reflect the transfer
learning strength of our models.

##  5 Conclusion

In this paper, we reported our efforts to develop two powerful Transformer-
based language models for Arabic. Our moddels are trained on large datasets
from MSA (ARBERT) or both MSA and DA (MARBERT). We also traced, collected, and
benchmarked $45$ models on the $41$ comprising AraNLU across $5$ tasks/task
clusters. When fine-tuned on the various labeled datasets, our models achieve
new SOTA on all the tasks on the majority of the datasets. More precisely, out
of the $45$ models we train, our models acquire SOTA in $37$ cases
(%=$82.22$).

Compared to multilingual models such as mBERT and XLM-R, our models have
better representation of Arabic and yield better performance. Even when
compared to XLM-RLarge model, which is larger, our models compare more
favorably (in addition to being less computationally costly at inference
time). Compared to Arabic-specific models such as AraBERT, ARBERT has
generally better MSA representation and performs better in most cases.
Compared to both ARBERT and AraBERT, MARBERT covers DA and is much more
powerful. Our models are publicly available for research. In the future, we
plan to evaluate our models on more Arabic NLP tasks and further pre-train
them to improve their performance on the datasets where they are currently
outperformed.

## Acknowledgements

MAM gratefully acknowledges support from the Natural Sciences and Engineering
Research Council of Canada, the Social Sciences and Humanities Research
Council of Canada, Canadian Foundation for Innovation, Compute Canada
(www.computecanada.ca) and UBC ARC-Sockeye (https://doi.org/10.14288/SOCKEYE).
We also thank the Google TFRC program for providing us with free TPU access.

## References

  * Abbas et al. (2011) Mourad Abbas, Kamel Smaïli, and Daoud Berkani. 2011.  Evaluation of topic identification methods on arabic corpora.  _JDIM_ , 9(5):185–192.
  * Abdelali et al. (2016) Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016.  Farasa: A fast and furious segmenter for arabic.  In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations_ , pages 11–16.
  * Abdelali et al. (2020) Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2020\.  Arabic Dialect Identification in the Wild.  _arXiv preprint arXiv:2005.06557_.
  * Abdul-Mageed et al. (2014) Muhammad Abdul-Mageed, Mona Diab, and Sandra Kübler. 2014.  Samar: Subjectivity and sentiment analysis for arabic social media.  _Computer Speech & Language_, 28(1):20–37.
  * Abdul-Mageed and Diab (2012) Muhammad Abdul-Mageed and Mona T Diab. 2012.  AWATIF: A Multi-Genre Corpus for Modern Standard Arabic Subjectivity and Sentiment Analysis.  In _LREC_ , volume 515, pages 3907–3914. Citeseer.
  * Abdul-Mageed et al. (2020a) Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020a.  NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_.
  * Abdul-Mageed et al. (2020b) Muhammad Abdul-Mageed, Chiyu Zhang, Azadeh Hashemi, and El Moatez Billah Nagoudi. 2020b.  AraNet: A deep learning toolkit for Arabic social media.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 16–23, Marseille, France. European Language Resource Association.
  * Abdulla et al. (2013) Nawaf Abdulla, N Mahyoub, M Shehab, and Mahmoud Al-Ayyoub. 2013.  Arabic sentiment analysis: Corpus-based and lexicon-based.  In _Proceedings of The IEEE conference on Applied Electrical Engineering and Computing Technologies (AEECT)_.
  * Akbik et al. (2019) Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019.  Flair: An easy-to-use framework for state-of-the-art nlp.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 54–59.
  * Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018.  Contextual string embeddings for sequence labeling.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1638–1649.
  * Al-Twairesh et al. (2017) Nora Al-Twairesh, Hend Al-Khalifa, AbdulMalik Al-Salman, and Yousef Al-Ohali. 2017\.  Arasenti-tweet: A corpus for Arabic sentiment analysis of saudi tweets.  _Procedia Computer Science_ , 117:63–72.
  * Alhuzali et al. (2018) Hassan Alhuzali, Muhammad Abdul-Mageed, and Lyle Ungar. 2018.  Enabling deep learning of emotion with first-person seed expressions.  In _Proceedings of the Second Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media_ , pages 25–35.
  * Alomari et al. (2017) Khaled Mohammad Alomari, Hatem M ElSherif, and Khaled Shaalan. 2017.  Arabic tweets sentimental analysis using machine learning.  In _International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems_ , pages 602–610. Springer.
  * Alshehri et al. (2020a) Ali Alshehri, Muhammad Abdul-Mageed, et al. 2020a.  Understanding and detecting dangerous speech in social media.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 40–47.
  * Alshehri et al. (2020b) Ali Alshehri, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2020b.  Understanding and detecting dangerous speech in social media.  In _The 4th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT4), LREC_.
  * Altınel and Ganiz (2018) Berna Altınel and Murat Can Ganiz. 2018.  Semantic text classification: A survey of past and recent advances.  _Information Processing & Management_, 54(6):1129–1153.
  * Aly and Atiya (2013) Mohamed Aly and Amir Atiya. 2013.  Labr: A large scale arabic book reviews dataset.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , volume 2, pages 494–498.
  * Antoun et al. (2020) Wissam Antoun, Fady Baly, and Hazem Hajj. 2020.  Arabert: Transformer-based model for arabic language understanding.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 9–15.
  * Baly et al. (2019) Ramy Baly, Alaa Khaddaj, Hazem Hajj, Wassim El-Hajj, and Khaled Bashir Shaban. 2019\.  ArSentD-LEV: A multi-topic corpus for target-based sentiment analysis in Arabic levantine tweets.  _arXiv preprint arXiv:1906.01830_.
  * Benajiba and Rosso (2007) Yassine Benajiba and Paolo Rosso. 2007.  Anersys 2.0: Conquering the ner task for the arabic language by combining the maximum entropy with pos-tag information.  In _IICAI_ , pages 1814–1823.
  * Bouamor et al. (2018) Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018.  The MADAR Arabic dialect corpus and lexicon.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)_.
  * Bouamor et al. (2019) Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019.  The MADAR shared task on Arabic fine-grained dialect identification.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 199–207.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Chouigui et al. (2017) Amina Chouigui, Oussama Ben Khiroun, and Bilel Elayeb. 2017.  Ant corpus: an arabic news text collection for textual classification.  In _2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA)_ , pages 135–142. IEEE.
  * Columbia University (2016) Columbia University. 2016.  Arabic Named Entity Recognition Task.  http://www1.ccls.columbia.edu/ ybenajiba.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _arXiv preprint arXiv:1911.02116_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Dadas et al. (2020) Sławomir Dadas, Michał Perełkiewicz, and Rafał Poświata. 2020.  Pre-training Polish Transformer-based Language Models at Scale.  _arXiv preprint arXiv:2006.04229_.
  * Darwish (2013) Kareem Darwish. 2013.  Named entity recognition using cross-lingual resources: Arabic as an example.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1558–1567.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Djandji et al. (2020) Marc Djandji, Fady Baly, Wissam Antoun, and Hazem Hajj. 2020.  Multi-task learning using AraBert for offensive language detection.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 97–101, Marseille, France. European Language Resource Association.
  * El-Khair (2016) Ibrahim Abu El-Khair. 2016.  1.5 billion words arabic corpus.  _arXiv preprint arXiv:1611.04033_.
  * El Mekki et al. (2020) Abdellah El Mekki, Ahmed Alami, Hamza Alami, Ahmed Khoumsi, and Ismail Berrada. 2020\.  Weighted combination of BERT and N-GRAM features for Nuanced Arabic Dialect Identification.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , Barcelona, Spain.
  * Elaraby and Abdul-Mageed (2018) Mohamed Elaraby and Muhammad Abdul-Mageed. 2018.  Deep models for Arabic dialect identification on benchmarked data.  In _Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018)_ , pages 263–274, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Elmadany et al. (2018) AbdelRahim Elmadany, Hamdy Mubarak, and Walid Magdy. 2018.  ArSAS: An Arabic speech-act and sentiment corpus of tweets.  _OSACT_ , 3:20.
  * Elnagar et al. (2018) Ashraf Elnagar, Yasmin S Khalifa, and Anas Einea. 2018.  Hotel Arabic-reviews dataset construction for sentiment analysis applications.  In _Intelligent Natural Language Processing: Trends and Applications_ , pages 35–52. Springer.
  * Farha and Magdy (2019) Ibrahim Abu Farha and Walid Magdy. 2019.  Mazajak: An online arabic sentiment analyser.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 192–198.
  * Farha and Magdy (2020) Ibrahim Abu Farha and Walid Magdy. 2020.  From Arabic Sentiment Analysis to Sarcasm Detection: The ArSarcasm Dataset.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 32–39.
  * Ghanem et al. (2019) Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019.  IDAT@FIRE2019: Overview of the Track on Irony Detection in Arabic Tweets. .  In _Mehta P., Rosso P., Majumder P., Mitra M. (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019). CEUR Workshop Proceedings. In: CEUR-WS.org, Kolkata, India, December 12-15_.
  * Hassan et al. (2020) Sabit Hassan, Younes Samih, Hamdy Mubarak, Ahmed Abdelali, Ammar Rashed, and Shammur Absar Chowdhury. 2020.  ALT submission for OSACT shared task on offensive language detection.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 61–65, Marseille, France. European Language Resource Association.
  * Husain (2020) Fatemah Husain. 2020.  OSACT4 shared task on offensive language detection: Intensive preprocessing-based approach.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 53–60, Marseille, France. European Language Resource Association.
  * Khalifa and Shaalan (2019) Muhammad Khalifa and Khaled Shaalan. 2019.  Character convolutions for arabic named entity recognition with long short-term memory networks.  _Computer Speech & Language_, 58:335–346.
  * Kiritchenko et al. (2016) Svetlana Kiritchenko, Saif Mohammad, and Mohammad Salameh. 2016.  Semeval-2016 task 7: Determining sentiment intensity of english and arabic phrases.  In _Proceedings of the 10th international workshop on semantic evaluation (SEMEVAL-2016)_ , pages 42–51.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017.  Race: Large-scale reading comprehension dataset from examinations.  _arXiv preprint arXiv:1704.04683_.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_.
  * Le et al. (2020) Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020.  FlauBERT: Unsupervised Language Model Pre-training for French.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 2479–2490.
  * Liu et al. (2019a) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2019b) Zihan Liu, Yan Xu, Genta Indra Winata, and Pascale Fung. 2019b.  Incorporating word and subword units in unsupervised machine translation using language model rescoring.  _arXiv preprint arXiv:1908.05925_.
  * Malmsten et al. (2020) Martin Malmsten, Love Börjeson, and Chris Haffenden. 2020.  Playing with Words at the National Library of Sweden–Making a Swedish BERT.  _arXiv preprint arXiv:2007.01658_.
  * Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  CamemBERT: a tasty French language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7203–7219, Online. Association for Computational Linguistics.
  * Mikolov et al. (2017) Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2017.  Advances in pre-training distributed word representations.  _arXiv preprint arXiv:1712.09405_.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  In _Advances in neural information processing systems_ , pages 3111–3119.
  * Mitchell et al. (2005) Alexis Mitchell, Stephanie Strassel, Shudong Huang, and Ramez Zakhary. 2005.  Ace 2004 multilingual training corpus.  _Linguistic Data Consortium, Philadelphia_ , 1:1–1.
  * Mitchell et al. (2004) Alexis Mitchell, Stephanie Strassel, Mark Przybocki, J Davis, George Doddington, Ralph Grishman, and B Sundheim. 2004.  Tides extraction (ace) 2003 multilingual training data. ldc2004t09.  _Linguistic Data Consortium, Philadelphia Web Download_.
  * Mohammad et al. (2018) S. Bravo-Marquez Mohammad, M. F. Salameh, and S. Kiritchenko. 2018.  Semeval-2018 task 1: Affect in tweets.  In _Proceedings of International Workshop on Semantic Evaluation (SemEval-2018)_. Association for Computational Linguistics.
  * Mozannar et al. (2019) Hussein Mozannar, Karl El Hajal, Elie Maamary, and Hazem Hajj. 2019.  Neural arabic question answering.  _arXiv preprint arXiv:1906.05394_.
  * Mubarak et al. (2020) Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020\.  Overview of OSACT4 Arabic offensive language detection shared task.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 48–52, Marseille, France. European Language Resource Association.
  * Nabil et al. (2015) Mahmoud Nabil, Mohamed Aly, and Amir F Atiya. 2015.  Astd: Arabic sentiment tweets dataset.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2515–2519.
  * Nguyen and Nguyen (2020) Dat Quoc Nguyen and Anh Tuan Nguyen. 2020.  PhoBERT: Pre-trained language models for Vietnamese.  _arXiv preprint arXiv:2003.00744_.
  * Obeid et al. (2020) Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020.  Camel tools: An open source python toolkit for arabic natural language processing.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 7022–7032.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _EMNLP_ , volume 14, pages 1532–1543.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  _arXiv preprint arXiv:1802.05365_.
  * Pyysalo et al. (2020) Sampo Pyysalo, Jenna Kanerva, Antti Virtanen, and Filip Ginter. 2020.  WikiBERT models: deep transfer learning for many languages.  _arXiv preprint arXiv:2006.01538_.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_.
  * Rosenthal et al. (2017) Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017.  Semeval-2017 task 4: Sentiment analysis in twitter.  In _Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017)_ , pages 502–518.
  * Saad and Ashour (2010) Motaz K Saad and Wesam M Ashour. 2010.  Osac: Open source arabic corpora.  _Osac: Open source arabic corpora_ , 10.
  * Salameh et al. (2015) Mohammad Salameh, Saif M Mohammad, and Svetlana Kiritchenko. 2015.  Sentiment after translation: A case-study on arabic social media posts.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 767–777.
  * Schuster and Nakajima (2012) Mike Schuster and Kaisuke Nakajima. 2012.  Japanese and korean voice search.  In _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5149–5152. IEEE.
  * Suárez et al. (2019) Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019.  Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures.  In _7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7)_. Leibniz-Institut für Deutsche Sprache.
  * Talafha et al. (2020) Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein T Al-Natsheh. 2020.  Multi-dialect arabic bert for country-level dialect identification.  _arXiv preprint arXiv:2007.05612_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pages 6000–6010.
  * Virtanen et al. (2019) Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019.  Multilingual is not enough: BERT for Finnish.  _arXiv preprint arXiv:1912.07076_.
  * de Vries et al. (2019) Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019.  Bertje: A dutch bert model.  _arXiv preprint arXiv:1912.09582_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Zaghouani and Charfi (2018) Wajdi Zaghouani and Anis Charfi. 2018.  Arap-Tweet: A Large Multi-Dialect Twitter Corpus for Gender, Age and Language Variety Identification.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)_.
  * Zaidan and Callison-Burch (2014) Omar F Zaidan and Chris Callison-Burch. 2014.  Arabic dialect identification.  _Computational Linguistics_ , 40(1):171–202.
  * Zeroual et al. (2019) Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019.  OSIAN: Open source international Arabic news corpus-preparation and integration into the CLARIN-infrastructure.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 175–182.
  * Zhang and Abdul-Mageed (2019a) Chiyu Zhang and Muhammad Abdul-Mageed. 2019a.  Multi-task bidirectional transformer representations for irony detection.  _arXiv preprint arXiv:1909.03526_.
  * Zhang and Abdul-Mageed (2019b) Chiyu Zhang and Muhammad Abdul-Mageed. 2019b.  No army, no navy: Bert semi-supervised learning of arabic dialects.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 279–284.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.

In the majority of our tasks, we either use the train/dev/test splits of a
given work we compare to or split existing data for which we identified no
clear splits into the standard train/dev/test. We present results on the DEV
splits across the different tasks here, noting that a few of the works we
compare to (namely Antoun et al. (2020), Abdelali et al. (2020), Obeid et al.
(2020), and Farha and Magdy (2020)) did not have a DEV split and hence do not
show in our appendices.

##  Appendix A Sentiment Analysis

Data (#classes) LM Dev Acc. F1 F1PN AraNET sentiment (2) mBERT 84.80 83.84
84.00 XLM-RB 92.40 91.93 92.00 XLM-RL 93.50 93.11 93.00 AraBERT 87.14 86.44
86.50 3-8 ARBERT 89.55 88.95 89.00 MARBERT 92.77 92.31 92.00 AraSenTi-Tweet
(3) mBERT 93.05 92.94 93.00 XLM-RB 94.25 94.10 93.50 XLM-RL 95.44 95.34 95.00
AraBERT 92.75 92.60 91.50 3-8 ARBERT 92.60 92.49 92.00 MARBERT 93.87 93.81
93.50 BBN(3) mBERT 64.52 56.08 68.00 XLM-RB 71.77 62.55 75.00 XLM-RL 46.77
31.30 47.00 AraBERT 66.13 46.69 70.00 3-8 ARBERT 75.00 66.45 79.50 MARBERT
75.81 68.57 78.50 SYTS(3) mBERT 70.78 41.34 62.00 XLM-RB 79.87 67.93 80.50
XLM-RL 66.88 26.72 40.00 AraBERT 72.73 43.22 65.00 3-8 ARBERT 69.48 45.86
69.00 MARBERT 76.62 48.39 72.50 Saad2019 (2) mBERT 79.81 79.76 80.00 XLM-RB
95.60 95.60 95.50 XLM-RL 95.53 95.53 95.50 AraBERT 81.24 81.24 81.50 3-8
ARBERT 89.90 89.90 90.00 MARBERT 96.02 96.02 96.00 SAMAR(5) mBERT 46.60 24.22
26.00 XLM-RB 55.02 50.30 54.50 XLM-RL 59.22 50.29 61.00 AraBERT 49.84 36.06
42.50 3-8 ARBERT 56.31 40.19 50.50 MARBERT 58.58 52.18 62.50 AWATIF(4) mBERT
65.84 59.01 63.50 XLM-RB 68.33 59.83 62.00 XLM-RL 70.82 65.35 67.50 AraBERT
71.53 61.99 65.00 3-8 ARBERT 73.31 66.89 70.50 MARBERT 75.09 65.51 72.00
Abdullah2013 (2) mBERT 87.56 87.41 87.50 XLM-RB 91.04 90.96 91.00 XLM-RL 95.52
95.47 95.50 AraBERT 92.54 92.52 92.50 3-8 ARBERT 99.00 98.99 99.00 MARBERT
97.01 96.98 97.00

Table A.1: Results of sentiment analysis models on DEV.

##  Appendix B Sentiment Analysis Results

DEV Dataset (#classes) Task LM Acc. F1 mBERT 91.10 84.04 XLM-RB 91.70 85.26
XLM-RL 92.40 86.72 AraBERT 92.70 87.21 5-10 ARBERT 93.50 88.77 OSACT-A (2)
Offensive MARBERT 95.10 91.68 mBERT 96.70 75.91 XLM-RB 96.40 76.56 XLM-RL
96.50 78.00 AraBERT 95.00 72.09 5-10 ARBERT 96.30 75.01 OSACT-B (2) Hate
Speech MARBERT 97.00 82.91 mBERT 78.05 67.35 XLM-RB 78.21 65.09 XLM-RL 81.30
69.95 AraBERT 80.98 67.73 5-10 ARBERT 80.33 68.58 AraDange (2) Dangerous
MARBERT 83.25 75.50 mBERT 62.93 61.34 XLM-RB 73.05 72.09 XLM-RL 73.71 72.78
AraBERT 67.22 65.46 5-10 ARBERT 69.20 68.05 AraNETEmo (8) Emotion MARBERT
75.80 75.18 mBERT 81.14 81.08 XLM-RB 83.13 83.12 XLM-RL 81.39 81.29 AraBERT
79.16 79.12 5-10 ARBERT 84.86 84.83 FIRE2019 (2) Irony MARBERT 86.85 86.77
mBERT 56.19 56.33 XLM-RB 59.73 59.70 XLM-RL 53.45 53.63 AraBERT 57.51 57.67
5-10 ARBERT 58.48 58.60 Arab_Tweet (3) Age MARBERT 62.05 62.19 mBERT 68.60
68.06 XLM-RB 71.25 71.10 XLM-RL 71.54 71.23 AraBERT 68.49 67.61 5-10 ARBERT
70.30 69.97 Arab_Tweet (2) Gender MARBERT 73.07 72.81

Table B.1: Results of social meaning models on DEV.

##  Appendix C Social Meaning

DEV Data (#classes) LM Acc. F1 mBERT 98.75 97.87 XLM-RB 98.80 97.75 XLM-RL
98.80 97.61 AraBERT 98.93 97.94 5-9 ARBERT 98.66 97.56 OSAC (10) MARBERT 98.80
97.66 mBERT 94.55 94.48 XLM-RB 95.43 95.32 XLM-RL 96.31 96.09 AraBERT 95.96
95.65 5-9 ARBERT 96.13 96.16 Khallej (4) MARBERT 96.31 96.31 mBERT 85.79 85.04
XLM-RB 86.96 86.74 XLM-RL 88.30 87.41 AraBERT 88.55 87.98 5-9 ARBERT 87.25
87.06 ANTText (5) MARBERT 86.30 85.80 mBERT 80.08 79.46 XLM-RB 81.70 80.77
XLM-RL 82.62 82.04 AraBERT 83.67 83.56 5-9 ARBERT 81.29 81.10 ANTTitle (5)
MARBERT 81.73 82.36 mBERT 87.41 87.24 XLM-RB 87.25 86.36 XLM-RL 88.71 88.45
AraBERT 88.68 88.76 5-9 ARBERT 87.41 87.27 ANTTitle+Text (5) MARBERT 64.08
85.99

Table C.1: Results of TC models on DEV.

##  Appendix D Dialect Identification

DEV Dataset (#classes) Task LM Acc. F1 mBERT 48.07 33.75 XLM-Rbase 48.91 34.54
XLM-Rlarge 49.34 33.28 AraBERT 48.84 33.47 6-10 ARBERT 51.84 39.24 MADAR-TL
(21) Country MARBERT 53.48 40.61 mBERT 48.54 35.38 XLM-Rbase 49.15 33.50 XLM-
Rlarge 49.82 36.31 AraBERT 49.61 35.89 6-10 ARBERT 51.53 39.14 AOC (4) Region
MARBERT 87.15 83.56 mBERT 87.46 87.07 XLM-Rbase 87.24 86.80 XLM-Rlarge 88.58
88.21 AraBERT 88.94 88.46 6-10 ARBERT 89.98 89.57 AOC (3) Region MARBERT 91.93
91.56 mBERT 88.36 87.89 XLM-Rbase 88.03 87.63 XLM-Rlarge 88.81 88.38 AraBERT
89.15 88.76 6-10 ARBERT 89.72 89.32 AOC (4) Binary MARBERT 90.06 89.66 mBERT
33.17 14.49 XLM-Rbase 35.34 17.30 XLM-Rlarge 37.44 18.62 AraBERT 36.51 16.18
6-10 ARBERT 42.87 23.73 NADI (21) Country MARBERT 48.86 26.40 mBERT 4.10 2.32
XLM-Rbase 4.70 3.91 XLM-Rlarge 2.78 0.65 AraBERT 4.12 3.04 6-10 ARBERT 7.16
6.05 Province (100 ) Province MARBERT 7.91 5.23

Table D.1: Results of dialect identification models on DEV.

##  Appendix E NER

DEV Dataset (#classes) LM Acc. F1 mBERT 97.74 86.20 XLM-RB 88.06 87.24 XLM-RL
98.63 89.64 AraBERT 98.51 90.24 5-9 ARBERT 97.77 83.32 ANERcorp. MARBERT xx xx
mBERT 96.77 86.57 XLM-RB 97.64 88.21 XLM-RL 97.93 90.49 AraBERT 97.76 89.76
5-9 ARBERT 97.31 86.17 ACE 2004 NW MARBERT xx xx mBERT 96.46 80.35 XLM-RB
91.55 40.36 XLM-RL 97.01 83.39 AraBERT 96.84 81.05 5-9 ARBERT 98.38 90.91 ACE
2003 BN MARBERT xx xx mBERT 96.63 87.21 XLM-RB 97.78 90.08 XLM-RL 98.12 91.94
AraBERT 97.65 89.70 5-9 ARBERT 97.23 89.33 ACE 2003 NW MARBERT xx xx mBERT
96.21 52.60 XLM-RB 98.01 73.48 XLM-RL 98.29 77.70 AraBERT 97.59 73.61 5-9
ARBERT 97.91 70.78 Twitter MARBERT xx xx

Table E.1: Results of NER models on DEV.