utf8

# ArabicaQA: A Comprehensive Dataset for Arabic Question Answering

Abdelrahman Abdallah<EMAIL_ADDRESS>University of
InnsbruckInnrain 52InnsbruckAustria ,  Mahmoud Kasem<EMAIL_ADDRESS>Assuit UniversityAssuitEgypt ,  Mahmoud Abdalla<EMAIL_ADDRESS>DISCO AIArd El-Marwaha, El-Katamey,Cairo, 11936,CairoEgypt ,  Mohamed Mahmoud
<EMAIL_ADDRESS>Assuit UniversityAssuitEgypt ,  Mohamed Elkasaby
<EMAIL_ADDRESS>DISCO AIArd El-Marwaha, El-Katamey,Cairo,
11936,CairoEgypt ,  Yasser Elbendary<EMAIL_ADDRESS>DISCO AIArd El-
Marwaha, El-Katamey,Cairo, 11936,CairoEgypt and  Adam Jatowt
<EMAIL_ADDRESS>University of InnsbruckInnrain 52InnsbruckAustria

(2024)

###### Abstract.

In this paper, we address the significant gap in Arabic natural language
processing (NLP) resources by introducing ArabicaQA, the first large-scale
dataset for machine reading comprehension and open-domain question answering
in Arabic. This comprehensive dataset, consisting of 89,095 answerable and
3,701 unanswerable questions created by crowdworkers to look similar to
answerable ones, along with additional labels of open-domain questions marks a
crucial advancement in Arabic NLP resources. We also present AraDPR, the first
dense passage retrieval model trained on the Arabic Wikipedia corpus,
specifically designed to tackle the unique challenges of Arabic text
retrieval. Furthermore, our study includes extensive benchmarking of large
language models (LLMs) for Arabic question answering, critically evaluating
their performance in the Arabic language context. In conclusion, ArabicaQA,
AraDPR, and the benchmarking of LLMs in Arabic question answering offer
significant advancements in the field of Arabic NLP. The dataset and code are
publicly accessible for further
research111https://github.com/DataScienceUIBK/ArabicaQA.

Arabic question answering, Question generation, Information retrieval, LLM

††copyright: acmcopyright††journalyear: 2024††conference: International
Conference on Information Retrieval (SIGIR); July 14–18, 2024; Washington,
USA††booktitle: SIGIR 2024††ccs: Information systems Question answering

##  1\. Introduction

Figure 1. An example from the ArabicaQA dataset illustrating a passage about
the Battle of Chosin Reservoir during the Korean War, along with corresponding
question-answer pairs.

Question Answering (QA) has been a central area of study in the field of
Natural Language Processing (NLP) for many years. The goal of QA is to design
systems that can accurately answer questions posed in natural language. There
are several types of QA systems, each with its unique challenges and
requirements. For instance, factoid QA systems (Soricut and Brill, 2006;
Abdallah et al., 2023b, 2020) aim to answer factual questions, typically by
retrieving information from a structured knowledge base. On the other hand,
non-factoid QA systems (Bolotova et al., 2022; Bolotova-Baranova et al., 2023)
deal with more complex questions that require reasoning or inference. Recent
years have seen significant advancements in deep learning, particularly in
machine translation (Singh et al., 2017), speech recognition (Kamath et al.,
2019), Question Answering (Allam and Haggag, 2012), and image recognition (Wu
and Chen, 2015). These strides are largely due to the development of neural
methods and the availability of large datasets for training. Machine Reading
Comprehension (MRC) (Nguyen et al., 2016) and open-domain question answering
(Open-domain QA) (Yang et al., 2015) have emerged as key areas of interest.
MRC involves designing systems that can understand a passage and answer
questions related to it, while Open-domain QA aims to answer factual questions
from a large knowledge corpus without any explicit evidence given.

Arabic, being one of the most widely spoken languages in the world, has unique
linguistic features that pose challenges for NLP and IR tasks (Darwish et al.,
2021; Abdallah et al., 2023a; Bashir et al., 2023). These include rich
morphological variations, the use of diacritics, and the existence of Modern
Standard Arabic and various dialects. Despite these challenges, Arabic NLP has
still not received as much attention as other main languages, leading to a
significant gap in resources and research.

In particular, Arabic NLP has seen a noticeable lack of representation in
question-answering datasets suitable for MRC and Open-domain QA. This has
prevented the development of effective QA systems in Arabic, a language known
for its unique linguistic features. In contrast, English has seen significant
progress, driven by extensive datasets such as the Stanford QA Dataset (SQuAD)
(Rajpurkar et al., 2016), Natural Question(NQ) (Kwiatkowski et al., 2019), and
WikiQA (Yang et al., 2015). Our research introduces two major innovations to
bridge the gap in Arabic NLP: ArabicaQA and AraDPR. ArabicaQA is a novel
dataset for Arabic MRC and Open-domain QA, consisting of 89,095 answerable and
3,701 unanswerable questions for MRC. 76,266 question-answer pairs are also
open domain. This makes ArabicaQA the most comprehensive dataset of its kind
for Arabic.

Figure 1 illustrates an example of the question-answer pairs derived from an
example passage in our dataset, showing the complex storyline structures and
rich content that our dataset includes. The creation of ArabicaQA involved a
annotation process to ensure high-quality and relevant question-answer pairs.
For the MRC, annotators formulate questions based on Wikipedia passages and
select the corresponding answers directly from the passages. A second
annotator reviews every question and answer, providing a layer of verification
and quality control. In the open-domain section, annotators identify questions
suitable for Open-domain QA and categorize them based on whether they require
a short or long answer, a critical distinction in developing robust Open-
domain QA systems.

As our second contribution, we also introduce AraDPR, the first dense passage
retrieval model trained on the Arabic Wikipedia corpus. It leverages the power
of pre-trained transformers such as BERT and AraBERT (Devlin et al., 2018;
Antoun et al., 2020), which is an active area of research in the field of
information retrieval (Zhang et al., 2023a). AraDPR adopts a Bi-encoder
architecture (Vaswani et al., 2017), allowing for the offline computation of
document representations. This transforms the document ranking task into a
nearest-neighbor search problem, given the query vector representation. The
Bi-encoder architecture offers an attractive alternative to the cross-encoder
architecture, where queries and documents are concatenated and fed into a
transformer directly. While cross-encoders are practical in a reranking setup,
processing candidates generated by the first-stage retrieval system, Bi-
encoders directly support single-stage retrieval.

We not only introduce AraDPR but also conduct an extensive benchmarking of
Large Language Models (LLMs) for Arabic QA. Our comprehensive benchmarking of
LLMs like GPT-3 (Brown et al., 2020), Llama (Touvron et al., 2023), and Falcon
(Almazrouei et al., 2023) provides several insights into their performance in
the Arabic context. These LLMs, known for their remarkable capabilities in
understanding and generating text based on context, can exhibit varying
performance depending on the language and the specific task at hand. Our
findings from this benchmarking exercise offer guidance for researchers and
practitioners working on Arabic QA, helping them choose the most suitable
models.

In summary, our contributions are as follows:

  1. (1)

ArabicaQA Dataset: We introduce the first large-scale dataset for Arabic MRC
and Open-domain QA, comprising both answered and unanswered questions, along
with a distinct open-domain component.

  2. (2)

AraDPR: We develop the first Arabic dense passage retrieval model trained
specifically for Arabic text, addressing the unique challenges of Arabic
language retrieval.

  3. (3)

Benchmarking of LLMs: We provide a comprehensive evaluation of LLMs in Arabic
QA, establishing a baseline for their performance in Arabic NLP.

Table 1. Comparison of related datasets.

Dataset #Questions Answer Type Question Source Corpus Source Language QA Type
Unanswerable MS MARCO (Nguyen et al., 2016) 1M Generative, Boolean Query logs
Web documents English ODQA No SQuAD 1.1 (Rajpurkar et al., 2016) 108K
Extractive Crowd-sourced Wikipedia English MRC No SQuAD 2.0 (Rajpurkar et al.,
2018) 158K Extractive Crowd-sourced Wikipedia English MRC Yes NaturalQuestions
(Kwiatkowski et al., 2019) 323K Extractive, Boolean Query logs Wikipedia
English ODQA Yes CNN/Daily Mail (Nallapati et al., 2016) 1M Cloze
Automatically Generated News English MRC No NewsQA (Trischler et al., 2016)
119K Extractive Crowd-sourced News English MRC Yes ArchivalQA (Wang et al.,
2022) 532K Extractive Automatically Generated News English ODQA No ParSQuAD
(Abadani et al., 2021) 65k Extractive Translated SQuAD Wikipedia Persian MRC
No PersianQuAD (Kazemi et al., 2022; Mozafari et al., 2022) 20k Extractive
Crowd-sourced Wikipedia Persian ODQA No Arabic-SQuAD (Mozannar et al., 2019b)
48,344 Extractive Translated SQuAD Wikipedia Arabic MRC No ARCD(Mozannar et
al., 2019b) 1,395 Extractive Crowd-sourced Wikipedia Arabic MRC No ArabiQA
(Benajiba et al., 2007) 200 Extractive Crowd-sourced Wikipedia Arabic MRC No
DefArabicQA (Trigui et al., 2010) 50 Extractive Crowd-sourced Wikipedia and
Googles search Arabic ODQA No Translated TREC and CLEF (Abouenour et al.,
2010) 2,264 Extractive Translated news articles scientific papers and Web
pages. Arabic ODQA No DAWQUAS (Ismail and Homsi, 2018) 3,205 Extractive
Automatically Generated Web documents Arabic ODQA No QArabPro (Akour et al.,
2011) 335 Extractive Crowd-sourced Wikipedia Arabic MRC No ArabicaQA 87K
Extractive Crowd-sourced Wikipedia Arabic MRC&ODQA Yes

Table 2. Basic Statistics of ArabicaQA.

Dataset Training Set Development Set Test Set MRC (with answers) 62,186 13,483
13,426 MRC (unanswerable) 2,596 561 544 Open-Domain‡ 62,057 13,475 13,414
Open-Domain† 58,676 12,715 12,592

##  2\. Related Work

###  2.1. Open domain QA Benchmarks

The field of open-domain QA has seen significant advancements in recent years,
largely due to the introduction of various QA benchmarks. One of the
pioneering datasets in this field was the Stanford QA Dataset (SQuAD) 1.1
(Rajpurkar et al., 2016). SQuAD 1.1 offers 108K question-answer pairs derived
from Wikipedia articles. The dataset has been instrumental in the development
of advanced Machine Reading Comprehension (MRC) models. Its extension, Open-
SQuAD, has further contributed to the field by incorporating unanswerable
questions, thereby enhancing the complexity of the tasks and pushing the
boundaries of what QA systems can achieve. NarrativeQA (Kočiskỳ et al., 2018),
has taken a distinct approach by using summaries of movies and books to create
its question-answer pairs. This has diversified the types of datasets
available for training and evaluating QA systems, allowing exploration of
different types of questions and answers.

The use of search engine query logs has also been a key development in the
field. Datasets like MS MARCO (Nguyen et al., 2016), HotpotQA (Yang et al.,
2018), and Natural Questions (NQ) (Kwiatkowski et al., 2019) use queries from
Bing and Google and accompany web documents and Wikipedia pages as evidence.
The emergence of datasets like CNN/Daily Mail (Nallapati et al., 2016),
WhoDidWhat (Onishi et al., 2016), and ReCoRD (Zhang et al., 2018) has
propagated cloze-style tasks, which differ from traditional QA formats. These
tasks involve filling in the blanks in a given text, expanding the scope of
tasks that QA systems can handle. NewsQA (Trischler et al., 2016), another
notable dataset, focuses on MRC sourcing answers from CNN news articles. This
marked a category of temporal document collections in MRC, which allow QA
systems to handle time-sensitive questions related to past news articles.
ArchivalQA (Wang et al., 2022) dataset stands out here in terms of scale and
scope. It covers a wider and older range of news articles and is designed
specifically to handle questions related to historical events and long-term
trends, providing a unique challenge for QA systems.

In the context of Arabic datasets, Shaheen et al. (Shaheen and Ezzeldin, 2014)
and Atef et al. (2020) have made significant advancements. Shaheen et al.’s
work includes an analysis of system components like question analysis and
passage retrieval, emphasizing the unique linguistic features of Arabic and
the need for specialized approaches in this field. AQAD developed by Atef et
al. (2020) is designed to cover Arabic question-answer pairs for MRC. It
consists of 17,911 question-answer pairs extracted from 3,381 paragraphs from
299 Wikipedia articles, with 35% of the questions being unanswerable.

###  2.2. Machine Reading Comprehension datasets

The use of data to enhance reading comprehension has a long history, beginning
with the work of Hirschman et al. (1999). They developed a dataset consisting
of 600 questions from reading comprehension tests typically administered to
students between the 3rd and 6th grades. This initial effort, which used a
pattern-matching baseline, was later improved through a rule-based system
(Riloff and Thelen, 2000). MCTest dataset (Richardson et al., 2013), which was
created by crowdworkers and contains 660 stories, each accompanied by four
questions and four possible answers was created in 2013. The complexity of
MCTest lies in its reliance on commonsense reasoning and the need to
synthesize information across multiple sentences. The SQuAD (Rajpurkar et al.,
2016) is one of the most well-known MRC datasets. It offers a large number of
question-answer pairs derived from Wikipedia articles, providing a rich
resource for training and evaluating MRC systems. The CoQA dataset (Reddy et
al., 2019) contains 127k questions with answers, obtained from 8k
conversations about text passages from seven diverse domains. Each dialogue in
CoQA contains multiple questions and answers, providing a rich resource for
training and evaluating QA systems. The SciMRC dataset (Zhang et al., 2023b)
includes perspectives designed for beginners, students, and experts. Having
been constructed from 741 scientific papers it contains 6,057 question-answer
pairs where the different perspectives (questions for beginners, students, and
experts) contain 3,306, 1,800, and 951 QA pairs, respectively.

In the field of Arabic Machine Reading Comprehension (MRC), various approaches
have been used to tackle different types of questions. For instance, Azmi and
Alshenaifi (2016) worked on answering “why” questions using classic
Information Retrieval (IR) methods and rhetorical structure theory. Their work
was evaluated on a set of 100 questions, demonstrating the application of
traditional methods in MRC. The DefArabicQA (Trigui et al., 2010) focuses on
definition questions and employs an answer ranking module based on word
frequency. QArabPro (Akour et al., 2011) represents a significant step in
Arabic MRC, using a rule-based system and achieving high accuracy on a set of
questions from Wikipedia. The SemEval tasks of 2015, 2016, and 2017 introduced
community QA in Arabic, involving tasks like ranking paragraphs, questions,
and multiple answers in order of relevance. Arabic SQuAD (Mozannar et al.,
2019a), a machine-translated version of the English SQuAD dataset, has become
a key resource in Arabic MRC, despite the challenges posed by the translation.
With its large number of questions and wide coverage of topics, it represents
a significant step in developing neural machine reading comprehension
benchmarks for Arabic.

In this context, the objective of our paper is to develop ArabicaQA, a
comprehensive, large-scale, crowd-sourced dataset for Arabic QA. This dataset,
derived from a diverse collection of Standard Arabic documents, aims to foster
the advancement of MRC and Open-Domain QA systems for Arabic language
resources. Furthermore, ArabicaQA is designed to use with the advancements in
LLMs, offering a robust platform for benchmarking and enhancing these models
specifically for the Arabic language. In addition to ArabicaQA, we introduce
AraDPR, a novel dense passage retrieval model tailored for Arabic text. AraDPR
addresses specific challenges inherent to Arabic NLP, marking a significant
stride in Arabic text retrieval. This model, when used in conjunction with
ArabicaQA, provides a complete solution for Arabic question answering and
information retrieval, pushing the boundaries of current capabilities. In
Table 1, we highlight the differences between ArabicaQA and other related
datasets. 11footnotetext: †Human annotated 22footnotetext: ‡Converted from MRC
to Open-domain using an automatic script: https://github.com/deepset-
ai/haystack/blob/v1.24.x/haystack/utils/squad_to_dpr.py It leverages the
Haystack library to facilitate the conversion, utilizing Elasticsearch or
FAISS for document storage and BM25 or Dense Passage Retriever for retrieving
contexts.

Figure 2. The workflow of the QA Dataset Generation Framework integrates human
expertise across its stages. Starting with the Article Selection Module, where
experts select relevant articles, the process moves to Question Generation,
with human-crafted QA pairs. The Filtering Module then involves expert review
to discard low-quality pairs, followed by the Classification Module where
pairs are categorized into ’Concise Answer’ and ’Elaborate Answer’ types based
on human assessment. In the Open Domain Module, these curated pairs are
compiled into the final QA dataset, which later undergoes a Human Evaluation
phase for assessing answerability, relevance, clarity, and fluency.

##  3\. Methodology

In this section, we detail the methodology used to create ArabicaQA. The
development process is divided into five primary modules: Article Selection,
Question Generation, Filtering/Transforming, Human Answer Categorization, and
ODQA Annotation Module. These modules, illustrated in Figure 2, were
meticulously designed to generate high-quality, relevant question-answer pairs
in Modern Standard Arabic that mirror real-world scenarios and information-
seeking behaviors.

###  3.1. Article Selection Module

The creation of our dataset, ArabicaQA, starts with the selection of source
articles. We chose articles from Arabic Wikipedia as our primary source. The
articles were chosen randomly to prevent bias towards certain topics. Each
article was examined carefully to ensure it adhered to our length criteria,
with a minimum of 20 words and a maximum of 500 words per paragraph. This word
limit was set to provide enough context for generating questions while
preserving the focus and coherence of the content. We employed 4 proficient
Arabic-speaking crowdworkers for this task. Each crowdworker, who is
proficient in Arabic, was allocated a subset of these articles. The
crowdworkers’ responsibility was to generate question-answer pairs based on
the paragraphs, ensuring that every question was promptly grounded in the
provided content.

###  3.2. Question Generation Module

Question generation is the base of our dataset creation process. To maintain
high standards, crowdworkers were provided with a detailed structure and
guidelines for creating questions. We employed 10 Arabic-speaking crowdworkers
for this task. These guidelines emphasized the importance of generating fact-
seeking, clear, and unambiguous questions that could be answered with entities
(people, organizations, locations, etc.) or explanatory information directly
from the text.

Key aspects of the guidelines included:

  1. (1)

Clarity and Precision: Questions should be understandable without additional
interpretation and should avoid any assumptions.

  2. (2)

Reformulation: Avoid copying text verbatim from the passage. Questions should
use synonyms and varied sentence structures to ensure they are distinct from
the answer text.

  3. (3)

Natural Language Use: Questions should reflect natural language inquiry as if
asking another person for information.

  4. (4)

Answer Formatting: Concise answers are meant to be short and to the point,
while Elaborate answers could include whole sentences or text passages, but
not excessively long (no more than 8-10 sentences).

  5. (5)

Inclusion of Imperfections: Grammatical errors or misspellings, if found, were
decided to be included, reflecting the natural variance in user queries.

###  3.3. Filtering Module

The final step in our dataset creation process was the filtering and
transforming module, overseen by experts in the Arabic language. This phase
involved a thorough review of the generated question-answer pairs. Any
questions or answers that were inappropriate, ambiguous, or did not meet the
set guidelines were sent back to the annotators for revision.

This iterative process of generation, review, and refinement was crucial in
ensuring the quality and reliability of our dataset. It allowed us to remove
any inconsistencies, inaccuracies, or biases in questions and answers.

Elaborate Answer \- مع أي أندية لعب فرناندو بيليغرينو خلال مسيرته الاحترافية؟
لعب مع أتلتيكو أتلانتا وأرسنال ساراندي وانيستتوتو وبانفيلد وجيمناسيا لابلاتا
وديفنسا خوستيكا وريفر بليت وفيرو كاريل أويستي ونادي ساليرنيتانا
English Translation:

> Q: With which clubs did Fernando Pellegrino play during his professional
> career?
> A: He played with Atlético Atlanta, Arsenal de Sarandí, Instituto, Banfield,
> Gimnasia La Plata, Defensa y Justicia, River Plate, Ferro Carril Oeste, and
> Salernitana

Concise Answer في أي مدينة وُلد ساي مين تون؟
\- تاونغيي
English Translation:

> Q: In which city was Sai Min Tun born?
> A: Taunggyi

Figure 3. Elaborate/Concise answer examples

###  3.4.  Answer Categorization

An important component of our dataset construction process is the
categorization of answers based on an evaluation by language experts. This
step ensures that each question is paired with an appropriately categorized,
elaborate, or concise answer based on its content and complexity. This
classification is important for creating a dataset that can effectively train
and evaluate QA systems, particularly those designed for Arabic language
processing, in recognizing and generating responses of varying lengths and
detailedness levels. Fig. 3 shows samples from the elaborate and concise
answer questions.

#### Elaborate Answer Criteria

After a thorough examination by our Arabic linguistics experts, the dataset
includes 39,937 questions categorized under the Elaborate Answer Criteria.
These questions are identified for their depth and comprehensiveness,
extending beyond mere length to provide a full understanding of the query.
This category is representative of questions necessitating detailed
explanations or contextual information for satisfactory answers. Our experts’
analysis ensures each answer aligns with the question, offering a
comprehensive perspective where necessary, thus maintaining the relevance and
depth of the responses.

#### Concise Answer Criteria

Conversely, the dataset features 52,658 questions classified as having concise
answers. This categorization is tailored to responses that are succinct and to
the point, typically involving entities, dates, names, or other specific
details that directly address the query. In instances where a concise answer
is unfeasible or the question doesn’t lend itself to a concise response,
annotators are instructed to note the absence of a possible short answer,
emphasizing our focus on accurately capturing the essence of the query-
response dynamic in Arabic language processing.

###  3.5. ODQA Annotation Module

The Open Domain QA Annotation Module, the final part of the ArabicaQA dataset
creation, focuses on annotating broad questions applicable to be asked against
a document collection. This process begins with annotators identifying
potential open domain questions that span various subjects, followed by
conducting internet searches to ensure these questions can be realistically
answered with available online information. Each question’s answer is verified
for accuracy and relevance, undergoing quality assessment for clarity and
coherence. Issues identified at any stage require a revision, to maintain the
dataset’s high standards.

##  4\. Dataset Analysis

###  4.1. Dataset Statistics

ArabicaQA is divided into three parts: the training set (70% of the dataset,
with 62,186 answerable and 2,596 unanswerable questions), the development set
(15% of the dataset, with 13,483 answerable and 561 unanswerable questions),
and the test set (also 15% of the dataset, with 13,426 answerable and 544
unanswerable questions).

(a) Questions and Answers NER  Figure 4. Distribution of entity types in the
dataset Figure 5. Bi-gram Frequencies from ArabicaQA Questions

For the open-domain tasks, the distribution is slightly different, with 58,676
for training, 12,715 for development, and 12,592 for testing. More detailed
dataset statistics are presented in Table 2. Also, we conducted named entities
analysis for both the questions and answers of our dataset. Named entities
refer to specific, identifiable items like dates, names, or locations.
Identifying these entities helps in understanding the types of questions our
model is capable of answering. As shown in Fig. 4, we observed various
entities such as dates, people, organizations, locations, products,
competitions, events, diseases, and prices. For the answers, the most common
entities are dates, people, organizations, and locations. The most frequently
mentioned entities for the questions are people, locations, organizations, and
products.

This entity recognition is based on Arabic BERT (Antoun et al., 2020). The NER
Arabic BERT was trained on a corpus of 378,000 tokens (14,000 sentences)
collected from the Web and manually
annotated222https://huggingface.co/hatmimoha/arabic-ner. The results on a
validation corpus of 30,000 tokens point to F-1 of $87$%.

Fig. 5 also presents the distribution of frequent trigram prefixes. An
interesting aspect of our trigram analysis is the prominence of ما, which
accounts for over $50$% of the trigram prefixes. In Arabic, ما is a
multifaceted term that can take on multiple meanings depending on the context
and the words that follow it. While ما translates to ’what’ in English, its
usage in Arabic is more complex and varied. For instance, in the phrase ما هو
عدد, it means how many, indicating a question about quantity. In another
context, it can be used in a statement such as ما هي اللغة القديمه للمصريين ؟,
which means What is the ancient language of the Egyptians?, It is crucial to
note that ما in Arabic is context-dependent and does not always directly
translate to what in English. Its frequent occurrence in our dataset is not
just a reflection of its common usage but also highlights the complexity and
context-dependent nature of Arabic syntax and semantics.

###  4.2. Human Evaluation

In order to validate the quality of the question-answer pairs generated for
our dataset, we followed the human evaluation strategy similar to (Wang et
al., 2022). We randomly selected 1,000 pairs from our dataset, each
accompanied by their original context. Four experts in the Arabic language
were then recruited to assess each question-answer pair. The experts, all with
proficiency in Arabic linguistics, were instructed to assign ratings on a
scale from 1 (very bad) to 5 (very good). The criteria for evaluation were as
follows:

  * •

Fluency: Assesses the grammatical correctness and readability of a question.

  * •

Answerability: Determines if the provided answer correctly answers the
question.

  * •

Relevance: Evaluates whether the question relates to the content of the given
passage.

  * •

Non-ambiguity: Judges if the question is formulated unambiguously.

Each expert’s scores were collected, and an average score for each criterion
was computed to reflect the overall evaluation. The results are presented in
Table 3.

Table 3. Human Evaluation Scores of the Question-Answer Pairs Criteria | Fluency | Answerability | Relevance | Non-ambiguity
---|---|---|---|---
Expert 1 | 4.908 | 4.735 | 4.730 | 4.754
Expert 2 | 4.886 | 4.697 | 4.768 | 4.650
Expert 3 | 4.638 | 4.636 | 4.636 | 4.586
Expert 4 | 4.699 | 4.150 | 4.638 | 4.157
Average | 4.782 | 4.554 | 4.693 | 4.536

As indicated in Table 3 the average scores were commendable across all
metrics. The high scores, particularly in fluency and relevance, underline
high-quality content in our dataset for the Arabic language. Notably, the non-
ambiguity scores are also relatively high, suggesting that the majority of the
questions were clear and direct.

###  4.3. Difficult/Easy Questions

We describe how the categorization of questions based on their difficulty
levels. The Anserini IR toolkit, an open-source platform, was employed in
conjunction with the BM25 ranking
function333https://github.com/castorini/anserini from Arabic Wikipedia to
facilitate this process. The dataset was divided into two subsets: easy and
difficult. Questions are classified as ‘easy’ if the paragraphs used to
formulate the questions are found within the top 10 documents retrieved.
Conversely, if the paragraphs do not appear within these top 10 documents, the
questions are deemed ‘difficult’. Fig. 6 illustrates the distribution of the
dataset, which is ranked using the BM25 method. The breakdown of the dataset
is as follows: Training set: 27,640 easy and 34,417 difficult
questions.Development set: 5,933 easy and 7,542 difficult questions Test set:
5,820 easy and 7,594 difficult questions.

Figure 6. Question Difficulty Distribution

##  5\. Models

In this section, we describe the methodologies and models employed in our
research. We will discuss baseline models used for MRC, the AraDPR model, and
the LLMs.

###  5.1. MRC

The Term Frequency-Inverse Document Frequency (TF-IDF ) (Mozannar et al.,
2019a) Reader is one of our baseline models. It is a simple yet still
relatively effective non-learning algorithm that uses 4-gram features to find
relevant documents. The Embedding Reader (Joulin et al., 2016) is another
baseline model we use. Unlike the TF-IDF Reader, Embedding Reader uses the
fast-text embeddings444https://fasttext.cc/docs/en/crawl-vectors.html. It
calculates the cosine similarity between the embeddings of documents and
queries to find the most relevant documents. AraBERT (Antoun et al., 2020) is
an advanced model we implemented for our MRC approach. AraBERT is a variant of
BERT, specifically pre-trained on Arabic text. It leverages the power of
transformer-based architectures to understand the context of words in a
sentence, making it highly effective for tasks like MRC. Multilingual RoBERTa
(Liu et al., 2019) is another advanced model we use. Like AraBERT, RoBERTa is
a variant of BERT, but it is known for its high effectiveness. RoBERTa was
trained on more data and for more iterations than BERT, leading to improved
performance. It is particularly effective for tasks that require understanding
the context of words in a sentence, such as MRC.

###  5.2. AraDPR: Dense Retrieval for Arabic

We focus now on Arabic language retriever for open-domain QA. The model that
we built, AraDPR, is based on Bi-encoder framework within a monolingual
setting. The efficacy of retrieval models using multilingual transformers has
been well recognized, especially considering their high generalization across
languages. This capability is critical, as it suggests the probability of
leveraging cross-lingual knowledge to enhance retrieval systems in languages
such as Arabic, which presents its unique set of linguistic challenges. In the
lineage of pre-trained transformer models, mBERT and AraBERT emerged as
influential models, benefiting from the concept of multilingual embeddings by
being trained on a diverse set of languages. In our methodology, we adapt the
architecture of DPR (Dense Passage Retrieval), a seminal model in dense
retrieval, same as Karpukhin et al. (Karpukhin et al., 2020), to suit the
Arabic language. DPR computes embeddings for a query $q$ and a passage $p$
through separate encoders, $E_{Q}$ and $E_{P}$, which are typically
initialized with English BERT’s embeddings. The core of the retrieval
mechanism is the computation of a similarity score, $s_{q,p}$, as the dot
product of the embeddings:

$\displaystyle
s_{q,p}\mathrel{\ensurestackMath{\stackon[1pt]{=}{\scriptstyle\Delta}}}\text{sim}(q,p)=E_{Q}(q)^{T}E_{P}(p)$

DPR is trained with a contrastive loss function, which aims to distinguish
between relevant (positive) and non-relevant (negative) passages for a given
query:

$\displaystyle
L(q,p^{+},\\{p^{-}\\})=-\log\left(\frac{\exp(s_{q,p^{+}})}{\exp(s_{q,p^{+}})+\sum_{p^{-}}\exp(s_{q,p^{-}})}\right),$

where $p^{+}$ denotes the positive passage associated with the query $q$, and
the set $\\{p^{-}\\}$ comprises multiple negative passages. In retrieval
tasks, it is common to have explicit positive examples, while negative
examples need to be chosen from a large pool. For instance, in our ArabicaQA
dataset, passages relevant to a question might be provided, or they can be
identified using the answer. All other passages in the collection, although
not explicitly specified, are deemed irrelevant by default. The selection of
negative examples, often an overlooked aspect, can be crucial for training a
high-quality encoder. We use the top passages returned by
BM25555https://github.com/deepset-ai/haystack that do not contain the answer
but match most question tokens. This approach ensures a balanced and effective
training process for our models.

Given a collection of $M$ text passages, AraDPR aims to index these passages
in a low-dimensional, continuous vector space. This indexing facilitates the
efficient retrieval of the top $k$ passages most relevant to a given input
question, aiding the reader module in real-time comprehension tasks. In our
experiments, $M$ represents the entire Arabic Wikipedia, which amounts to
approximately $4$ million articles.

Table 4. Performance Comparison on MRC Tasks Method | Dev | Test
---|---|---
EM | Recall | Precision | F1 | Contains | EM | Recall | Precision | F1 | Contains
TF-IDF Reader | 0.08 | 5.347 | 16.480 | 7.039 | 4.357 | 0.128 | 5.317 | 16.38 | 7.09 | 4.18
Embedding Reader | 0.16 | 8.307 | 14.063 | 8.265 | 5.788 | 0.23 | 8.08 | 13.74 | 7.99 | 5.91
RoBERTa | 0.29 | 21.32 | 5.31 | 7.50 | 15.43 | 0.37 | 21.93 | 5.41 | 7.67 | 16.17
AraBERT | 4.05 | 34.82 | 18.44 | 19.69 | 14.39 | 4.507 | 35.74 | 18.88 | 20.42 | 14.83

###  5.3. LLM Models

In this section, we describe the details of the Large Language Models (LLMs)
utilized in our research. We discuss four distinct models: Llama, Mistral,
Mixtral, Falcon, Qwen and GPT3.5. Each of these models brings unique strengths
to our research and has been chosen for specific reasons.

Llama666https://huggingface.co/meta-llama/Llama-2-7b-chat-hf, part of the
Llama 2 family of LLMs developed by Meta (Touvron et al., 2023), comprises
pre-trained and fine-tuned generative text models. These models, specifically
optimized for dialogue use cases, have shown superior performance on numerous
benchmarks. They employ supervised fine-tuning and reinforcement learning with
human feedback (Touvron et al., 2023) to align with human preferences for
helpfulness and safety.

Mistral777https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2, released
by Mistral AI, is known for its power and efficiency. It surpasses the Llama 2
13B on all benchmarks. (Jiang et al., 2023). It leverages instruction fine-
tuning, where the prompt should be surrounded by [INST] and [/INST] tokens.

Mixtral888https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1, another
innovation from Mistral AI, is a trained generative Sparse Mixture of Experts
(Jiang et al., 2023) that outperforms the Llama 2 70B model on most
benchmarks. The model leverages up to 45B parameters but only uses about 12B
during inference, leading to better inference throughput at the cost of more
vRAM.

GPT3.5999https://platform.openai.com/docs/models/gpt-3-5 is a fine-tuned
version of the GPT3 model with 175B parameters (Brown et al., 2020). GPT3.5
focuses on reducing the generation of toxic outputs. It uses 12 stacks of
decoder blocks with multi-head attention blocks. The training data for GPT3.5
is a diverse mix of texts from the internet, including books, articles, and
websites. The model was trained on data up published before December
2023101010https://help.openai.com/en/articles/8555514-gpt-3-5-turbo-updates.

Falcon111111https://huggingface.co/docs/transformers/main/model_doc/falcon is
a class of causal decoder-only models. The largest Falcon checkpoints have
been trained on >=1T tokens of text, with a particular emphasis on the
RefinedWeb corpus (Penedo et al., 2023). Falcon architecture is modern and
optimized for inference, with multi-query attention and support for efficient
attention variants like FlashAttention (Dao et al., 2022).

PPLX121212https://www.searchenginejournal.com/perplexity-introduces-online-
llms-with-real-time-information/502523/, introduced by Perplexity, is a new
online LLM that leverages the internet for real-time data. The PPLX models,
pplx-7b-online and pplx-70b-online, build on top of the open-sourced
mistral-7B and llama2-70B models and have been specifically fine-tuned by
Perplexity on diverse, high-quality datasets to optimize for helpfulness and
factuality. These models can access the latest information from the internet
to provide up-to-date responses.

Qwen131313https://github.com/QwenLM/Qwen, proposed by Alibaba Cloud, is a
comprehensive language model series that encompasses distinct models with
varying parameter counts. It includes Qwen (Bai et al., 2023), the base
pretrained language models, and Qwen-Chat, the chat models fine-tuned with
human alignment techniques. The Qwen model series spans from 1.8 billion
parameters to 72 billion parameters.

###  5.4. Evaluation Metrics

#### Evaluation of MRC and LLMs:

We assess the performance of QA using several metrics, including exact match
(EM), precision, token recall, answer string containment, and F1 score. These
are standard measures widely used in QA research. However, given that LLMs
often generate verbose answers, many standard QA metrics may not be well-
suited for evaluating answer quality. For instance, the exact match will
almost always be zero due to the presence of non-ground-truth tokens, and the
F1 score will be penalized by other potentially useful tokens. To address
this, we use a set of model-agnostic metrics, namely token recall and answer
string containment (Adlakha et al., 2023; Liu et al., 2023; Mallen et al.,
2023).

#### Retriever Evaluation

The evaluation of the retriever component in our research employs the top-K
retrieval accuracy metric (Sachan et al., 2022). This metric calculates the
fraction of questions for which at least one passage in the top-K retrieved
passages contains a span matching the human-annotated answer. The formula is
given by:

(1) $\text{Retriever Accuracy@k}=\frac{\sum(\text{any(Correct@k)})}{\text{\\#
retrieved documents}}$

##  6\. Experimental Results

The results of our experiments on the ArabicaQA dataset provide insights into
the performance of various models in the domain of Machine Reading
Comprehension (MRC), Language Model in-context learning, and information
retrieval.

###  6.1. MRC Result

Table 4 presents a comparison of the performance of different methods on MRC
tasks, evaluated on both the development (Dev) and test sets. The methods
include TF-IDF Reader, Embedding Reader, Roberta, and ARaBERT.

TF-IDF Reader and Embedding Reader show a relatively low performance across
all metrics. This is expected as these models rely on simpler algorithms
compared to deep learning-based models. RoBERTa shows a significant
improvement in Exact Match (EM) and F1 scores, indicating its better
understanding of context compared to the more basic methods. ARaBERT,
specifically tailored for the Arabic language, demonstrates the highest
performance across almost all metrics. Its superior understanding of Arabic
nuances and context is evident in the results.

In Table 5, the performance of the LLMs is evaluated in terms of in-context
learning for MRC tasks. Falcon shows remarkable performance in terms of EM and
Contains scores. Its high capacity (180B parameters) plays a significant role
in achieving this result. GPT3.5 and PPLX display moderate results, with PPLX
showing a notably high Recall but lower EM and Precision. Qwen, LLama2,
Mistral, and Mixtral demonstrate varying degrees of performance, with none
dominating across all metrics. This indicates that the model’s size (in terms
of parameters) matters. In Fig. 7, the structure of the prompt is presented in
both Arabic and English. In the prompt, ’Context’ is replaced with a passage
of text, and ’Question’ is a query related to that passage.

Figure 7. Prompt Structure in Arabic and English Translation Table 5. In-
Context Learning with LLMs for MRC

Method Parameter Test EM Recall Precision F1 Contains Mistral 7B 0.067 34.80
6.110 9.311 22.28 Qwen 14B 0.185 15.81 6.86 8.533 7.862 Mixtral 8x7B 0.078
33.23 7.228 10.39 14.00 LlaMA2 70B 0.0 16.80 2.233 16.80 7.11 PPLX 70B 0.014
69.52 13.06 20.17 40.23 GPT3.5 175B 0.164 32.16 24.30 32.16 32.161 Falcon 180B
0.613 46.94 15.57 20.93 24.79

Table 6. Retriever Module Performance Comparison Method | Type | Dev | Test
---|---|---|---
1 | 10 | 20 | 50 | 100 | 1 | 10 | 20 | 50 | 100
TF-IDF | Unigram | 10.96 | 30.25 | 37.68 | 47.87 | 55.78 | 10.33 | 29.11 | 36.09 | 46.42 | 54.68
TF-IDF | Hierarchical | 10.97 | 30.44 | 37.27 | 47.91 | 56.48 | 10.43 | 29.73 | 37.00 | 47.08 | 55.77
TF-IDF | Bigram | 13.80 | 39.73 | 48.96 | 51.34 | 55.87 | 14.35 | 40.86 | 46.87 | 51.71 | 55.36
BM25 | - | 27.60 | 44.10 | 49.20 | 55.70 | 60.60 | 28.70 | 43.40 | 48.20 | 54.60 | 59.30
DPR | multi-language BERT | 36.50 | 58.60 | 62.70 | 67.20 | 70.20 | 36.40 | 57.80 | 62.10 | 66.60 | 69.50
AraDPR | AraBERT | 36.30 | 59.60 | 64.10 | 69.30 | 72.50 | 36.10 | 58.40 | 63.40 | 68.60 | 71.90

###  6.2. Retriever Results

The performance of different retrieval methods is presented in Table 6, with
methods including various TF-IDF  configurations, BM25, and two versions of
DPR (multi-language BERT and AraBERT). TF-IDF in its various forms (Unigram,
Hierarchical, Bigram) shows the worst performance. The Bigram model generally
outperforms the Unigram and Hierarchical models. BM25 outperforms TF-IDF
methods, especially at lower values of K, which suggests its effectiveness in
quickly identifying relevant documents. DPR with multi-language BERT and
AraBERT both show superior performance over the other methods, with AraBERT
slightly leading. This indicates the effectiveness of dense retrieval methods
and the advantage of language-specific tuning (AraBERT).

###  6.3. Retrieval-augmented Generation

Retrieval-Augmented Generation (RAG) (Lewis et al., 2020; Abdallah and Jatowt,
2023), merges the capabilities of advanced language models and information
retrieval systems for response generation in question-answering scenarios. In
this approach, upon receiving a question, pertinent documents are initially
sourced from an extensive corpus. Subsequently, these documents serve as a
supplementary context for a language model. This model then crafts an answer,
taking into account both the initial query and the information from the
retrieved documents. In this experiment, we aim to evaluate the efficiency of
appending the retrieved context to LLMs for question answering. Table 7
evaluates the performance of LLMs (GPT3.5, Mixtral, LLama, Mistral, Qwen) when
provided with the first retrieved document for Open-domain QA tasks. GPT3.5
shows moderate performance, with a balance across all metrics. Mixtral and
LLama have lower scores, indicating a potential limitation in utilizing the
first retrieved document for generating accurate responses. Mistral and Qwen
show somewhat better performance, particularly in Recall and Contains scores,
suggesting their capability to use the retrieved documents effectively. In
this experiment, we used the same prompt as in Section 6.1

Table 7. Retrieval-augmented Generation with first retrieved document Result

Method Parameter Test EM Recall Precision F1 Contains Mistral 7B 0.0 24.46
5.35 8.01 8.47 Mixtral 8x7B 0.03 8.33 2.03 2.94 2.46 LLama 70B 0.0 9.63 1.45
2.35 4.10 Qwen 14B 0.25 29.94 12.72 16.08 15.17 GPT3.5 175B 0.037 36.80 13.77
18.32 17.53

##  7\. Dataset Use

ArabicaQA is a versatile resource for Natural Language Processing, especially
for tackling Arabic’s complex linguistic features in Machine Reading
Comprehension and Open-Domain Question Answering. Its diverse and extensive
content aids in developing models that grasp Arabic’s rich morphology and
syntax. ArabicaQA is instrumental for improving Information Retrieval systems,
and enhancing document relevance through a deep understanding of context,
making it a critical tool for advancing Arabic NLP technologies. ArabicaQA
could enhance passage ranking algorithms. This is particularly crucial for
Arabic, where the relevance of retrieved documents is heavily influenced by
contextual nuances.

Moreover, ArabicaQA is an ideal resource for training models in question
generation tasks. This field has seen limited progress in Arabic due to the
lack of specialized datasets (AL-Sarayreh et al., 2023; AlSaleh et al., 2021;
Boujou et al., 2021). By providing a substantial volume of question-answer
pairs, our dataset can contribute to advancing research in this area as well.
The development of NLP models for Arabic is fraught with challenges. Arabic’s
rich morphology results in a high degree of word form variability. This,
coupled with the language’s reliance on context for meaning, makes accurate
comprehension a formidable task for current models. Our dataset, with its
diverse range of texts and contexts, is specifically designed to address these
challenges. It provides a varied and rich corpus for training models that can
navigate the complexities of Arabic syntax and semantics. Given the detailed
and specific nature of our dataset, it also holds potential for educational
applications, particularly in teaching and evaluating knowledge in Arabic
language courses. The diverse content of the dataset can be used to stimulate
self-learning of students, especially in understanding the nuances and
complexities of Arabic.

To sum up, ArabicaQA stands as a scalable and substantial resource for
researchers aiming to contribute to Arabic language processing. The lack of
comprehensive and challenging datasets has been a major bottleneck in Arabic
NLP research. By introducing a dataset that is both extensive and tailored to
the unique aspects of Arabic. This is especially crucial for developing models
that can handle the complexities of Arabic, thereby contributing to the
broader goal of creating truly global and inclusive NLP technologies.

##  8\. Conclusion

This paper introduces ArabicaQA, a large dataset designed to boost the
development of Arabic NLP, specifically MRC and open-domain QA. The dataset is
unique, containing both answered and unanswered questions, as well as answers
that are concise or elaborate, making it a challenging and enriching resource
for testing and improving NLP models. ArabicaQA has been manually created and
is currently the largest dataset for Arabic QA. Our study evaluated various
models on the ArabicaQA and found that advanced models like AraBert and
Roberta, which are fine-tuned for Arabic, outperform traditional models. Large
Language Models (LLMs) like Falcon, GPT3.5, and PPLX showed varying success
levels in in-context learning for MRC tasks, highlighting the complexity of
language processing and the need for model-specific tuning.

Our research also confirmed that dense retrieval methods, such as DPR models,
are more effective than traditional retrieval methods. However, LLMs face
challenges in using retrieved documents to improve MRC accuracy, suggesting a
need for further research in this area. In conclusion, ArabicaQA is a
significant contribution to Arabic NLP research, providing valuable resources
and insights. Our hope is that this work will inspire further research and
development in Arabic NLP leading to more advanced and efficient language
processing systems.

Limitations We acknowledge the following limitations of our work. ArabicaQA is
focused on Modern Standard Arabic and is sourced from Wikipedia, which limits
its linguistic diversity and domain coverage. The dataset might also benefit
from incorporating a broader range of question complexities and addressing
potential annotation biases. Annotation biases introduced through crowd-
sourcing could also impact the dataset’s neutrality and quality, despite
rigorous review processes. Addressing these limitations forms a part of our
future work.

## References

  * (1)
  * Abadani et al. (2021) Negin Abadani, Jamshid Mozafari, Afsaneh Fatemi, Mohammd Ali Nematbakhsh, and Arefeh Kazemi. 2021.  ParSQuAD: Machine Translated SQuAD dataset for Persian Question Answering. In _2021 7th International Conference on Web Research (ICWR)_. 163–168.  https://doi.org/10.1109/ICWR51868.2021.9443126
  * Abdallah et al. (2023a) Abdelrahman Abdallah, Mahmoud Abdalla, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2023a.  Amurd: annotated multilingual receipts dataset for cross-lingual key information extraction and classification.  _arXiv preprint arXiv:2309.09800_ (2023).
  * Abdallah and Jatowt (2023) Abdelrahman Abdallah and Adam Jatowt. 2023.  Generator-retriever-generator: A novel approach to open-domain question answering.  _arXiv preprint arXiv:2307.11278_ (2023).
  * Abdallah et al. (2020) Abdelrahman Abdallah, Mahmoud Kasem, Mohamed A Hamada, and Shaymaa Sdeek. 2020.  Automated question-answer medical model based on deep learning technology. In _Proceedings of the 6th International Conference on Engineering & MIS 2020_. 1–8.
  * Abdallah et al. (2023b) Abdelrahman Abdallah, Bhawna Piryani, and Adam Jatowt. 2023b.  Exploring the state of the art in legal QA systems.  _Journal of Big Data_ 10, 1 (2023), 127.
  * Abouenour et al. (2010) Lahsen Abouenour, Karim Bouzouba, and Paolo Rosso. 2010.  An evaluated semantic query expansion and structure-based approach for enhancing Arabic question/answering.  _International Journal on Information and Communication Technologies_ 3, 3 (2010), 37–51.
  * Adlakha et al. (2023) Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2023.  Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering.  _CoRR_ abs/2307.16877 (2023).  https://doi.org/10.48550/ARXIV.2307.16877 arXiv:2307.16877
  * Akour et al. (2011) Mohammed Akour, Sameer Abufardeh, Kenneth Magel, and Qasem Al-Radaideh. 2011.  QArabPro: A rule based question answering system for reading comprehension tests in Arabic.  _American Journal of Applied Sciences_ 8, 6 (2011), 652–661.
  * AL-Sarayreh et al. (2023) Sallam AL-Sarayreh, Azza Mohamed, and Khaled Shaalan. 2023.  Challenges and Solutions for Arabic Natural Language Processing in Social Media. In _International conference on Variability of the Sun and sun-like stars: from asteroseismology to space weather_. Springer, 293–302.
  * Allam and Haggag (2012) Ali Mohamed Nabil Allam and Mohamed Hassan Haggag. 2012.  The question answering systems: A survey.  _International Journal of Research and Reviews in Information Sciences (IJRRIS)_ 2, 3 (2012).
  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023.  Falcon-40B: an open large language model with state-of-the-art performance.  (2023).
  * AlSaleh et al. (2021) Deem AlSaleh, Mashael Bin AlAmir, and Souad Larabi-Marie-Sainte. 2021.  SNAD arabic dataset for deep learning. In _Intelligent Systems and Applications: Proceedings of the 2020 Intelligent Systems Conference (IntelliSys) Volume 1_. Springer, 630–640.
  * Antoun et al. (2020) Wissam Antoun, Fady Baly, and Hazem Hajj. 2020.  Arabert: Transformer-based model for arabic language understanding.  _arXiv preprint arXiv:2003.00104_ (2020).
  * Atef et al. (2020) Adel Atef, Bassam Mattar, Sandra Sherif, Eman Elrefai, and Marwan Torki. 2020.  AQAD: 17,000+ arabic questions for machine comprehension of text. In _2020 IEEE/ACS 17th International Conference on Computer Systems and Applications (AICCSA)_. IEEE, 1–6.
  * Azmi and Alshenaifi (2016) Aqil M Azmi and Nouf A Alshenaifi. 2016.  Answering arabic why-questions: Baseline vs. rst-based approach.  _ACM Transactions on Information Systems (TOIS)_ 35, 1 (2016), 1–19.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.  Qwen Technical Report.  _arXiv preprint arXiv:2309.16609_ (2023).
  * Bashir et al. (2023) Muhammad Huzaifa Bashir, Aqil M Azmi, Haq Nawaz, Wajdi Zaghouani, Mona Diab, Ala Al-Fuqaha, and Junaid Qadir. 2023.  Arabic natural language processing for Qur’anic research: A systematic review.  _Artificial Intelligence Review_ 56, 7 (2023), 6801–6854.
  * Benajiba et al. (2007) Yassine Benajiba, Paolo Rosso, and Abdelouahid Lyhyaoui. 2007.  Implementation of the ArabiQA question answering system’s components. In _Proc. Workshop on Arabic Natural Language Processing, 2nd Information Communication Technologies Int. Symposium, ICTIS-2007, Fez, Morroco, April_. 3–5.
  * Bolotova et al. (2022) Valeriia Bolotova, Vladislav Blinov, Falk Scholer, W Bruce Croft, and Mark Sanderson. 2022.  A non-factoid question-answering taxonomy. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1196–1207.
  * Bolotova-Baranova et al. (2023) Valeriia Bolotova-Baranova, Vladislav Blinov, Sofya Filippova, Falk Scholer, and Mark Sanderson. 2023.  WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 5291–5314.
  * Boujou et al. (2021) ElMehdi Boujou, Hamza Chataoui, Abdellah El Mekki, Saad Benjelloun, Ikram Chairi, and Ismail Berrada. 2021.  An open access nlp dataset for arabic dialects: Data collection, labeling, and model construction.  _arXiv preprint arXiv:2102.11000_ (2021).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  _Advances in Neural Information Processing Systems_ 35 (2022), 16344–16359.
  * Darwish et al. (2021) Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R El-Beltagy, Wassim El-Hajj, et al. 2021\.  A panoramic survey of natural language processing in the Arab world.  _Commun. ACM_ 64, 4 (2021), 72–81.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Hirschman et al. (1999) Lynette Hirschman, Marc Light, Eric Breck, and John D Burger. 1999.  Deep read: A reading comprehension system. In _Proceedings of the 37th annual meeting of the Association for Computational Linguistics_. 325–332.
  * Ismail and Homsi (2018) Walaa Saber Ismail and Masun Nabhan Homsi. 2018.  Dawqas: A dataset for arabic why question answering system.  _Procedia computer science_ 142 (2018), 123–131.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023\.  Mistral 7B.  _arXiv preprint arXiv:2310.06825_ (2023).
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016.  Fasttext. zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_ (2016).
  * Kamath et al. (2019) Uday Kamath, John Liu, and James Whitaker. 2019.  _Deep learning for NLP and speech recognition_. Vol. 84.  Springer.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_ (2020).
  * Kazemi et al. (2022) Arefeh Kazemi, Jamshid Mozafari, and Mohammad Ali Nematbakhsh. 2022.  PersianQuAD: The Native Question Answering Dataset for the Persian Language.  _IEEE Access_ 10 (2022), 26045–26057.  https://doi.org/10.1109/ACCESS.2022.3157289
  * Kočiskỳ et al. (2018) Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018.  The narrativeqa reading comprehension challenge.  _Transactions of the Association for Computational Linguistics_ 6 (2018), 317–328.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019\.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ 7 (2019), 453–466.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020\.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ 33 (2020), 9459–9474.
  * Liu et al. (2023) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023.  Lost in the Middle: How Language Models Use Long Contexts.  _CoRR_ abs/2307.03172 (2023).  https://doi.org/10.48550/ARXIV.2307.03172 arXiv:2307.03172
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 9802–9822.  https://doi.org/10.18653/V1/2023.ACL-LONG.546
  * Mozafari et al. (2022) Jamshid Mozafari, Arefeh Kazemi, Parham Moradi, Mohammad Ali Nematbakhsh, et al. 2022\.  PerAnSel: a novel deep neural network-based system for Persian question answering.  _Computational Intelligence and Neuroscience_ 2022 (2022).
  * Mozannar et al. (2019a) Hussein Mozannar, Karl El Hajal, Elie Maamary, and Hazem Hajj. 2019a.  Neural Arabic question answering.  _arXiv preprint arXiv:1906.05394_ (2019).
  * Mozannar et al. (2019b) Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019b.  Neural Arabic Question Answering. In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_. Association for Computational Linguistics, Florence, Italy, 108–118.  https://doi.org/10.18653/v1/W19-4612
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016\.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  _arXiv preprint arXiv:1602.06023_ (2016).
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset. In _CoCo@ NIPS_.
  * Onishi et al. (2016) Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016.  Who did what: A large-scale person-centered cloze dataset.  _arXiv preprint arXiv:1608.05457_ (2016).
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023.  The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only.  _arXiv preprint arXiv:2306.01116_ (2023).  arXiv:2306.01116 https://arxiv.org/abs/2306.01116
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  _arXiv preprint arXiv:1806.03822_ (2018).
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_ (2016).
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D Manning. 2019.  Coqa: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ 7 (2019), 249–266.
  * Richardson et al. (2013) Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013.  Mctest: A challenge dataset for the open-domain machine comprehension of text. In _Proceedings of the 2013 conference on empirical methods in natural language processing_. 193–203.
  * Riloff and Thelen (2000) Ellen Riloff and Michael Thelen. 2000.  A rule-based question answering system for reading comprehension tests. In _ANLP-NAACL 2000 workshop: reading comprehension tests as evaluation for computer-based language understanding systems_.
  * Sachan et al. (2022) Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022.  Improving Passage Retrieval with Zero-Shot Question Generation.  (2022).  https://arxiv.org/abs/2204.07496
  * Shaheen and Ezzeldin (2014) Mohamed Shaheen and Ahmed Magdy Ezzeldin. 2014.  Arabic question answering: systems, resources, tools, and future trends.  _Arabian Journal for Science and Engineering_ 39 (2014), 4541–4564.
  * Singh et al. (2017) Shashi Pal Singh, Ajai Kumar, Hemant Darbari, Lenali Singh, Anshika Rastogi, and Shikha Jain. 2017.  Machine translation using deep learning: An overview. In _2017 international conference on computer, communications and electronics (comptelix)_. IEEE, 162–167.
  * Soricut and Brill (2006) Radu Soricut and Eric Brill. 2006.  Automatic question answering using the web: Beyond the factoid.  _Information Retrieval_ 9 (2006), 191–206.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023\.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ (2023).
  * Trigui et al. (2010) Omar Trigui, Lamia Hadrich Belguith, and Paolo Rosso. 2010.  DefArabicQA: Arabic definition question answering system. In _Workshop on Language Resources and Human Language Technologies for Semitic Languages, 7th LREC, Valletta, Malta_. 40–45.
  * Trischler et al. (2016) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016.  Newsqa: A machine comprehension dataset.  _arXiv preprint arXiv:1611.09830_ (2016).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2022) Jiexin Wang, Adam Jatowt, and Masatoshi Yoshikawa. 2022.  ArchivalQA: A Large-scale Benchmark Dataset for Open-Domain Question Answering over Historical News Collections. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 3025–3035.
  * Wu and Chen (2015) Meiyin Wu and Li Chen. 2015.  Image recognition based on deep learning. In _2015 Chinese automation congress (CAC)_. IEEE, 542–546.
  * Yang et al. (2015) Yi Yang, Wen-tau Yih, and Christopher Meek. 2015.  Wikiqa: A challenge dataset for open-domain question answering. In _Proceedings of the 2015 conference on empirical methods in natural language processing_. 2013–2018.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  _arXiv preprint arXiv:1809.09600_ (2018).
  * Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018.  Record: Bridging the gap between human and machine commonsense reading comprehension.  _arXiv preprint arXiv:1810.12885_ (2018).
  * Zhang et al. (2023a) Xinyu Zhang, Kelechi Ogueji, Xueguang Ma, and Jimmy Lin. 2023a.  Toward best practices for training multilingual dense retrieval models.  _ACM Transactions on Information Systems_ 42, 2 (2023), 1–33.
  * Zhang et al. (2023b) Xiao Zhang, Heqi Zheng, Yuxiang Nie, Heyan Huang, and Xian-Ling Mao. 2023b.  SciMRC: Multi-perspective Scientific Machine Reading Comprehension.  _arXiv preprint arXiv:2306.14149_ (2023).