# Evening the Score: Targeting SARS-CoV-2 Protease Inhibition in Graph
Generative Models for Therapeutic Candidates

Jenna Bilbrey, Sutanay Choudhury, Neeraj Kumar
Pacific Northtwest National Laboratory
Richland, WA, USA
{jenna.pope, sutanay.choudhury<EMAIL_ADDRESS>
Logan Ward, Ganesh Sivaraman
Argonne National Laboratory
Lemont, IL, USA
{lward<EMAIL_ADDRESS>

###### Abstract

We examine a pair of graph generative models for the therapeutic design of
novel drug candidates targeting SARS-CoV-2 viral proteins. Due to a sense of
urgency, we chose well-validated models with unique strengths: an autoencoder
that generates molecules with similar structures to a dataset of drugs with
anti-SARS activity and a reinforcement learning algorithm that generates
highly novel molecules. During generation, we explore optimization toward
several design targets to balance druglikeness, synthetic accessability, and
anti-SARS activity based on IC50. This generative
framework111https://github.com/exalearn/covid-drug-design will accelerate drug
discovery in future pandemics through the high-throughput generation of
targeted therapeutic candidates.

##  1 Introduction

Discovering lead candidates for SARS-CoV-2 presents a challenge to the
scientific community, and systematically developing AI-based automated
workflows can accelerate the design of effective therapeutic compounds with a
target set of properties [You et al. (2018); Li et al. (2018); Huang et al.
(2020); Zhou et al. (2020); Horwood & Noutahi (2020); Khemchandani et al.
(2020); Chenthamarakshan et al. (2020); Hu et al. (2020); Sivaraman et al.
(2020); Joshi & Kumar (2021)]. The potential chemical space is composed of
over $10^{60}$ compounds that ideally need to be tested for the therapeutic
design [Reymond (2015)] and discovery before identifying a lead for a given
target protein of SARS-CoV-2. However, candidates with suitable activity
against specific proteins only narrows the search space to $10^{4}-10^{5}$
compounds. There is a need for rapid development of machine-learning (ML)
workflows that serve as a search-and-screen process of this reduced chemical
space. Candidate molecules generated by ML models are passed to downstream
verification via virtual high-throughput drug-protein binding techniques,
chemical synthesis, biochemical and biophysical assay, and finally clinical
trials [Batra et al. (2020); Smith & Smith (2020)]. In the current work, we
develop, evaluate, and compare two leading graph-generative approaches to
design candidate therapeutic compounds targeting SARS-CoV-2 protease
inhibition (Figure 1). This systematic comparison of two leading approaches
distinguishes our work in the context of the broad effort undertaken by the
research community in its quest for novel therapeutics (see section 2 for a
thorough review).

First, we train a junction-tree variational autoencoder (JT-VAE) [Jin et al.
(2018)] to generate molecules with similar structures to a curated dataset of
known drugs with anti-SARS activity. The trained JT-VAE is then used in
Bayesian optimization to generate novel candidates with targeted properties.
We next examine graph-based deep reinforcement learning (DQN) [Zhou et al.
(2019)] to generate candidates that are not constrained by their structural
proximity to known anti-SARS compounds. To benchmark the two graph generative
models, we use the same set of property scoring functions as optimization
targets.

Contributions. The goal of our study is to perform multi-objective
optimization to generate molecular candidates by considering important
bioactivity properties along with druglikeness and synthetic accessability. We
focus on pIC50 (the inverse log of the half maximal inhibitory concentration,
IC50), which is an experimentally measured property that captures the potency
of a therapeutic candidate towards a protease target, where higher values
indicate exponentially more potent inhibitors. Notably, pIC50 cannot be
accurately modeled through ab initio methods [Bag & Ghorai (2016); Sebaugh
(2011)]. Our ML workflow for the high-throughput generation of therapeutic
candidates with anti-SARS activity contains the following key components: a
surrogate model for pIC50 prediction, candidate generation via JT-VAE and DQN
models, and validation of top-ranking candidates against a Drug Target Binding
Affinity (DBTA) classifier [Huang et al. (2020)] to asses their potential
activity against SARS-CoV-2. Alongside this paper, we release the ML workflow
and a curated dataset of drug molecules with anti-SARS activity for use by the
community.

Figure 1: Depiction of workflow developed for this work. The anti-SARS
database is used to train a MPNN to predict pIC50 and the JT-VAE model. The
trained MPNN is used as the scoring function in both JT-VAE (top row) and DQN-
based molecular generation (bottom). Candidate molecules are screened by pIC50
($>$8) and validated by a Drug Target Binding Affinity classifier.

##  2 Background and Related Work

Biological factors in drug design. A variety of properties can be considered
for lead optimization [Copeland et al. (2006)]. Here we give an explanation of
the various properties and their ideal values. Lipinski’s Rule of Five is the
standard starting place when considering properties of potential oral drug
compounds [Lipinski (2004)]. The Rule of Five identifies that molecules with
no more than 5 hydrogen bond donors, no more than 10 hydrogen bond acceptors,
a molecular mass under 500 Da, and an octanol–water partition coefficient (log
P) below 5 are more likely to show pharmacological activity. LogP is a measure
of lipophilicity, which provides an understanding of the behavior of a drug in
the body. More recent analysis has shown that logP values between 1 and 3 may
be more appropriate considering the effect of logP on absorption,
distribution, metabolism, elimination, and toxicology (ADMET) properties
[Waring (2010)]. Though oral bioavailability is an important factor, a sole
focus on logP has the potential to screen out otherwise useful compounds
[Zhang & Wilkinson (2007)]. QED has been proposed as a more holistic
druglikeness metric [Bickerton et al. (2012)], which from 0 (low) to 1 (high).
However, druglikeness does not indicate the activity or effectiveness of a
drug towards a specific target. The half maximal inhibitory concentration
(IC50) provides a quantitative measure of the potency of a compound to inhibit
a specific biological process. IC50 is obtained by measurement, and no
universal ab initio method of computing its value exists. A number of methods
have been developed to approximate IC50, many based on QSPR and recently some
based on machine learning [Patankar & Jurs (2000); Bag & Ghorai (2016);
Armutlu et al. (2008); Bjerrum (2017)]. Similarity to known drugs is also an
important factor in drug discovery [Maggiora et al. (2014); Kumar & Zhang
(2018)], as is the ability to synthesize the molecule, which can be estimated
by the synthetic accessibility (SA) score – from 1 (easy) to 10 (difficult)
[Ertl & Schuffenhauer (2009)].

COVID-19-focused efforts. In response to the COVID-19 pandemic, researchers
have pushed to identify marketed drugs that can be repurposed for SARS-CoV-2
treatment [Zhou et al. (2020); Smith & Smith (2020); Gordon et al. (2020);
Zhavoronkov et al. (2020); Skwark et al. (2020); Mittal et al. (2020); Chang
(2020); Chen et al. (2020); Wu et al. (2020)]. Most of these the
COVID-19-focused efforts relied on molecular simulations and docking studies
to identify known drugs that interacted with 3-chymotrypsin-like main protease
(3CLpro), also called the main protease (Mpro), of SARS-Cov-2. Machine
learning (ML) approaches to drug repurposing were also reported. For instance,
Batra et al. (2020) applied an ML-based screening approach to find known
compounds with binding affinity to either the isolated SARS-CoV-2 S-protein at
its host receptor region or to the S-protein-human ACE2 interface complex.
Huang et al. (2020) developed DeepPurpose, a deep learning toolkit for drug
repurposing, with the goal of recommending known candidates with high binding
affinity to target amino acid sequences.

An array of de novo design approaches specifically targeting anti-SARS-CoV-2
drugs have also been reported. Hu et al. (2020) trained the GPT2 transformer-
based model to generate SMILES strings of potential drug candidates follow the
Lipinski’s ”Rule of Five”, optimizing towards 3CLpro–target interaction.
Chenthamarakshan et al. (2020) developed the generative modeling framework
CogMol which relies upon a varitional autoencoder sampled by Controlled Latent
attribute Space Sampling (CLaSS) to design candidates optimized toward binding
affinity. Born et al. (2020) amended their PaccMann RL approach, which coupled
reinforcement learning with variational autoencoders with reinforcement
learning, to generate molecules optimized towards binding affinity and
pharmacological toxicity predictors. Zhavoronkov et al. (2020) applied a
similar approach of using reinforcement learning to optimize a generative
model, but examined genetic algorithms, language models, generative
adversarial networks as well as autoencoders. Tang et al. (2020) applied a
deep Q learning approach with a structure-based optimization policy that
targets QED, the presence of specific fragments, and the presence of
pharmacophores. In a novel direction, Skwark et al. (2020) focused on the
design of proteins that bind to SARS-CoV-2 in place of ACE2 using
reinforcement learning.

Graph generative methods for molecule design. Heterogeneous graphs provide a
natural representation for small-molecule organic compounds, with nodes
representing atoms in the molecular structure and edges representing bonds
between the atoms [Weininger (1990)]. This approach motivated the exploration
of graph-generative models, such as graph convolutional policy networks [You
et al. (2018)], variational autoencoders [Jin et al. (2018); Samanta et al.
(2019); Jin et al. (2020)], and variants of deep reinforcement learning [Zhou
et al. (2019); Ståhl et al. (2019)], for the target-driven optimization of
drug molecules. In the current work, we develop, evaluate, and compare two
leading graph-generative approaches for candidate therapeutic compounds,
variational autoencoders and reinforcement learning, specifically targeting
molecules able to inhibit the function of SARS-CoV-2 proteins. Our focus is to
understand the distinction between the variational autoencoder and
reinforcement learning approaches in terms of similarity versus novelty in the
generated molecules, as well as the optimal approach to developing a wholistic
scoring function that accounts for both druglikeness and SARS-Cov-2 protease
inhibition.

##  3 Method

###  3.1 Surrogate Model for pIC50 Prediction

We trained a message-passing neural network (MPNN) [Scarselli et al. (2008);
Gilmer et al. (2017)] to predict pIC50 (the inverse log of IC50) for a given
molecular structure. Following the formalism of Gilmer et al. (2017), our
network is composed of message, update, and readout operations (eqns. 1-3) and
our choices for these functions are based on networks developed by John et al.
(2019) for polymer property prediction.

The original state of each atom ($h_{v}$) and bond ($\alpha_{vw}$) in our
molecule ($G$) is a 256-length vector with values defined by an embedding
table based on the atomic number and bond type (e.g., single, double,
aromatic). The states of these atoms are modified by eight successive
“message” layers. Each message layer uses a two-layer multi-layer perceptron
(MLP) with sigmoid activations to compute a message that uses the state of an
atom ($h_{v}$), the state of the neighboring atom ($h_{w}$) and the bond which
joins them ($\alpha_{vw}$). The atom and bond states are updated according to
the following equations:

$m^{t+1}_{v}=\sum_{w\in
Neighbors(v)}M_{t}(h^{t}_{v},h^{t}_{w},\alpha^{t}_{vw})$ (1)
$h^{t+1}_{v}=h^{t}_{v}+m^{t+1}_{v}$ (2)
$\alpha^{t+1}_{vw}=\alpha^{t}_{vw}+M_{t}(h^{t}_{v},h^{t}_{w},\alpha^{t}_{vw})$
(3)

The atom states output from the last layer ($h^{T}_{v}$) are used to predict
the pIC50 of the molecule using a ”readout” function ($R$):

$\hat{y}=R(h^{T}_{v}|v\in G)$ (4)

We examined several variants of the readout function in our study. We tested
both “molecular fingerprints,” where the states of each node are combined
before using a multi-layer perceptron (MLP) to reduce to compute pIC50, and an
“atomic contribution,” where we combine the nodes after MLP to compute a per-
node contribution to pIC50. We experimented with the use of five different
functions to reduce the atomic state/contributions to a single value for each
graph: summation, mean, maximum, softmax, and attention. The attention
functions are created by learning an attention map by passing the node states
through a MPNN. We tested all combinations of “molecular fingerprint” vs.
“atomic contribution” and the five readout functions, for a total of 10
networks, training each on network the same 90% of our pIC50 dataset and
comparing its performance on the withheld 10% of the data. We used an MPNN
that uses attention maps to reduce contributions from each atom to a single
pIC50 of a molecule in all subsequent experiments.

Junction-Tree Variational Autoencoder. We used a junction-tree variational
autoencoder (JT-VAE) [Jin et al. (2018)] to generate molecules with high
proximity to anti-SARS therapeutic molecules. The autoencoder generates novel
molecular graphs by laying a tree-structured scaffold over substructures in
the molecule, which are then combined into a valid molecule using a MPNN. JT-
VAE allows for the expansion of a molecular graph through the addition of
subgraphs, or “vocabulary” of valid components, derived from the training set
(Fig. 1). The subgraphs are used to encode a molecule into its vector
representation and decode latent vectors into valid molecular graphs. The use
of subgraphs maintains chemical validity at each step, while also
incorporating chemical units common to the training set. Chemical graphs
generated from the vocabulary are structurally similar to those in the
training set, which is a benefit when attempting to design molecules with
similar properties to known drugs. We omit the details here for brevity and
point the reader to Jin et al. (2018) for details on JT construction and
Kusner et al. (2017) for Bayesian optimization.

Deep Reinforcement Learning. We follow the Q-learning approach of Zhou et al.
for our deep reinforcement learning approach [Zhou et al. (2019)]. In this
setting, we cast the molecule generation problem as a Markov Decision Process
(MDP) [Mnih et al. (2013)] to learn a policy network $\pi$ that determines the
best sequence of actions to transform an initial molecular graph to a larger
graph with desirable properties in a step-by-step fashion (Fig. 1). At each
step, we enumerate all possible actions and select those which produce valid
molecules (e.g., respect valency rules). Next, we train a MLP to predict the
value of a certain action by passing the Morgan fingerprints [Rogers & Hahn
(2010)] as input. The MLP approximates the value of an action computed using
the Bellman equation, where the score of a state and the maximum score of the
subsequent state is multiplied by a decay factor. As established with other
Deep Q-Learning approaches, the addition of the value of the next state
increases the value of moves which will lead to higher future rewards.

SARS-CoV-2-specific Scoring Functions. The scoring functions described in this
section are used for both Bayesian optimization in the JT-VAE approach and
reward computation for the deep reinforcement-learning approach.

$logP^{P}(m)=logP(m)-SA(m)-cycle(m)$ (5) $QED^{P}(m)=QED(m)-SA(m)-cycle(m)$
(6) $pIC_{50}(m)=MPNN(m)$ (7)
$pIC_{50}+QED^{P}(m)=MPNN(m)+QED(m)-SA(m)-cycle(m)$ (8)

Following Jin et al. (2018), we first compute a score that penalizes the
partition coefficient between octanol and water (logP) by the synthetic
accessibility (SA) score (in which higher SA values are discouraged) and the
number of cycles with more than 6 atoms (eqn. 5). Considering that the
Quantitative Estimate of Druglikeness (QED) is a more comprehensive heuristic
than logP, we also use a similar scoring function composed of QED penalized by
the SA score and number of long cycles (eqn. 6). We then examine the utility
of a SARS-specific scoring function based on the pIC50 predicted by our MPNN
(eqn. 7). Finally, we examine a multi-objective scoring function that includes
both pIC50 and penalized QED (eqn. 8).

##  4 Experimental Analysis

The focus of our study is two fold: 1) Perform multi-objective optimization
for generating therapeutic candidates with activity towards protease target of
SARS-CoV-2, 2) Build novel yet unique compounds by optimizing the scoring
function including synthesizibility score. To this end, we are poised to
utilize best possible graph based ML models.

Dataset Preparation. We assembled and curated a protease dataset containing
molecules active against various protease in enzymatic assays from
experimental pharmacology databases, such as CheMBL, BindingDB, and ToxCat
[Bento et al. (2014)]. The dataset was filtered based on pIC50 activity and
potency. Molecules larger than 1,000 Dalton were removed along with non-drug
like molecules containing metals and polypeptides. In some instances,
duplicate entries were generated from multiple experimental studies, in which
case we used the mean pIC50. The resulting dataset contains SMILES strings and
experimental pIC50 values of 6,545 unique molecules.

Quantitative Characterization. We computed the metrics included in our scoring
functions for each molecule in the dataset. The highest pIC50 is 10.89, while
the lowest is 1.22. The most common pIC50 is 4.0, which is shared by 320
structures, and the vast majority of structures (91.5%) have pIC50 values
greater than 4.0. LogP values range from -10.36 to 16.65 in a near Gaussian
distribution with a mean of 3.70; 77.0% of all structures meet the requirement
of Lipinski’s Rule of 5 that logP be no greater than 5. QED values range from
0.01 to 0.94, while SA values range from 1.35 to 8.24, with 73.8% being below
4. We computed the Tanimoto similarity [Bajusz et al. (2015)] for all pairs of
compounds to gain insight into the structural diversity of molecules in our
dataset (Fig. S2). We observed that structures tend to become more similar to
their neighbors as pIC50 increases, indicating that compounds with high pIC50
values tend to be structurally similar, supporting the consideration of
molecular similarity during drug design and discovery.

Training pIC50 Surrogate Model. We found that limiting the pIC50 prediction to
contributions from only a few specific atoms in the molecule improved
performance. This improved performance can be explained by a physical
mechanism. The presence or absence of a specific chemical pattern in a
molecular structure (e.g., functional groups, substituents, aromatic rings)
controls binding of the molecule to a certain portion of a target protein,
while the other atoms in the molecule play a role in determining whether the
molecule will stay affixed at the target site. We hypothesize that such
contributions can have a critical effect in tuning the binding affinity, which
influences pIC50.

JT-VAE Setup. We trained the JT-VAE on our dataset for 8,300 iterations with
the following hyperparameters: hidden state dimension of 450, latent code
dimension of 56, and graph message passing depth of 3. To optimize towards the
specified scoring functions, we trained a sparse Gaussian process (SGP) to
predict a score given the latent representation learned by the JT-VAE and then
performed 10 iterations of batched Bayesian optimization (sampling = 50) based
on the expected improvement.

Table 1: Properties of the top-3 molecules generated using the specified
scoring function.

Scoring Function | pIC50 | QED | logP | SA Score
---|---|---|---|---
1st | 2nd | 3rd | 1st | 2nd | 3rd | 1st | 2nd | 3rd | 1st | 2nd | 3rd
logPP (JT-VAE) | $4.93$ | $4.57$ | $4.60$ | $0.45$ | $0.78$ | $0.71$ | $4.05$ | $4.27$ | $4.20$ | $1.70$ | $2.08$ | $2.08$
logPP (DQN) | $6.10$ | $8.17$ | $4.98$ | $0.04$ | $0.07$ | $0.11$ | $13.86$ | $12.64$ | $12.52$ | $3.59$ | $2.97$ | $2.93$
QEDP (JT-VAE) | $4.15$ | $4.23$ | $4.71$ | $0.91$ | $0.84$ | $0.91$ | $3.72$ | $2.19$ | $2.20$ | $1.80$ | $1.71$ | $2.25$
QEDP (DQN) | $6.80$ | $6.80$ | $6.80$ | $0.77$ | $0.77$ | $0.77$ | $3.46$ | $3.46$ | $3.46$ | $2.05$ | $2.05$ | $2.05$
pIC50 (JT-VAE) | $10.22$ | $10.07$ | $10.07$ | $0.15$ | $0.12$ | $0.12$ | $4.86$ | $3.80$ | $3.06$ | $4.52$ | $4.94$ | $4.98$
pIC50 (DQN) | $10.57$ | $10.56$ | $10.39$ | $0.09$ | $0.11$ | $0.41$ | $1.51$ | $2.03$ | $-0.44$ | $6.90$ | $6.71$ | $5.57$
pIC50+QEDP (JT-VAE) | $8.58$ | $5.98$ | $8.18$ | $0.83$ | $0.87$ | $0.70$ | $4.02$ | $3.37$ | $3.82$ | $1.93$ | $1.74$ | $1.50$
pIC50+QEDP (DQN) | $10.27$ | $10.27$ | $10.27$ | $0.80$ | $0.80$ | $0.80$ | $3.02$ | $3.02$ | $3.02$ | $2.90$ | $2.90$ | $2.90$

DQN Setup. Our DQN approach constructs molecules atom-by-atom and bond-by-bond
in a step-wise manner. Each episode starts with either a single atom or a pre-
seeded molecule and the RL agent is allowed up to 40 steps to construct an
optimized molecule. A move is determined either randomly or by the Q-function
model. We update the model to predict the move after each step in each episode
and, as this model improves during training, we gradually reduce the
probability that a random move is chosen over the model prediction. If we
start with a single atom approach the DQN finds tens of thousands of candidate
molecules that needed to be optimized. Starting from a pre-seeded molecule led
to an increased number of molecules generated with high pIC50, while starting
from a single atom produced molecules with higher QED, as shown in Fig. S3.

Comparative Effect of the Scoring Function. Table 1 shows the top-3 molecules
generated by the two generative models with each scoring function. DQN always
outperforms JT-VAE in finding a molecule with a superior value of the scoring
function being optimized. The performance disparity is particularly apparent
when optimizing for logP: the maximum logP from DQN is 12.6 compared to only
4.1 for JT-VAE. We attribute the difference in performance to JT-VAE
implicitly sampling from a distribution of drug-like molecules which typically
have logP values between -0.4 and 5.6, while DQN has no such constraints.
However, the JT-VAE-generated molecules have consistently better QED and SA
scores even when those values are not explicitly optimized for. The molecules
in our dataset were experimentally synthesized, leading the SA scores of these
molecules to be quite low. Likewise, because they are drug molecules, their
QED are high. This leads JT-VAE to generate similar molecules with high QED
and low SA scores. On the other hand, the RL agent uses no information about
the space of experimentally studied drug candidates during its training
process and, accordingly, finds molecules far outside that region.

Overall, we find different purposes for JT-VAE and RL-based molecular
optimization. JT-VAE implicitly uses the distribution of molecules in its
training set to bias towards realistic molecules, albeit at the expense of
highly novel candidates. The RL-based approach lacks such constraints and, for
better or worse, optimizes without implicitly regarding synthesizability or
other characteristics not explicitly encoded in the scoring function, leading
to many more novel candidates which are otherwise impossible to optimize
toward.

Qualitative Analysis. We observed an interesting structural trend in the JT-
VAE-generated molecules optimized towards pIC50. Fig. 2 shows the structures
of generated molecules with pIC50 $>$ 8 and the anti-HIV drug Indinavir. A
common backbone is shared between Indinavir and the top-6 predictions. The
Tanimoto similarity of these six generated molecules against Indinavir range
from 0.65 to 0.91. Indinavir has been proposed as a drug candidate to treat
SARS-CoV-2 due to favorable binding affinity to the 3-chymotrypsin-like main
protease (3CLpro) that is pivotal for the replication of SARS-CoV-2 [Yoshino
et al. (2020); Chang (2020); Gyebi et al. (2020); Harrison (2020)]. Notably,
three of the generated molecules have a higher predicted pIC50 than Indinavir.

Figure 2: Top-6 molecules generated by the JT-VAE method optimized towards
pIC50.

_In silico_ Drug-Target Binding Affinity (DTBA) methods offer a way to
evaluate drug–target interactions [He et al. (2017)]. We employ a ML-based
DTBA model to validate the interaction of molecules generated by JT-VAE
against 3CLpro [Chen et al. (2020)]. We trained a DBTA binary classification
model using the extended connectivity fingerprint [Rogers & Hahn (2010)]
encoding for the drug molecule and the target protease sequence encoding from
the DeepPurpose toolkit [Huang et al. (2020)]. The DBTA model classified 4 of
the top-11 molecules (including the top-2 in Fig. 2) with $>$50% probability
of interaction with 3CLpro.

##  5 Conclusions

To possibly explore the novelty and uniqueness of generated molecules, we
compared two graph generative models, JT-VAE and DQN, for the task of
generating small-molecule therapeutic candidates with activity against 3CLpro
target of SARS-CoV-2. DQN always outperformed JT-VAE in finding molecules with
a superior value of the scoring function being optimized. However, JT-VAE
generated molecules that were more structurally similar to those in the
dataset due to substructure representation, which produced a lower SA score
and logP $<$ 5.

## References

  * Armutlu et al. (2008) P. Armutlu, M. E. Ozdemir, F. Uney-Yuksektepe, I. H. Kavakli, and M. Turkay.  Classification of drug molecules considering their ic50 values using mixed-integer linear programming based hyper-boxes method.  _BMC Bioinformatics_ , 9:411, 2008.  ISSN 1471-2105.  doi: 10.1186/1471-2105-9-411.
  * Bag & Ghorai (2016) A. Bag and P. K. Ghorai.  Development of quantum chemical method to calculate half maximal inhibitory concentration (ic50).  _Mol Inform_ , 35(5):199–206, 2016.  ISSN 1868-1743.  doi: 10.1002/minf.201501004.
  * Bajusz et al. (2015) Dávid Bajusz, Anita Rácz, and Károly Héberger.  Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations?  _Journal of cheminformatics_ , 7(1):20, 2015.
  * Batra et al. (2020) Rohit Batra, Henry Chan, Ganesh Kamath, Rampi Ramprasad, Mathew J. Cherukara, and Subramanian K.R.S. Sankaranarayanan.  Screening of therapeutic agents for covid-19 using machine learning and ensemble docking studies.  _The Journal of Physical Chemistry Letters_ , 11(17):7058–7065, 2020.  doi: 10.1021/acs.jpclett.0c02278.  PMID: 32787328.
  * Bento et al. (2014) A Patrícia Bento, Anna Gaulton, Anne Hersey, Louisa J Bellis, Jon Chambers, Mark Davies, Felix A Krüger, Yvonne Light, Lora Mak, Shaun McGlinchey, et al.  The chembl bioactivity database: an update.  _Nucleic Acids Research_ , 42(D1):D1083–D1090, 2014.
  * Bickerton et al. (2012) G. Richard Bickerton, Gaia V. Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L. Hopkins.  Quantifying the chemical beauty of drugs.  _Nature chemistry_ , 4(2):90–98, 2012.  ISSN 1755-4349 1755-4330.  doi: 10.1038/nchem.1243.  URL https://pubmed.ncbi.nlm.nih.gov/22270643https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3524573/.
  * Bjerrum (2017) Esben Jannik Bjerrum.  SMILES enumeration as data augmentation for neural network modeling of molecules.  _CoRR_ , abs/1703.07076, 2017.  URL http://arxiv.org/abs/1703.07076.
  * Born et al. (2020) Jannis Born, Matteo Manica, Joris Cadow, Greta Markert, Nil Adell Mill, Modestas Filipavicius, and María Rodríguez Martínez.  Paccmannrl on sars-cov-2: Designing antiviral candidates with conditional generative models.  _arXiv e-prints_ , pp. arXiv–2005, 2020.
  * Chang (2020) Y. et al. Chang.  Potential therapeutic agents for covid-19 based on the analysis of protease and rna polymerase docking.  _Preprints_ , pp. 2020020242, 2020.  doi: 10.20944/preprints202002.0242.v1.
  * Chen et al. (2020) Yu Wai Chen, Chin-Pang Bennu Yiu, and Kwok-Yin Wong.  Prediction of the sars-cov-2 (2019-ncov) 3c-like protease (3cl pro) structure: virtual screening reveals velpatasvir, ledipasvir, and other drug repurposing candidates.  _F1000Research_ , 9, 2020.
  * Chenthamarakshan et al. (2020) Vijil Chenthamarakshan, Payel Das, Inkit Padhi, Hendrik Strobelt, Kar Wai Lim, Ben Hoover, Samuel C Hoffman, and Aleksandra Mojsilovic.  Target-specific and selective drug design for covid-19 using deep generative models.  _arXiv preprint arXiv:2004.01215_ , 2020.
  * Copeland et al. (2006) Robert A Copeland, David L Pompliano, and Thomas D Meek.  Drug–target residence time and its implications for lead optimization.  _Nature Reviews Drug Discovery_ , 5(9):730–739, 2006.
  * Ertl & Schuffenhauer (2009) Peter Ertl and Ansgar Schuffenhauer.  Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions.  _Journal of Cheminformatics_ , 1(1):8, 2009.  ISSN 1758-2946.  doi: 10.1186/1758-2946-1-8.  URL https://doi.org/10.1186/1758-2946-1-8.
  * Gilmer et al. (2017) Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl.  Neural message passing for quantum chemistry.  In _34th International Conference on Machine Learning-Volume 70_ , pp. 1263–1272. JMLR. org, 2017.
  * Gordon et al. (2020) David E. Gordon, Gwendolyn M. Jang, Mehdi Bouhaddou, Jiewei Xu, Kirsten Obernier, Kris M. White, Matthew J. O’Meara, Veronica V. Rezelj, Jeffrey Z. Guo, Danielle L. Swaney, Tia A. Tummino, Ruth Huettenhain, Robyn M. Kaake, Alicia L. Richards, Beril Tutuncuoglu, Helene Foussard, Jyoti Batra, Kelsey Haas, Maya Modak, Minkyu Kim, Paige Haas, Benjamin J. Polacco, Hannes Braberg, Jacqueline M. Fabius, Manon Eckhardt, Margaret Soucheray, Melanie J. Bennett, Merve Cakir, Michael J. McGregor, Qiongyu Li, Bjoern Meyer, Ferdinand Roesch, Thomas Vallet, Alice Mac Kain, Lisa Miorin, Elena Moreno, Zun Zar Chi Naing, Yuan Zhou, Shiming Peng, Ying Shi, Ziyang Zhang, Wenqi Shen, Ilsa T. Kirby, James E. Melnyk, John S. Chorba, Kevin Lou, Shizhong A. Dai, Inigo Barrio-Hernandez, Danish Memon, Claudia Hernandez-Armenta, Jiankun Lyu, Christopher J. P. Mathy, Tina Perica, Kala B. Pilla, Sai J. Ganesan, Daniel J. Saltzberg, Ramachandran Rakesh, Xi Liu, Sara B. Rosenthal, Lorenzo Calviello, Srivats Venkataramanan, Jose Liboy-Lugo, Yizhu Lin, Xi-Ping Huang, YongFeng Liu, Stephanie A. Wankowicz, Markus Bohn, Maliheh Safari, Fatima S. Ugur, Cassandra Koh, Nastaran Sadat Savar, Quang Dinh Tran, Djoshkun Shengjuler, Sabrina J. Fletcher, Michael C. O’Neal, Yiming Cai, Jason C. J. Chang, David J. Broadhurst, Saker Klippsten, Phillip P. Sharp, Nicole A. Wenzell, Duygu Kuzuoglu, Hao-Yuan Wang, Raphael Trenker, Janet M. Young, Devin A. Cavero, Joseph Hiatt, Theodore L. Roth, Ujjwal Rathore, Advait Subramanian, Julia Noack, Mathieu Hubert, Robert M. Stroud, Alan D. Frankel, Oren S. Rosenberg, Kliment A. Verba, David A. Agard, Melanie Ott, Michael Emerman, Natalia Jura, et al.  A sars-cov-2 protein interaction map reveals targets for drug repurposing.  _Nature_ , 2020.  ISSN 1476-4687.  doi: 10.1038/s41586-020-2286-9.  URL https://doi.org/10.1038/s41586-020-2286-9.
  * Gyebi et al. (2020) Gideon A. Gyebi, Olalekan B. Ogunro, Adegbenro P. Adegunloye, Oludare M. Ogunyemi, and Saheed O. Afolabi.  Potential inhibitors of coronavirus 3-chymotrypsin-like protease (3cl(pro)): an in silico screening of alkaloids and terpenoids from african medicinal plants.  _Journal of Biomolecular Structure and Dynamics_ , pp. 1–13, 2020\.  ISSN 1538-0254 0739-1102.  doi: 10.1080/07391102.2020.1764868.  URL https://pubmed.ncbi.nlm.nih.gov/32367767https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7256353/.
  * Harrison (2020) C Harrison.  Coronavirus puts drug repurposing on the fast track.  _Nature Biotechnology_ , 38(4):379, 2020.
  * He et al. (2017) Tong He, Marten Heidemeyer, Fuqiang Ban, Artem Cherkasov, and Martin Ester.  Simboost: a read-across approach for predicting drug–target binding affinities using gradient boosting machines.  _Journal of Cheminformatics_ , 9(1):1–14, 2017\.
  * Horwood & Noutahi (2020) Julien Horwood and Emmanuel Noutahi.  Molecular design in synthetically accessible chemical space via deep reinforcement learning.  _arXiv preprint arXiv:2004.14308_ , 2020.
  * Hu et al. (2020) Fan Hu, Dongqi Wang, Yishen Hu, Jiaxin Jiang, and Peng Yin.  Generating novel compounds targeting sars-cov-2 main protease based on imbalanced dataset.  In _2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , pp. 432–436. IEEE, 2020.
  * Huang et al. (2020) Kexin Huang, Tianfan Fu, Cao Xiao, Lucas Glass, and Jimeng Sun.  Deeppurpose: a deep learning based drug repurposing toolkit.  _arXiv preprint arXiv:2004.08919_ , 2020.
  * Jin et al. (2018) Wengong Jin, Regina Barzilay, and Tommi S. Jaakkola.  Junction tree variational autoencoder for molecular graph generation.  _CoRR_ , abs/1802.04364, 2018.  URL http://arxiv.org/abs/1802.04364.
  * Jin et al. (2020) Wengong Jin, Regina Barzilay, and Tommi Jaakkola.  Multi-Objective Molecule Generation using Interpretable Substructures.  _arXiv e-prints_ , art. arXiv:2002.03244, February 2020.
  * John et al. (2019) Peter C. St. John, Caleb Phillips, Travis W. Kemper, A. Nolan Wilson, Yanfei Guan, Michael F. Crowley, Mark R. Nimlos, and Ross E. Larsen.  Message-passing neural networks for high-throughput polymer screening.  _Journal of Chemical Physics_ , 150(23):234111, June 2019.  doi: 10.1063/1.5099132.  URL https://doi.org/10.1063/1.5099132.
  * Joshi & Kumar (2021) Rajendra P Joshi and Neeraj Kumar.  Artificial intelligence based autonomous molecular design for medical therapeutic: A perspective.  _arXiv preprint arXiv:2102.06045_ , 2021.
  * Khemchandani et al. (2020) Yash Khemchandani, Steve O’Hagan, Soumitra Samanta, Neil Swainston, Timothy J Roberts, Danushka Bollegala, and Douglas B Kell.  Deepgraphmol, a multi-objective, computational strategy for generating molecules with desirable properties: a graph convolution and reinforcement learning approach.  _bioRxiv_ , 2020.
  * Kumar & Zhang (2018) Ashutosh Kumar and Kam Y. J. Zhang.  Advances in the development of shape similarity methods and their application in drug discovery.  _Frontiers in chemistry_ , 6:315–315, 2018.  ISSN 2296-2646.  doi: 10.3389/fchem.2018.00315.  URL https://pubmed.ncbi.nlm.nih.gov/30090808https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6068280/.
  * Kusner et al. (2017) Matt J. Kusner, Brooks Paige, and José Miguel Hernández-Lobato.  Grammar variational autoencoder, 2017.
  * Li et al. (2018) Yibo Li, Liangren Zhang, and Zhenming Liu.  Multi-objective de novo drug design with conditional graph generative model.  _Journal of Cheminformatics_ , 10(1):33, 2018\.  ISSN 1758-2946.  doi: 10.1186/s13321-018-0287-6.  URL https://doi.org/10.1186/s13321-018-0287-6.
  * Lipinski (2004) Christopher A Lipinski.  Lead-and drug-like compounds: the rule-of-five revolution.  _Drug Discovery Today: Technologies_ , 1(4):337–341, 2004.
  * Maggiora et al. (2014) Gerald Maggiora, Martin Vogt, Dagmar Stumpfe, and Jurgen Bajorath.  Molecular similarity in medicinal chemistry: miniperspective.  _Journal of medicinal chemistry_ , 57(8):3186–3204, 2014.
  * Mittal et al. (2020) Lovika Mittal, Anita Kumari, Mitul Srivastava, Mrityunjay Singh, and Shailendra Asthana.  Identification of potential molecules against covid-19 main protease through structure-guided virtual screening approach.  _Journal of Biomolecular Structure and Dynamics_ , pp. 1–19, 2020\.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Patankar & Jurs (2000) S. J. Patankar and P. C. Jurs.  Prediction of ic50 values for acat inhibitors from molecular structure.  _Journal of Chemical Information and Computer Sciences_ , 40(3):706–723, 2000.  ISSN 0095-2338.  doi: 10.1021/ci990125r.  URL https://doi.org/10.1021/ci990125r.
  * Reymond (2015) Jean-Louis Reymond.  The chemical space project.  _Accounts of Chemical Research_ , 48(3):722–730, 2015.
  * Rogers & Hahn (2010) David Rogers and Mathew Hahn.  Extended-connectivity fingerprints.  _Journal of Chemical Information and Modeling_ , 50(5):742–754, 2010.
  * Samanta et al. (2019) Bidisha Samanta, DE Abir, Gourhari Jana, Pratim Kumar Chattaraj, Niloy Ganguly, and Manuel Gomez Rodriguez.  Nevae: A deep generative model for molecular graphs.  In _AAAI Conference on Artificial Intelligence_ , volume 33, pp. 1110–1117, 2019.
  * Scarselli et al. (2008) Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.  The graph neural network model.  _IEEE Transactions on Neural Networks_ , 20(1):61–80, 2008.
  * Sebaugh (2011) JL Sebaugh.  Guidelines for accurate ec50/ic50 estimation.  _Pharmaceutical statistics_ , 10(2):128–134, 2011\.
  * Sivaraman et al. (2020) Ganesh Sivaraman, Nicholas Jackson, Benjamin Sanchez-Lengeling, Alvaro Vasquez-Mayagoitia, Alan Aspuru-Guzik, Venkatram Vishwanath, and Juan de Pablo.  A machine learning workflow for molecular analysis: application to melting points.  _Machine Learning: Science and Technology_ , 2020.
  * Skwark et al. (2020) Marcin J Skwark, Nicolás López Carranza, Thomas Pierrot, Joe Phillips, Slim Said, Alexandre Laterre, Amine Kerkeni, Uğur Şahin, and Karim Beguir.  Designing a prospective covid-19 therapeutic with reinforcement learning.  _arXiv preprint arXiv:2012.01736_ , 2020.
  * Smith & Smith (2020) Micholas Smith and Jeremy C. Smith.  Repurposing therapeutics for covid-19: Supercomputer-based docking to the sars-cov-2 viral spike protein and viral spike protein-human ace2 interface, Feb 2020.  URL https://chemrxiv.org/articles/Repurposing_Therapeutics_for_the_Wuhan_Coronavirus_nCov-2019_Supercomputer-Based_Docking_to_the_Viral_S_Protein_and_Human_ACE2_Interface/11871402/3.
  * Ståhl et al. (2019) Niclas Ståhl, Göran Falkman, Alexander Karlsson, Gunnar Mathiason, and Jonas Bostrom.  Deep reinforcement learning for multiparameter optimization in de novo drug design.  _Journal of Chemical Information and Modeling_ , 59(7):3166–3176, 2019.
  * Tang et al. (2020) Bowen Tang, Fengming He, Dongpeng Liu, Meijuan Fang, Zhen Wu, and Dong Xu.  Ai-aided design of novel targeted covalent inhibitors against sars-cov-2.  _BioRxiv_ , 2020.
  * Waring (2010) Michael J Waring.  Lipophilicity in drug discovery.  _Expert Opinion on Drug Discovery_ , 5(3):235–248, 2010.  doi: 10.1517/17460441003605098.  URL https://doi.org/10.1517/17460441003605098.  PMID: 22823020.
  * Weininger (1990) David Weininger.  Smiles. 3. depict. graphical depiction of chemical structures.  _Journal of Chemical Information and Computer Sciences_ , 30(3):237–243, 1990.
  * Wu et al. (2020) Canrong Wu, Yang Liu, Yueying Yang, Peng Zhang, Wu Zhong, Yali Wang, Qiqi Wang, Yang Xu, Mingxue Li, Xingzhou Li, et al.  Analysis of therapeutic targets for sars-cov-2 and discovery of potential drugs by computational methods.  _Acta Pharmaceutica Sinica B_ , 10(5):766–788, 2020.
  * Yoshino et al. (2020) Ryunosuke Yoshino, Nobuaki Yasuo, and Masakazu Sekijima.  Identification of key interactions between sars-cov-2 main protease and inhibitor drug candidates.  _Scientific reports_ , 10(1):1–8, 2020.
  * You et al. (2018) Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec.  Graph convolutional policy network for goal-directed molecular graph generation.  In _Advances in Neural Information Processing Systems_ , pp. 6410–6421, 2018.
  * Zhang & Wilkinson (2007) Ming-Qiang Zhang and Barrie Wilkinson.  Drug discovery beyond the ‘rule-of-five’.  _Current Opinion in Biotechnology_ , 18(6):478 – 488, 2007.  ISSN 0958-1669.  doi: https://doi.org/10.1016/j.copbio.2007.10.005.  URL http://www.sciencedirect.com/science/article/pii/S0958166907001279.  Chemical biotechnology / Pharmaceutical biotechnology.
  * Zhavoronkov et al. (2020) Alex Zhavoronkov, Vladimir Aladinskiy, Alexander Zhebrak, Bogdan Zagribelnyy, Victor Terentiev, Dmitry S Bezrukov, Daniil Polykovskiy, Rim Shayakhmetov, Andrey Filimonov, Philipp Orekhov, et al.  Potential covid-2019 3c-like protease inhibitors designed using generative deep learning approaches.  _Insilico Medicine Hong Kong Ltd A_ , 307:E1, 2020.
  * Zhou et al. (2020) Yadi Zhou, Yuan Hou, Jiayu Shen, Yin Huang, William Martin, and Feixiong Cheng.  Network-based drug repurposing for novel coronavirus 2019-ncov/sars-cov-2.  _Cell Discovery_ , 6(1):14, 2020.  ISSN 2056-5968.  doi: 10.1038/s41421-020-0153-3.  URL https://doi.org/10.1038/s41421-020-0153-3.
  * Zhou et al. (2019) Zhenpeng Zhou, Steven Kearnes, Li Li, Richard N. Zare, and Patrick Riley.  Optimization of molecules via deep reinforcement learning.  _Scientific Reports_ , 9(1):10752, 2019.  ISSN 2045-2322.  doi: 10.1038/s41598-019-47148-x.  URL https://doi.org/10.1038/s41598-019-47148-x.

## Supplementary Material

##  Appendix A Dataset Analysis

We computed the metrics included in our scoring functions for each molecule in
the database. The range of values can be seen in Figure S1. The highest pIC50
is 10.89, while the lowest pIC50 is 1.22. The most common pIC50 is 4.0, which
is shared by 320 structures, and the vast majority of structures (91.5%) have
pIC50 values greater than 4.0. LogP values range from -10.36 to 16.65 in a
near Gaussian distribution with a mean of 3.70; 77.0% of all structures meet
the requirement of Lipinski’s Rule of 5 that logP be no greater than 5. QED
values range from 0.01 to 0.94, while SA values range from 1.35 to 8.24, with
96.3% being below 5.

Figure S1 also shows the range of properties of molecules generated by our
trained JT-VAE without optimization. In general, JT-VAE reproduced the
distribution of logP, QED, pIC50, and SA scores present in the database. The
pIC50 values of the JT-VAE-generated molecules were obtained from our trained
MPNN, discussed in Section 3.1.

Figure S1: We generated 1,000 molecules using the trained JT-VAE, of which 560
were unique. The figure shows the comparison of the QED, logP, SA score, and
pIC50 of compounds in the database and those generated by the JT-VAE. The JT-
VAE reproduced the range of values present in the database, minus outliers.
The similar values indicate that the JT-VAE is able to reproduce the wide
range of structures present in the database. The pIC50 values for generated
molecules were estimated by our MPNN.

We also computed the Tanimoto similarity for all pairs of compounds to gain
insight into the structural diversity of molecules in our database (Figure
S2). The entries in the matrix were ordered in increasing pIC50 values. The
similarity is represented by the color bar, with yellow representing low
similarity (0) and red high similarity (1). We observe that structures tend to
become more similar to their neighbors as pIC50 increases, indicating that
compounds with high pIC50 values tend to be structurally similar, supporting
the consideration of molecular similarity during drug discovery.

Figure S2: Heat map of the Tanimoto similarity between all compounds in the
database. Entries are ordered in increasing pIC50. The Tanimoto similarity is
generally higher among structures with high pIC50 (located towards bottom
right of the matrix), indicating the importance of considering structural
similarity in drug discovery.

##  Appendix B Effects of Pre-seeding RL

Figure S3: (QED vs pIC50 for molecules generated by RL with starting from a
single atom (MolDQN) or pre-seeded with a drug-like molecule (Seeded MolDQN)
optimized towards the multi-objective scoring function in eqn. S8.


# KazQAD: Kazakh Open-Domain Question Answering Dataset

###### Abstract

We introduce KazQAD—a Kazakh open-domain question answering (ODQA)
dataset—that can be used in both reading comprehension and full ODQA settings,
as well as for information retrieval experiments. KazQAD contains just under
6,000 unique questions with extracted short answers and nearly 12,000 passage-
level relevance judgements. We use a combination of machine translation,
Wikipedia search, and in-house manual annotation to ensure annotation
efficiency and data quality. The questions come from two sources: translated
items from the Natural Questions (NQ) dataset (only for training) and the
original Kazakh Unified National Testing (UNT) exam (for development and
testing). The accompanying text corpus contains more than 800,000 passages
from the Kazakh Wikipedia. As a supplementary dataset, we release around
61,000 question-passage-answer triples from the NQ dataset that have been
machine-translated into Kazakh. We develop baseline retrievers and readers
that achieve reasonable scores in retrieval (NDCG@10 = 0.389 MRR = 0.382),
reading comprehension (EM = 38.5 F1 = 54.2), and full ODQA (EM = 17.8 F1 =
28.7) settings. Nevertheless, these results are substantially lower than
state-of-the-art results for English QA collections, and we think that there
should still be ample room for improvement. We also show that the current
OpenAI’s ChatGPTv3.5 is not able to answer KazQAD test questions in the
closed-book setting with acceptable quality. The dataset is freely available
under the Creative Commons licence (CC BY-SA) at
https://github.com/IS2AI/KazQAD.

Keywords: open-domain question answering, benchmarks for low-resource
languages, evaluation

KazQAD: Kazakh Open-Domain Question Answering Dataset

|  Rustem Yeshpanov1, Pavel Efimov2, Leonid Boytsov3$\star$
---
Ardak Shalkarbayuli4, Pavel Braslavski5
1Institute of Smart Systems and Artificial Intelligence, Nazarbayev
University, Astana, Kazakhstan
2ITMO University, Saint Petersburg, Russia
3Amazon AWS AI Labs, Pittsburgh, USA
4Suleyman Demirel University, Almaty, Kazakhstan
5School of Engineering and Digital Sciences, Nazarbayev University, Astana,
Kazakhstan
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

$\star$$\star$footnotetext: Work done outside of the scope of employment.

Open-domain question answering (ODQA) is the task of finding a concise and
accurate answer to a natural language question in a large collection of text
documents Hirschman and Gaizauskas (2001). A more restricted question-
answering (QA) task that involves locating an answer within a single document
is commonly referred to as _reading comprehension_ (RC). A traditional ODQA
system has two main components: a _retriever_ and an _answer extractor_ Prager
(2006); Ferrucci et al. (2010). In recent literature, answer extractors are
called _readers_ Chen et al. (2017); Zhu et al. (2021).

QA is a popular practical application and an active research area that serves
as a testbed for information retrieval (IR) and natural language processing
(NLP) techniques. The success of evaluation initiatives, such as TREC Voorhees
and Harman (2005) and CLEF Ferro and Peters (2019), as well as datasets such
as SQuAD Rajpurkar et al. (2016) have demonstrated that standardised
evaluation approaches and test collections are key factors for measurable
progress in solving IR and NLP tasks. However, despite significant research
investment and impressive progress in English QA Calijorne Soares and
Parreiras (2020), advances in other languages have been less impressive. This
is in part due to the scarcity of training and test datasets, which are more
difficult to create for low-resource languages Ruder (2022).

To mitigate this shortcoming, we create a new ODQA dataset KazQAD (/kæsked/),
which stands for a Kazakh open-domain Question Answering Dataset. KazQAD can
be used in both reading comprehension and full ODQA settings, as well as for
information retrieval experiments.

Kazakh, as a member of the Turkic language family and specifically of its
Kipchak branch, is characterised as an agglutinative language Campbell and
King (2020). Written communication in Kazakh relies on an extended Cyrillic
script. It is estimated that there are approximately 13 million native
speakers of Kazakh, over 10 million of whom reside in Kazakhstan. The
remaining three million speakers are scattered across various other countries,
including China, Mongolia, Russia, and Turkey.

Kazakh is considered a language with limited resources. Annotated datasets
specifically tailored to IR and NLP tasks in Kazakh are scarce. A notable
exception is a recent dataset focusing on named entity recognition
(NER)—KazNERD Yeshpanov et al. (2022).

The main idea behind KazQAD is to leverage and repurpose existing data in
addition to manually labeling a new resource. At the same time, we refrained
from adopting a fully automated approach to dataset construction such as
relying solely on machine translation to prevent an extensive presence of
unrealistic synthetic data such as _translationese_ Baroni and Bernardini
(2006). For the training set, we extracted questions from the English _Natural
Questions_ (NQ) dataset Kwiatkowski et al. (2019), machine-translated them
into Kazakh and aligned them with the corresponding Kazakh Wikipedia articles.
For the development and test sets, we used questions from the Kazakh Unified
National Testing (UNT) exam and matched them with Wikipedia pages using Google
search. Then, in-house native Kazakh annotators extracted answers from the
retrieved Wikipedia passages. The data processing and annotation is described
in detail in Section 3. Overall, KazQAD contains just under 6,000 unique
questions with extracted short answers and nearly 12,000 passage-level
relevance judgements.

We develop baseline retrievers and readers that achieve reasonable scores in
retrieval (NDCG@10 = 0.389 MRR = 0.382), reading comprehension (EM = 38.5 F1 =
54.2), and full ODQA (EM = 17.8 F1 = 28.7) settings. Yet, these results are
mostly worse than those reported by Chen et al. (2017), who described possibly
the first neural ODQA English system. Thus, we believe that there is still
much room for improvement. In addition, we submitted KazQAD test questions to
ChatGPTv3.5 and evaluated its answers. The combination of automatic and manual
evaluation shows that OpenAI’s model still struggles to answer factual
questions in Kazakh. The dataset and baseline models are freely available
under the Creative Commons licence.111https://github.com/IS2AI/KazQAD

## 2\. Related Work

TREC Voorhees and Harman (2005), the Cross-Language Evaluation Forum (CLEF)
Ferro and Peters (2019), the Russian information retrieval evaluation
initiative (also known as ROMIP) Dobrov et al. (2004), and NII Testbeds and
Community for Information access Research
(NTCIR)222https://research.nii.ac.jp/ntcir/ evaluation campaigns have featured
both cross-language retrieval (involving query and document collections in
different languages) and monolingual retrieval in non-English languages.
However, the datasets produced by these evaluation initiatives have been
relatively small. With the proliferation of data-intensive neural methods in
IR and QA, the demand for larger annotated collections has increased
significantly.

Since the release of the English SQuAD dataset Rajpurkar et al. (2016), we
have experienced a “QA dataset explosion” Rogers et al. (2023) that has, inter
alia, led to the emergence of many non-English datasets. These datasets were
created through various approaches, including machine translations of SQuAD,
such as the Spanish Carrino et al. (2019) and Turkish Ünlü Menevşe et al.
(2022) variants, as well as the application of the SQuAD annotation approach
to Wikipedia in other languages (cf. Russian SberQuAD Efimov et al. (2020)).

Subsequently, multilingual QA datasets appeared, encompassing multiple
languages simultaneously. The approaches to their creation also varied. For
instance, XQuAD Artetxe et al. (2020) involved manual translation of a small
portion of English SQuAD questions and contexts into 10 languages, while MLQA
Lewis et al. (2020) focused on translating only the questions and annotating
the answers in parallel contexts for each of the seven languages
independently. TyDi QA Clark et al. (2020) utilised annotators that
independently generated questions based on short prompts from Wikipedia for 11
typologically different languages, annotated them at paragraph level, and
extracted short answers. MKQA Longpre et al. (2020) involved the manual
translation of original English questions from NQ Kwiatkowski et al. (2019)
into 25 languages and their subsequent annotatiion with answers from scratch.

The Kazakh language is present in the recently released massive multilingual
Belebele dataset Bandarkar et al. (2023). The dataset contains 900 multiple-
choice questions to 488 passages from the FLoRes collection Goyal et al.
(2022), translated from English into 122 languages, including Kazakh. Although
KazQAD is a monolingual dataset, it has the following advantages:

  * •

It is larger;

  * •

It has a much more diverse set of potential answers compared to the multiple-
choice Belebele;

  * •

It uses original Kazakh questions (for testing) and passages instead of
translations;

  * •

It can be used in both RC and ODQA settings, as well as a testbed for IR
models.

.

The English MS MARCO dataset Bajaj et al. (2016) and its subsequent editions
Craswell et al. (2020) have become the de facto standard for conducting search
and ranking experiments using neural models. MS MARCO has also been machine-
translated into 13 languages Bonifacio et al. (2021). Chen et al. (2017) and
later Karpukhin et al. (2020) showed that extractive QA datasets can be
leveraged to create datasets for search and ranking. Thus, the multilingual
TyDi QA dataset served as the foundation for the creation of the IR dataset
Mr. TyDi Zhang et al. (2021), which covers 11 languages. Later, the dataset
annotations were updated and extended to include data from an additional seven
languages, resulting in the creation of the MIRACL dataset Zhang et al.
(2022).

Large language models (LLMs) have the potential to revolutionise data
annotation efforts, model training, and evaluation. For example, LLMs can be
leveraged for generating training examples and subsequently fine-tuning a
smaller model on them Bonifacio et al. (2022). Furthermore, LLMs can also be
directly utilised for evaluation purposes Fu et al. (2023). However, when it
comes to low-resourced languages, these options entail certain risks. Limited
research has been conducted on the general capabilities of LLMs in such
languages, beyond their application in machine translation Jiao et al. (2023);
Lai et al. (2023). As we show in our study, the question answering
capabilities of ChatGPT in Kazakh are still in their infancy.

When creating KazQAD, we integrated various approaches employed in the
construction of established QA datasets. For the training set of our dataset,
we followed a similar strategy as for MKQA, where we “recycled” the English NQ
questions while sourcing the passages to be annotated from the corresponding
Kazakh Wikipedia (which is similar to the process of creating MLQA by Lewis et
al. (2020)). In addition, we leveraged question-answer pairs to identify
relevant passages for another segment of KazQAD, taking inspiration from the
DPR study Karpukhin et al. (2020), but the passages retrieved were annotated
manually.

## 3\. Dataset Creation

### 3.1. Data Sources

#### Kazakh Wikipedia

We used a dump of the Kazakh Wikipedia dated January 20, 2023, which contains
236,480 pages. To extract the text content of the pages, we employed a
combination of WikiExtractor Attardi (2015) and a custom parser specifically
designed for the templates prevalent in the Kazakh Wikipedia. We then split
the pages into passages using double newlines ($\backslash$n$\backslash$n). We
found that extremely long passages (tens of thousands of characters) were
still present in the collection, so we further subdivided them using single
newlines ($\backslash$n). Finally, we removed exact passage duplicates. As a
result, our collection contains around 815,000 passages, with an average
length of 277 characters and a median of 183 characters. We also aggregated
page views and edits for each page for the entire year 2022. We used these
data as page quality indicators in the annotation process.

Figure 1: KazQAD pipeline: _Q_ – question, _P_ – passage, _S_ – sentence, _A_
– answer; subscripts: en – English, kk – Kazakh. The upper part corresponds to
the training set, while the lower part represents the development and test
sets. In the case of the training set, we start with the pre-selected NQ
questions and in-context answers machine-translated into Kazakh. Candidate
passages are extracted from the parallel Kazakh Wikipedia pages. In the case
of the development and test sets, the starting point is an original Kazakh
question-answer pair from the UNT collection. Passages are retrieved from the
Kazakh Wikipedia using Google search. In both annotation scenarios, the
annotators are thus presented with question-paragraph pairs along with
candidate answers.

#### Natural Questions (NQ)

NQ is a dataset containing over 300,000 English questions from the Google
search log, in part along with _long answers_ (roughly corresponding to
Wikipedia paragraphs) and _short answers_ from the English Wikipedia
Kwiatkowski et al. (2019). Short extractive answers are provided for more than
100,000 questions. The rationale behind our choosing NQ lies in its size and
notably diverse nature as a QA collection. Furthermore, its questions are
representative of authentic user information needs, unlike datasets such as
SQuAD where the questions are generated by crowdworkers presented with
Wikipedia paragraphs. For the _training_ split of KazQAD, we utilise machine-
translated NQ questions.

#### NQ translation

In addition, we machine-translated a substantial part of the English NQ
dataset into Kazakh. We selected question-paragraph pairs for which there is a
short answer and the passage does not belong to the English Wikipedia page for
which there is a parallel Kazakh page. We labelled the answer in the passage
and translated these data using the Yandex Translate API.333Based on our
internal evaluation, Google Translate and Yandex Translate demonstrate
comparable performance on the English-Kazakh pair within the FLoRes dataset,
as measured by BLEU scores. The cost-effectiveness of Yandex Translate
significantly influenced our decision, as it offers a more economical solution
compared to Google Translate. In total, we ended up with more than 61,000
question-passage pairs with short answers that we used in experiments (we also
make them publicly available).

#### Unified National Testing (UNT)

The UNT is a comprehensive school leaving (i.e., graduation) qualification
exam in Kazakhstan that covers a diverse range of subjects. These include
Kazakh language, Kazakh literature, history of Kazakhstan, mathematics,
physics, biology, and several others. Part of the test consists of multiple-
choice questions. There are many examples of UNT questions and related study
materials on the Internet. We collected 8,582 questions along with the correct
answers in five subjects (biology, geography, history of Kazakhstan, Kazakh
literature, and world history). We instructed the annotators to rephrase the
multiple-choice questions into open-ended questions where possible, which
resulted in 8,562 questions. We use UNT questions for the _development_ and
_test_ splits of KazQAD.

### 3.2. Annotation

Since we were able to collect only a limited number of original Kazakh UNT
questions from the Internet, we decided to make them the basis of the
development and test sets. We chose NQ questions as the source for the
training set. Despite the fact that these data sources have different
structure, we unified the data using pre-processing, so that the annotators
could solve the same problem on different subsets of the data. For example, we
machine-translated English questions and potential answers into Kazakh, and
unsed Wikipedia langlinks to find candidate relevant passages. In the case of
UNT, we found potentially relevant passages using Google search with question
and answer as a query. It is important to note that in both cases we used the
same collection of original passages from the Kazakh Wikipedia. As a result of
the annotation, we obtained question-passage-answer triples (as well as
passages marked as non-relevant by the annotators). These data can be used for
various tasks: searching a collection of Wikipedia passages (answer-bearing
passages are considered relevant), reading comprehension (using full question-
passage-answer triples), and ODQA (using question-answer pairs).

The KazQAD annotation pipeline is outlined in Figure 1. The upper part
corresponds to NQ processing resulting in the training set, while the lower
part corresponds to UNT processing (the development and test sets). We hired
six Kazakh native speakers (three women and three men). The age distribution
of the annotators ranged from 27 to 35, with a mean age of 31. Their
educational backgrounds varied, with two holding Master’s degrees and the
remaining four having Bachelor’s degrees.

#### NQ

First, we identified NQ questions that had at least one short answer in the
dataset. In addition, to be included, the Wikipedia page had to have a
corresponding parallel Kazakh version available. Among the distinct Wikipedia
titles in the NQ dataset, a total of 10,016 titles were identified to have
corresponding Kazakh pages. Of the 42,451 unique questions associated with
these titles, 14,553 questions were identified that contained at least one
answer. These questions, along with the answer-bearing sentences, were
machine-translated into Kazakh using the Yandex translate
service.444https://translate.yandex.com/ To be able to locate the answer in
the translated sentence, we prudently placed the answer span in quotation
marks in the English source, as in Lewis et al. (2020). In about 10% of the
translated sentences, we could not locate the supposed answer. Several
semantically similar questions from NQ were translated into identical Kazakh
questions; their annotations were subsequently merged. Next, we extracted up
to four passages from the parallel Kazakh page using Wikipedia _langlinks_.
The total count of question-passage pairs amounted to 34,660, as some pages
contained minimal information, often limited to a single short paragraph. Note
that we machine-translated only the questions, not the passages.

Using the NQ data, we implemented a two-step annotation process. In the first
step, the annotators labelled passages as either relevant or non-relevant. In
the second step, they selected a short answer (or multiple answers) from the
passages marked as relevant. If a question was found to be ambiguous or
incomprehensible, the annotators had the option to discard it.

During the annotation process, questions that pertained to events that were no
longer true at the time of annotation, but could still be answered using the
provided passage, were deemed ambiguous and subsequently discarded. For
instance, if a question enquired about the current Queen of England, it was
considered unclear since, at the time of annotation, the reigning monarch was
King Charles III. This decision was made even if the passage contained
information regarding Queen Elizabeth II. Similarly, a question about the
venue of the upcoming 2022 football World Cup was marked as ambiguous as the
tournament had already taken place at the time of annotation, although the
passage indicated that it would be held in Qatar. By eliminating such
questions, we were able to maintain the accuracy and contemporaneity of the
dataset.

#### UNT

To complement the UNT data with contexts, we utilised a combination of a UNT
question and an answer as a query and employed Google custom search restricted
to the kk.wikipedia.org domain, specifically focusing on the Kazakh Wikipedia.
We took passages from our Wikipedia collection (see above) corresponding to
the top-ranked pages returned by Google search and re-ranked them using a
simple heuristic of similarity to the question and answer. The annotators were
presented with up to four top-ranked passages. It is crucial to acknowledge
that, after scrutinizing the results of the NQ annotation and consulting with
the annotators, we transitioned to a one-step annotation approach for the UNT
data, which was anticipated to streamline the process. Under this annotation
scheme, the annotators were presented with passages and answer hints obtained
through machine translation. Their task was to either identify short answers
or declare absence thereof. When an annotator found a short answer in a
passage, they proceeded to the next question. This difference in the
annotation scheme accounts for the varying ratios of relevant and non-relevant
labels and question-paragraph pairs with or without answers between the NQ and
UNT parts.

### 3.3. KazQAD Structure and Statistics

Using the annotations obtained, we constructed two datasets: 1) an IR dataset
consisting of a collection of Kazakh Wikipedia passages, a list of questions,
and pairs of question and passage IDs marked as either relevant or non-
relevant (represented in TREC _qrels_ format), and 2) a machine-reading
dataset in SQuAD-like format that contains question-passage-answer triples.

The statistics of the IR part can be found in Table 1. All NQ questions were
included in the _training_ set. From the UNT annotation, we retained 548
questions for the _development_ set to ensure a roughly equal distribution of
questions from each of the five subjects. The remaining 1,929 questions formed
the _test_ set. As one can see, the total number of annotated queries is
slightly below 6,000, while the number of passage-level annotations (both
relevant and non-relevant) is almost 12,000. These figures are on par with the
average numbers per language in the MIRACL dataset Zhang et al. (2022),
although the share of relevant labels is higher in KazQAD.

Source | Q | P+ | P–
---|---|---|---
NQ (train) | 3,487 | 3,893 | 3,558
UNT (dev) | 548 | 769 | 229
UNT (test) | 1,929 | 2,718 | 653
Total | 5,964 | 7,380 | 4,440
Table 1: KazQAD statistics: Q – annotated questions, P+ and P– – relevant and non-relevant passages. | KazQAD | NQtransl.
---|---|---
| train | dev | test
# question-passage pairs | 3,163 | 764 | 2,713 | 61,606
# unique questions | 2,920 | 545 | 1,927 | 61,198
# unique passages | 1,993 | 697 | 2,137 | 53,204
# short answers | 3,826 | 910 | 3,315 | 71,242
avg. words per passage | 72.9 | 105.1 | 107.1 | 72.7
avg. words per question | 6.1 | 6.8 | 6.8 | 6.6
avg. words per answer | 3.5 | 2.0 | 1.9 | 3.6
avg. question-passage LCS | 11.6 | 12.9 | 13.0 | 13.1
Note. LCS stands for the longest common substring.
Table 2: KazQAD and NQ translated statistics.

The statistics of the RC part of KazQAD are presented in Table 2. Note that
the number of unique questions in the RC part is slightly lower than in the IR
part: for some passages that were considered relevant, the annotators could
not extract short answers. At the same time, each question can have more than
one answer-bearing passage, which is reflected in the higher number of unique
question-passage pairs. A relevant passage can contain several correct answers
that cannot be extracted as a single span (e.g., in the case of list
questions):

Q What five countries border the Caspian Sea? A Kazakhstan, Russia,
Azerbaijan, Turkmenistan, Islamic Republic of Iran (Translated)

The UNT and NQ parts are rather similar in terms of question length, but UNT
answers are significantly shorter. Note that the original UNT data comprise
question-answer pairs, and most of the answers were kept unchanged by the
annotators. The last row in the Table 2 estimates the similarity of questions
and answer-bearing passages using the length of the longest common
substring.555We employed the difflib library, see
https://docs.python.org/3/library/difflib.html. A high lexical overlap between
the question and the corresponding context makes the task of locating the
answer easier. High question-paragraph similarity was identified as a
shortcoming of the SQuAD dataset, where the annotators generated questions
based on presented Wikipedia paragraphs. Although a direct comparison between
the languages is not precise due to morphological differences, we can surmise
that the question-passage similarity in KazQAD is much lower than, for
example, in English SQuAD and Russian SberQuAD/XQuAD Efimov et al. (2020).

Kazakh |  English translation
---|---
Q:  Әйелдер хоккейі қашан олимпиадалық спорт түріне айналды? |  Q: When did women’s hockey become an Olympic sport?
1998 жылдан бастап Олимпиадаларда әйелдер арасындағы хоккей турнирі де өткізіле бастады. |  Since 1998, a women’s hockey tournament has also been held at the Olympics.
Q: Қазақ халқынан шыққан алғашқы ғарышкер кім? |  Q:  Who is the first Kazakh astronaut?
Тоқтар Оңғарбайұлы Әубәкіров (27 шілде 1946 жыл, Қарқаралы ауданы, Қарағанды облысы, Қазақстан) — қазақтан шыққан тұңғыш ғарышкер, ұшқыш, Кеңес Одағының Батыры (1988), Қазақстан Республикасының Халық қаһарманы (1995), техника ғылымының докторы (1998), профессор (1997), Қорқыт Ата атындағы Қызылорда мемлекеттік университетінің құрметті профессоры. Академик Е.А.Бөкетов атындағы Қарағанды университетінің Құрметті профессоры (3 мамыр 2022 жыл). |  Toktar Ongarbayuly Aubakirov (July 27, 1946, Karkaraly district, Karaganda region, Kazakhstan) — the first Kazakh cosmonaut, pilot, Hero of the Soviet Union (1988), People’s hero of the Republic of Kazakhstan (1995), doctor of technical sciences (1998), professor (1997), honorary professor of the Korkyt Ata Kyzylorda State University. Honorary professor of Karaganda University named after Academician Y. A. Boketov (May 3, 2022).
Table 3: Examples of question-passage-answer triples from the MRC part of
KazQAD (Underlined are the annotated answers). The top example comes from the
training set based on NQ: the question is machine-translated, while the
passage comes from the Kazakh Wikipedia. The bottom example is an original
Kazakh question (possibly rephrased by an annotator) from UNT, aligned with a
passage from the Kazakh Wikipedia.

During our review of the obtained annotations, it came to our attention that
there exist a notable number of lengthy answers, with 142 answers in the NQ
part exceeding 100 characters. Upon closer analysis, it became apparent that
the long answers can be attributed, in part, to the nature of the questions
themselves: Q What is the function of albumin in the blood? A forms a complex
connection with vitamins, microelements, hormones and transports them
throughout the body (Translated) and, in part, to the inclusion of excessively
detailed descriptions related to the answer entities: Q What part of the world
is Greece located in? A In the south-east of Europe, in the south of the
Balkan Peninsula and on small islands in the Ionian, Mediterranean and Aegean
seas adjacent to it. (Translated)

Table 3 shows two question-passage-answer triples from KazQAD: the first
features the translated question from the English NQ, the second an original
Kazakh question from UNT (albeit with slight manual changes from a multiple-
choice question). These examples illustrate that UNT contains more ‘localised’
questions (at least in the subsections corresponding to history of Kazakhstan
and Kazakh literature). In this regard, we follow the recommendations
formulated by the creators of multilingual QA datasets that the data should
reflect the cultural, historical, and geographical context whenever possible
Longpre et al. (2020); Clark et al. (2020).

## 4\. Methods and Baselines

To showcase the difficulty of the dataset, we implemented and evaluated models
for both RC and IR models. Subsequently, we integrated retrieval and reading
comprehension modules to evaluate the performance of a complete ODQA system.
Prior to discussing the experimental results, we provide an overview of the
Transformer models employed as backbones for the IR, RC, or both systems.

#### Backbone Transformer Models

We use several Transformer-based models for our baseline solutions. Kazakh is
featured in popular pre-trained multilingual models, such as mBERT Devlin et
al. (2019) and XLM-R Conneau et al. (2019). Multilingual models enable cross-
lingual transfer learning, wherein the model is fine-tuned with data in one
language and subsequently applied to another language. mBERT is a multilingual
variant of BERT pre-trained on a mixture of the 104 largest Wikipedias. mBERT
has a shared 110,000 vocabulary, 12 hidden layers with 12 attention heads.
XLM-R Conneau et al. (2019) is a multilingual variant of RoBERTa, which in
turn follows the learning regime of BERT with some optimizations, trained on a
100-language corpus from Common Crawl. We use the Base version of the model
with a 250,000-token vocabulary and 12 hidden layers with 12 attention heads.
This model outperforms mBERT on a variety of tasks, especially in the case of
low-resourced languages. XLM-V Liang et al. (2023) is a successor to XLM-R
with an extended vocabulary aimed at overcoming the vocabulary bottleneck. The
architecture of XLM-V is similar to that of XLM-R, but has a vocabulary of one
million tokens, allowing for more meaningful tokenisation. XLM-V outperforms
its predecessor on multilingual natural language inference (NLI), NER, and QA
tasks. Kaz-RoBERTa666https://huggingface.co/kz-transformers/kaz-roberta-
conversational is a monolingual model trained on a collection of Kazakh texts
from Common Crawl, the Leipzig Corpora Collection, the OSCAR corpus, as well
as Kazakh books and news, with a total of about two billion tokens. The model
has a vocabulary of 52,000 tokens and features six hidden layers, in contrast
to the 12 layers found in other Transformer-based models in the study, with 12
attention heads.

It is important to note that the quantity of language data available for
Kazakh is relatively small compared to the entire training corpora of the
aforementioned multilingual models. For example, the number of Wikipedia
Kazakh articles used for mBERT pre-training, is about 30 times smaller than
the number of English articles. Moreover, because Kazakh articles are
typically shorter than English ones, the gap is even larger when the number of
tokens is taken into account. XLM-R was pre-trained on a larger multilingual
corpus derived from Common Crawl, in which Kazakh is 44th out of 100 languages
by the volume of the training data. In terms of the number of tokens, the
Kazakh subcorpus is 117 times smaller than the English subcorpus and 5.8 times
smaller than the Turkish subcorpus Conneau et al. (2019). The size of the
language data used for pre-training and the size of the model vocabulary
allocated to the language are among the main factors that impact the quality
of the model fine-tuned on downstream tasks and cross-lingual learning
performance Lauscher et al. (2020); Artetxe et al. (2020).

#### IR baselines

We implemented a classic filter-and-refine multi-stage retrieval _pipeline_
Matveeva et al. (2006), where top-$k$ (k=$10^{4}$) candidate documents
obtained with a fast BM25 Robertson et al. (2009) ranker are re-ranked using a
more accurate (but slower) ranker. Our experiments are conducted using the
FlexNeuART Boytsov and Nyberg (2020) framework.

Our candidate generator relies on whitespace tokenisation, which includes
lowercase conversion and punctuation removal. The index is built over the
concatenation of the title of the Wikipedia page and the passage text. Using
this candidate generator, we implemented two non-neural rankers that employ a
simple linear learning-to-rank (LETOR) algorithm, namely coordinate ascent, to
combine several scores. Linear weights are found using a subset of the
training set. The first non-neural ranker is a multi-field BM25 baseline,
which includes separate scores for the title and the passage text. Each of
these fields undergoes dual tokenisation: whitespace tokenisation and mBERT
tokenisation, resulting in a total of four fields.

The second ranker includes scores for the monolingual IBM Model 1 Brown et al.
(1993), a lexical translation model that requires a parallel corpus. As a
parallel corpus, we use questions paired with answer-bearing text snippets. To
find these snippets, we use BM25 to retrieve paragraphs and then select
substrings that contain the answer text. We restrict the snippets to include a
maximum of five additional words on both the left and right sides of the
answer.

The top-100 results produced by the best non-neural ranker (from $10^{4}$ BM25
candidates) are re-ranked with a standard cross-encoding BERT ranker Nogueira
and Cho (2019), which uses XLM-R or mBERT as its backbone model. The ranker is
first pre-trained on English MS MARCO (passage corpus) Craswell et al. (2020)
and then fine-tuned on the KazQAD data. Although KazQAD has manually annotated
relevance passages, we found that better results are obtained when training
with weakly supervised data Karpukhin et al. (2020). Similar to the creation
of the parallel corpus, we first retrieved paragraphs using BM25. The
paragraphs containing the answer text were deemed to be relevant. Note,
however, that, the evaluation results are based on human-provided relevance
data.

We evaluated both zero-shot and fine-tuned rankers. According to the results
in Table 4, both XLM-R and mBERT zero-shot rankers have strong performance,
which improves only by about 5% after fine-tuning using the KazQAD data. Thus,
our best pipeline uses the fine-tuned XLM-R ranker.

IR pipeline | NDCG@10 | MRR | R@100
---|---|---|---
candidate generators
BM25 (title+text) | 0.1446 | 0.1443 | 0.4491
non-neural/classic re-rankers
BM25 (multi-field) | 0.1992 | 0.1961 | 0.5559
BM25+Model1 (multi-field) | 0.2735 | 0.2723 | 0.6478
best classic + neural re-rankers
mBERT (zero-shot) | 0.345 | 0.3344 | 0.6478
mBERT (fine-tuned) | 0.3672 | 0.3628 | 0.6478
XLM-R (zero-shot) | 0.3764 | 0.3654 | 0.6478
XLM-R (fine-tuned) | 0.3892 | 0.3822 | 0.6478
Table 4: Effectiveness of baseline IR systems.

#### RC baselines

We conducted reading comprehension experiments using three pre-trained
Transformer-based models: Kaz-RoBERTa, XLM-R, and XLM-V. We fine-tuned these
models in several scenarios: on 61,000 machine-translated question-paragraph
pairs (NQtransl.), on 69,000 English question-passage pairs (NQen) that were
not used in the creation of KazQAD, on the KazQAD train set, and on their
combinations.

To fine-tune the models, we used scripts and parameters from the official
Hugging Face
repository.777https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-
answering For each training data configuration, we tried different numbers of
epochs and selected the best model based on the KazQAD validation set. In
scenarios with two data sources, we first selected the best model that was
tuned on the first dataset and then tuned it on the second dataset.

Model | Train | EM | F1
---|---|---|---
Kaz-RoBERTa | NQtransl. | 20.86 | 31.82
| KazQAD | 10.25 | 18.31
| NQtransl.+KazQAD | 22.45 | 35.46
XLM-RoBERTa | NQen | 32.62 | 47.18
| NQtransl. | 32.33 | 47.75
| KazQAD | 24.33 | 37.49
| NQen+KazQAD | 37.38 | 52.67
| NQtransl.+KazQAD | 36.56 | 51.87
XLM-V | NQen+KazQAD | 38.52 | 54.18
| NQtransl.+KazQAD | 38.48 | 52.78
Table 5: Effectiveness of baseline MRC models.

The results for different models and training set configurations are presented
in Table 5. We found that the monolingual Kaz-RoBERTa model performs
dramatically worse than the multilingual models. This may be due to the fact
Kaz-RoBERTa is a smaller model with only six hidden layers.

Another surprising result was that the multilingual models fine-tuned on
English and on the translated NQ show similar performance. Furthermore, when
we trained XLM-R on the manually annotated KazQAD training set, we obtained
much lower scores. However, additional fine-tuning on KazQAD improved both EM
and F1 by about five points. Using XLM-V instead of XLM-R further increased EM
and F1 by about two points.

IR-pipeline | Fusion | EM | F1
---|---|---|---
non-neural/classic retriever
BM25 (title+text) | ✗ | 7.7 | 12.6
BM25 (multi-field) | ✗ | 8.5 | 14.3
BM25+Model1 (multi-field) | ✗ | 12.4 | 19.8
BM25 (title+text) + neural ranker
XLM-R (fine-tuned) | ✗ | 15.2 | 25.2
best classic + neural ranker
XLM-R (fine-tuned) | ✗ | 17.3 | 28.1
XLM-R (fine-tuned) | ✓ | 17.8 | 28.7
Table 6: ODQA baselines (XLM-V reader).

#### ODQA baselines

We test our best reader (XLM-V model) in the ODQA setting Hirschman and
Gaizauskas (2001), where the reader extracts answers from top-$k$ passages
produced by an IR system (we use $k=100$). Because each passage generates an
answer, there should be a final answer aggregation and/or selection process.
We tried two simple approaches: using the top-1 retrieved document and
_fusion_ of the retriever and reader scores (with a subsequent selection of
the top-fusion-score answer) Yang et al. (2019). Fusion scores were estimated
using the development set.

In Table 6, we show QA accuracy for progressively improving retrieval systems.
Using BM25 alone for concatenated title and text fields produced a very low F1
score of 12.6, which increased 1.6 times when we used a strong classic IR
pipeline BM25+Model1 (multi-field). A further 1.4 times increase was achieved
when we additionally applied a neural ranker.

The neural ranker was quite helpful, even if we used it on top of our weakest
IR system (see scores for “BM25 (title + text) + neural ranker”). However, the
resulting score was still 1.1 times lower compared to our top-system without
fusion. This underscores the need for a carefully optimised retriever.
Unfortunately, fusing a reader and retriever scores was only marginally
helpful.

#### ChatGPT

We fed questions from the KazQAD test set to the OpenAI model
gpt-3.5-turbo-0125888https://platform.openai.com/docs/models/gpt-3-5-turbo
preceded by a simple prompt in English: “The question is in the Kazakh
language. Provide a short and concise answer in Kazakh.” Given the potential
verbosity of ChatGPT responses, we employed the recall of lemmatised words as
an evaluation metric. In instances where multiple answers were available in
KazQAD, we concatenated them for consistency. Tokenisation and lemmatisation
were carried out using the Stanza library Qi et al. (2020). An additional
evaluation criterion was the length of the longest common substring between
the responses generated by ChatGPT and those provided in KazQAD. Of the 1,927
test questions, 173 (9%) elicited responses with a recall score of 0.5 or
higher, with only 86 (4.5%) achieving a perfect recall score. Manual
examination of the results revealed instances where ChatGPT provided
conspicuously inaccurate answers, such as incorrectly identifying
“Shakespeare” as Ablai Khan’s successor instead of the correct answer, “Uali”.
Furthermore, inaccuracies were observed in responses such as the erroneous
identification of Kazakh as the language of inter-ethnic communication in
Kazakhstan instead of Russian. However, despite these errors, the model
occasionally gave correct answers, albeit with small variations. For example,
while the KazQAD answer for Кеңес елінің басшысы М. С. Горбачевтің жүргізген
реформасының аты қандай? (What is the name of M. S. Gorbachev’s reform?) is
Қайта құру кезеңі, the original Russian term Перестройка, returned by ChatGPT,
is also considered correct, which illustrates variations in answer
presentation. Differences in character representation were also observed, such
as the use of Cyrillic characters instead of Latin characters (e.g., [витамин
(vitamin)] D in KazQAD vs. Д by ChatGPT), or the use of the Cyrillic dotted і
where the Cyrillic и is expected (e.g., Галилей, Италия, испандық in KazQAD
vs. Галілей, Італия, іспандық by ChatGPT ), which overall reflects the the
markedly inferior performance of OpenAI’s model in answering factual questions
in Kazakh and underscores the importance of manually annotated datasets over
machine-translated data.

## 5\. Conclusion

We introduce KazQAD: a Kazakh IR, RC, and ODQA dataset, which is one of the
few annotated NLP/IR resources for the Kazakh language. The annotated data are
publicly available together with a collection of Kazakh Wikipedia passages and
a machine-translated subset of the Natural Questions dataset. In preparing the
dataset, we tried to reuse existing resources and reduce the cost of manual
annotation. We also created a set of baselines for retrieval, reading
comprehension, and ODQA tasks. We hope that both our dataset preparation
approach and the results presented herein will contribute to research and
applications concerning Kazakh and other low-resource languages.

## 6\. Ethics Statement

We ensured that the annotators involved in this study received fair
compensation for their work, with the workload staying well within the bounds
of a standard working day.

## 7\. Bibliographical References

## References

  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _ACL_ , pages 4623–4637.
  * Attardi (2015) Giusepppe Attardi. 2015.  Wikiextractor.  https://github.com/attardi/wikiextractor.
  * Bajaj et al. (2016) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016.  Ms marco: A human generated machine reading comprehension dataset.  _arXiv preprint arXiv:1611.09268_.
  * Bandarkar et al. (2023) Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023.  The belebele benchmark: a parallel reading comprehension dataset in 122 language variants.  _arXiv preprint arXiv:2308.16884_.
  * Baroni and Bernardini (2006) Marco Baroni and Silvia Bernardini. 2006.  A New Approach to the Study of Translationese: Machine-Learning the Difference between Original and Translated Text.  _Literary and Linguistic Computing_ , 21(3):259–274.
  * Bonifacio et al. (2022) Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022.  Inpars: Unsupervised dataset generation for information retrieval.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2387–2392.
  * Bonifacio et al. (2021) Luiz Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti, Marzieh Fadaee, Roberto Lotufo, and Rodrigo Nogueira. 2021.  mmarco: A multilingual version of the ms marco passage ranking dataset.  _arXiv preprint arXiv:2108.13897_.
  * Boytsov and Nyberg (2020) Leonid Boytsov and Eric Nyberg. 2020.  Flexible retrieval with nmslib and flexneuart.  _arXiv preprint arXiv:2010.14848_.
  * Brown et al. (1993) Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993.  The mathematics of statistical machine translation: Parameter estimation.  _Computational Linguistics_ , 19(2):263–311.
  * Calijorne Soares and Parreiras (2020) Marco Antonio Calijorne Soares and Fernando Silva Parreiras. 2020.  A Literature Review on Question Answering Techniques, Paradigms and Systems.  _Journal of King Saud University - Computer and Information Sciences_ , 32(6):635–646.
  * Campbell and King (2020) George L Campbell and Gareth King. 2020.  _Compendium of the World’s Languages_.  Routledge.
  * Carrino et al. (2019) Casimiro Pio Carrino, Marta R Costa-jussà, and José AR Fonollosa. 2019.  Automatic Spanish translation of the SQuAD dataset for multilingual question answering.  _arXiv preprint arXiv:1912.05200_.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Clark et al. (2020) Jonathan H Clark et al. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _TACL_ , 8:454–470.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _arXiv preprint arXiv:1911.02116_.
  * Craswell et al. (2020) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020.  Overview of the TREC 2019 deep learning track.  _CoRR_ , abs/2003.07820.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Dobrov et al. (2004) Boris Dobrov, Igor Kuralenok, Natalia Loukachevitch, Igor Nekrestyanov, and Ilya Segalovich. 2004.  Russian information retrieval evaluation seminar.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_.
  * Efimov et al. (2020) Pavel Efimov, Andrey Chertok, Leonid Boytsov, and Pavel Braslavski. 2020.  SberQuAD–Russian reading comprehension dataset: Description and analysis.  In _International Conference of the Cross-Language Evaluation Forum for European Languages_ , pages 3–15.
  * Ferro and Peters (2019) Nicola Ferro and Carol Peters. 2019.  From multilingual to multimodal: the evolution of clef over two decades.  _Information Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF_ , pages 3–44.
  * Ferrucci et al. (2010) David A. Ferrucci, Eric W. Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya Kalyanpur, Adam Lally, J. William Murdock, Eric Nyberg, John M. Prager, Nico Schlaefer, and Christopher A. Welty. 2010.  Building watson: An overview of the deepqa project.  _AI Mag._ , 31(3):59–79.
  * Fu et al. (2023) Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023.  Gptscore: Evaluate as you desire.  _arXiv preprint arXiv:2302.04166_.
  * Goyal et al. (2022) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022.  The flores-101 evaluation benchmark for low-resource and multilingual machine translation.  _Transactions of the Association for Computational Linguistics_ , 10:522–538.
  * Hirschman and Gaizauskas (2001) Lynette Hirschman and Robert Gaizauskas. 2001.  Natural language question answering: the view from here.  _natural language engineering_ , 7(4):275–300.
  * Jiao et al. (2023) Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023.  Is chatgpt a good translator? a preliminary study.  _arXiv preprint arXiv:2301.08745_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural Questions: A Benchmark for Question Answering Research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Lai et al. (2023) Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023.  Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning.  _arXiv preprint arXiv:2304.05613_.
  * Lauscher et al. (2020) Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020.  From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4483–4499.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020.  MLQA: Evaluating cross-lingual extractive question answering.  In _ACL_ , pages 7315–7330.
  * Liang et al. (2023) Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023.  Xlm-v: Overcoming the vocabulary bottleneck in multilingual masked language models.  _arXiv preprint arXiv:2301.10472_.
  * Longpre et al. (2020) Shayne Longpre, Yi Lu, and Joachim Daiber. 2020.  Mkqa: A linguistically diverse benchmark for multilingual open domain question answering.  _arXiv preprint arXiv:2007.15207_.
  * Matveeva et al. (2006) Irina Matveeva, Chris Burges, Timo Burkard, Andy Laucius, and Leon Wong. 2006.  High accuracy retrieval with multiple nested ranker.  In _SIGIR_ , pages 437–444. ACM.
  * Nogueira and Cho (2019) Rodrigo Frassetto Nogueira and Kyunghyun Cho. 2019.  Passage re-ranking with BERT.  _CoRR_ , abs/1901.04085.
  * Prager (2006) John M. Prager. 2006.  Open-domain question-answering.  _Found. Trends Inf. Retr._ , 1(2):91–231.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020.  Stanza: A python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 101–108, Online. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4):333–389.
  * Rogers et al. (2023) Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023.  Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension.  _ACM Computing Surveys_ , 55(10):1–45.
  * Ruder (2022) Sebastian Ruder. 2022.  The State of Multilingual AI.  http://ruder.io/state-of-multilingual-ai/.
  * Ünlü Menevşe et al. (2022) Merve Ünlü Menevşe, Yusufcan Manav, Ebru Arisoy, and Arzucan Özgür. 2022.  A framework for automatic generation of spoken question-answering data.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4659–4666, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Voorhees and Harman (2005) Ellen M Voorhees and Donna K Harman. 2005.  _TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing)_.  The MIT Press.
  * Yang et al. (2019) Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019.  End-to-end open-domain question answering with BERTserini.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 72–77.
  * Yeshpanov et al. (2022) Rustem Yeshpanov, Yerbolat Khassanov, and Huseyin Atakan Varol. 2022.  KazNERD: Kazakh named entity recognition dataset.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 417–426, Marseille, France. European Language Resources Association.
  * Zhang et al. (2021) Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021.  Mr. tydi: A multi-lingual benchmark for dense retrieval.  _arXiv preprint arXiv:2108.08787_.
  * Zhang et al. (2022) Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2022.  Making a miracl: Multilingual information retrieval across a continuum of languages.  _arXiv preprint arXiv:2210.09984_.
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021.  Retrieving and reading: A comprehensive survey on open-domain question answering.  _arXiv preprint arXiv:2101.00774_.