# Beyond English-Only Reading Comprehension:
Experiments in Zero-Shot Multilingual Transfer for Bulgarian

Momchil Hardalov1 Ivan Koychev1 Preslav Nakov2
1Sofia University “St. Kliment Ohridski”, Bulgaria,
2Qatar Computing Research Institute, HBKU, Qatar,
{hardalov<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Recently, reading comprehension models achieved near-human performance on
large-scale datasets such as SQuAD, CoQA, MS Macro, RACE, etc. This is largely
due to the release of pre-trained contextualized representations such as BERT
and ELMo, which can be fine-tuned for the target task. Despite those advances
and the creation of more challenging datasets, most of the work is still done
for English. Here, we study the effectiveness of multilingual BERT fine-tuned
on large-scale English datasets for reading comprehension (e.g., for RACE),
and we apply it to Bulgarian multiple-choice reading comprehension. We propose
a new dataset containing 2,221 questions from matriculation exams for twelfth
grade in various subjects —history, biology, geography and philosophy—, and
412 additional questions from online quizzes in history. While the quiz
authors gave no relevant context, we incorporate knowledge from Wikipedia,
retrieving documents matching the combination of question + each answer
option. Moreover, we experiment with different indexing and pre-training
strategies. The evaluation results show accuracy of 42.23%, which is well
above the baseline of 24.89%.

##  1 Introduction

The ability to answer questions is natural to humans, independently of their
native language, and, once learned, it can be easily transferred to another
language. After understanding the question, we typically depend on our
background knowledge, and on relevant information from external sources.

Machines do not have the reasoning ability of humans, but they are still able
to learn concepts. The growing interest in teaching machines to answer
questions posed in natural language has led to the introduction of various new
datasets for different tasks such as reading comprehension, both extractive,
e.g., span-based (Nguyen et al., 2016; Trischler et al., 2017; Joshi et al.,
2017; Rajpurkar et al., 2018; Reddy et al., 2019), and non-extractive, e.g.,
multiple-choice questions (Richardson et al., 2013; Lai et al., 2017; Clark et
al., 2018; Mihaylov et al., 2018; Sun et al., 2019a). Recent advances in
neural network architectures, especially the raise of the Transformer (Vaswani
et al., 2017), and better contextualization of language models (Peters et al.,
2018; Devlin et al., 2019; Radford et al., 2018; Grave et al., 2018; Howard
and Ruder, 2018; Radford et al., 2019; Yang et al., 2019b; Dai et al., 2019)
offered new opportunities to advance the field.

Here, we investigate skill transfer from a high-resource language, i.e.,
English, to a low-resource one, i.e., Bulgarian, for the task of multiple-
choice reading comprehension. Most previous work (Pan et al., 2018; Radford et
al., 2018; Tay et al., 2018; Sun et al., 2019b) was monolingual, and a
relevant context for each question was available a priori. We take the task a
step further by exploring the capability of a neural comprehension model in a
multilingual setting using external commonsense knowledge. Our approach is
based on the multilingual cased BERT (Devlin et al., 2019) fine-tuned on the
RACE dataset (Lai et al., 2017), which contains over 87,000 English multiple-
choice school-level science questions. For evaluation, we build a novel
dataset for Bulgarian. We further experiment with pre-training the model over
stratified Slavic corpora in Bulgarian, Czech, and Polish Wikipedia articles,
and Russian news, as well as with various document retrieval strategies.

Finally, we address the resource scarceness in low-resource languages and the
absence of question contexts in our dataset by extracting relevant passages
from Wikipedia articles.

Our contributions are as follows:

  * •

We introduce a new dataset for reading comprehension in a low-resource
language such as Bulgarian. The dataset contains a total of 2,636 multiple-
choice questions without contexts from matriculation exams and online quizzes.
These questions cover a large variety of science topics in biology,
philosophy, geography, and history.

  * •

We study the effectiveness of zero-shot transfer from English to Bulgarian for
the task of multiple-choice reading comprehension, using Multilingual and
Slavic BERT (Devlin et al., 2019), fine-tuned on large corpora, such as RACE
(Lai et al., 2017).

  * •

We design a general-purpose pipeline111The dataset and the source code are
available at http://github.com/mhardalov/bg-reason-BERT for extracting
relevant contexts from an external corpus of unstructured documents using
information retrieval.

The rest of this paper is organized as follows: The next section presents
related work. Section 3 describes our approach. Details about the newly-
proposed multiple-choice Bulgarian dataset are given in Section 4. All
experiments are described in Section 5. Finally, Section 6 concludes and
points to possible directions for future work.

##  2 Related Work

###  2.1 Machine Reading Comprehension

The growing interest in machine reading comprehension (MRC) has led to the
release of various datasets for both extractive (Nguyen et al., 2016;
Trischler et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018; Reddy et
al., 2019) and non-extractive (Richardson et al., 2013; Peñas et al., 2014;
Lai et al., 2017; Clark et al., 2018; Mihaylov et al., 2018; Sun et al.,
2019a) comprehension. Our work primarily focuses on the non-extractive
multiple-choice type, designed by educational experts, since their task is
very close to our newly-proposed dataset, and are expected to be well-
structured and error-free (Sun et al., 2019a).

These datasets brought a variety of models and approaches. The usage of
external knowledge has been an interesting topic, e.g., Chen et al. (2017a)
used Wikipedia knowledge for answering open-domain questions, Pan et al.
(2018) applied entity discovery and linking as a source of prior knowledge.
Sun et al. (2019b) explored different reading strategies such as back and
forth reading, highlighting, and self-assessment. Ni et al. (2019) focused on
finding essential terms and removing distraction words, followed by
reformulation of the question, in order to find better evidence before sending
a query to the MRC system. A simpler approach was presented by Clark et al.
(2016), who leveraged information retrieval, corpus statistics, and simple
inference over a semi-automatically constructed knowledge base for answering
fourth-grade science questions.

Current state-of-the-art approaches in machine reading comprehension are
grounded on transfer learning and fine-tuning of language models (Peters et
al., 2018; Conneau et al., 2018; Devlin et al., 2019). Yang et al. (2019a)
presented an open-domain extractive reader based on BERT (Devlin et al.,
2019). Radford et al. (2018) used generative pre-training of a Transformer
(Vaswani et al., 2017) as a language model, transferring it to downstream
tasks such as natural language understanding, reading comprehension, etc.

Finally, there has been a Bulgarian MRC dataset (Peñas et al., 2012). It was
used by Simov et al. (2012), who converted the question-answer pairs to
declarative sentences, and measured their similarity to the context,
transforming both to a bag of linguistic units: lemmata, POS tags, and
dependency relations.

###  2.2 (Zero-Shot) Multilingual Models

Multilingual embeddings helped researchers to achieve new state-of-the-art
results on many NLP tasks. While many pre-trained model (Grave et al., 2018;
Devlin et al., 2019; Lample and Conneau, 2019) are available, the need for
task-specific data in the target language still remains. Learning such models
is language-independent, and representations for common words remain close in
the latent vector space for a single language, albeit unrelated for different
languages. A possible approach to overcome this effect is to learn an
alignment function between spaces (Artetxe and Schwenk, 2018; Joty et al.,
2017).

Moreover, zero-shot application of fine-tuned multilingual language models
(Devlin et al., 2019; Lample and Conneau, 2019) on XNLI (Conneau et al.,
2018), a corpus containing sentence pairs annotated with textual entailment
and translated into 14 languages, has shown very close results to such by a
language-specific model.

Zero-shot transfer and multilingual models had been a hot topic in (neural)
machine translation (MT) in the past several years. Johnson et al. (2017)
introduced a simple tweak to a standard sequence-to-sequence (Sutskever et
al., 2014) model by adding a special token to the encoder’s input, denoting
the target language, allowing a zero-shot learning for new language pairs.
Recent work in zero-resource translation outlined different strategies for
learning to translate without having a parallel corpus between the two target
languages. First, a many-to-one approach was adopted by Firat et al. (2016)
based on building a corpus from a single language paired with many others,
allowing simultaneous training of multiple models, with a shared attention
layer. A many-to-many relationship between languages was later used by Aharoni
et al. (2019), in an attempt to train a single Transformer (Vaswani et al.,
2017) model.

Pivot-language approaches can also be used to overcome the lack of parallel
corpora for the source–target language pair. Chen et al. (2017b) used a
student-teacher framework to train an NMT model, using a third language as a
pivot. A similar idea was applied to MRC by Asai et al. (2018), who translated
each question to a pivot language, and then found the correct answer in the
target language using soft-alignment attention scores.

##  3 Model

Our model has three components: (i) a context retrieval module, which tries to
find good explanatory passages for each question-answer pair, from a corpus of
non-English documents, as described in Section 3.1, (ii) a multiple-choice
reading comprehension module pre-trained on English data and then applied to
the target language in a zero-shot fashion, i.e., without further training or
additional fine-tuning, to a target (non-English) language, as described in
Section 3.2, and (iii) a voting mechanism, described in Section 3.3, which
combines multiple passages from (i) and their scores from (ii) in order to
obtain a single (most probable) answer for the target question.

###  3.1 Context Retriever

Most public datasets for reading comprehension (Richardson et al., 2013; Lai
et al., 2017; Sun et al., 2019a; Rajpurkar et al., 2018; Reddy et al., 2019;
Mihaylov et al., 2018) contain not only questions with possible answers, but
also an evidence passage for each question. This limits the task to question
answering over a piece of text, while an open-domain scenario is much more
challenging and much more realistic. Moreover, a context in which the answer
can be found is not easy to retrieve, sometimes even for a domain expert.
Finally, data scarceness in low-resource languages poses further challenges
for finding resources and annotators.

In order to enable search for appropriate passages for non-English questions,
we created an inverted index from Wikipedia articles using
Elasticsearch.222http://www.elastic.co/ We used the original dumps for the
entire Wikipage,333http://dumps.wikimedia.org/ and we preprocessed the data
leaving only plain textual content, e.g., removing links, HTML tags, tables,
etc. Moreover, we split the article’s body using two strategies: a sliding
window and a paragraph-based approach. Each text piece with its corresponding
article title was processed by applying word-based tokenization, lowercasing,
stop-words removal, stemming (Nakov, 2003; Savoy, 2007), and $n$-gram
extraction. Finally, the matching between a question and a passage was done
using cosine similarity and BM25 Robertson and Zaragoza (2009).

###  3.2 BERT for Multiple-Choice RC

The recently-proposed BERT (Devlin et al., 2019) framework is applicable to a
vast number of NLP tasks. A shared characteristic between all of them is the
form of the input sequences: a single sentence or a pair of sentences
separated by the [SEP] special token, and a classification token ([CLS]) added
at the beginning of each example. In contrast, the input for multiple-choice
reading comprehension questions is assembled by three sentence pieces, i.e.,
context passage, question, and possible answer(s). Our model follows a simple
strategy of concatenating the option (candidate answer) at the end of a
question. Following the notation of Devlin et al. (2019), the input sequence
can be written as follows:

_[CLS] Passage [SEP] Question + Option [SEP]_

Figure 1: BERT for multiple-choice reasoning.

As recommended by Devlin et al. (2019), we introduce a new task-specific
parameter vector $L$, $L\in\mathbb{R}^{H}$, where $H$ is the hidden size of
the model. In order to obtain a score for each passage-question-answer
triplet, we take the dot product between $L$ and the final hidden vector for
the classification token ([CLS]), thus ending up with $N$ unbounded numbers:
one for each option. Finally, we normalize the scores by adding a softmax
layer, as shown in Figure 1. During fine-tuning, we optimize the model’s
parameters by maximizing the log-probability of the correct answer.

###  3.3 Answer Selection Strategies

Finding evidence passages that contain information about the correct answer is
crucial for reading comprehension systems. The context retriever may be
extremely sensitive to the formulation of a question. The latter can be very
general, or can contain insignificant rare words, which can bias the search.
Thus, instead of using only the first-hit document, we should also evaluate
lower-ranked ones. Moreover, knowing the answer candidates can enrich the
search query, resulting in improved, more answer-oriented passages. This
approach leaves us with a set of contexts that need to be evaluated by the MRC
model in order to choose a single correct answer. Prior work suggests several
different strategies: Chen et al. (2017a) used the raw predicted probability
from a recurrent neural network (RNN), Yang et al. (2019a) tuned a hyper-
parameter to balance between the retriever score and the reading model’s
output, while Pan et al. (2018) and Ni et al. (2019) concatenated the results
from sentence-based retrieval into a single contextual passage.

In our experiments below, we adopt a simple summing strategy. We evaluate each
result from the context retriever against the question and the possible
options (see Section 3.2 for more details), thus obtaining a list of raw
probabilities. We found empirically that explanatory contexts assign higher
probability to the related answer, while general or uninformative passages
lead to stratification of the probability distribution over the answer
options. We formulate this as follows:

$\
Pr(a_{j}|p;q)=\frac{exp(BERT(p,q+a_{j}))}{\sum_{j\prime}{exp(BERT(p,q+a_{j}\prime))}},\vspace{-0.1in}$
(1)

where $p$ is a passage, $q$ is a question, $A$ is the set of answer
candidates, and $a_{j}\in A$.

We select the final answer as follows:

$Ans=\operatorname*{arg\,max}_{a\in A}\sum_{p\in P}{Pr(A|p;q)}$ (2)

##  4 Data

Our goal is to build a task for a low-resource language, such as Bulgarian, as
close as possible to the multiple-choice reading comprehension setup for high-
resource languages such as English. This will allow us to evaluate the
limitations of transfer learning in a multilingual setting. One of the largest
datasets for this task is RACE (Lai et al., 2017), with a total of 87,866
training questions with four answer candidates for each. Moreover, there are
25,137 contexts mapped to the questions and their correct answers.

Domain | #QA-pairs | #Choices | Len Question | Len Options | Vocabulary Size
---|---|---|---|---|---
12th Grade Matriculation Exam
Biology | 437 | 4 | $10.437\,070\,938\,215\,102$ | $2.643\,592\,677\,345\,538$ |  $2,414$ ($12,922$)
Philosophy | 630 | 4 | $8.909\,523\,809\,523\,81$ | $2.938\,888\,888\,888\,889$ |  $3,636$ ($20,392$)
Geography | 612 | 4 | $12.830\,065\,359\,477\,125$ | $2.466\,094\,771\,241\,83$ |  $3,239$ ($17,668$)
History | 542 | 4 | $23.737\,614\,678\,899\,08$ | $3.644\,036\,697\,247\,706\,5$ |  $5,466$ ($20,456$)
Online History Quizzes
Bulgarian History | $229$ | 4 | $14.048\,034\,934\,497\,817$ | $2.801\,310\,043\,668\,122\,4$ |  $2,287$ ($10,620$)
PzHistory | 183 | 3 | $38.885\,245\,901\,639\,344$ | $2.435\,336\,976\,320\,582\,5$ |  $1,261$ ($7,518$)
Overall | $2,633$ | $3.93$ | $15.666\,160\,849\,772\,382$ | $2.886\,883\,505\,453\,141\,7$ |  $13,329$ ($56,104$)
RACE Train - Mid and High School
RACE-M | $25,421$ | $4$ | $9.0$ | $3.9$ | $32,811$
RACE-H | $62,445$ | $4$ | $10.4$ | $5.8$ | $125,120$
Overall | $87,866$ | $4$ | $10.0$ | $5.3$ | $136,629$
Table 1: Statistics about our Bulgarian dataset compared to the RACE dataset.

While there exist many datasets for reading comprehension, most of them are in
English, and there are a very limited number in other languages (Peñas et al.,
2012, 2014). Hereby, we collect our own dataset for Bulgarian, resulting in
2,633 multiple-choice questions, without contexts, from different subjects:
biology (16.6%), philosophy (23.93%), geography (23.24%), and history
(36.23%). Table 2 shows an example question with candidate answers chosen to
represent best each category. We use green to mark the correct answer, and
bold for the question category. For convenience all the examples are
translated to English.

(Biology) The thick coat of mammals in winter is an example of:
A. physiological adaptation
B. behavioral adaptation
C. genetic adaptation
D. morphological adaptation

(Philosophy) According to relativism in ethics:
A. there is only one moral law that is valid for all
B. there is no absolute good and evil
C. people are evil by nature
D. there is only good, and the evil is seeming

(Geography) Which of the assertions about the economic specialization of the
Southwest region is true?
A. The ratio between industrial and agricultural production is 15:75
B. Lakes of glacial origin in Rila and Pirin are a resource for the
development of tourism
C. Agricultural specialization is related to the cultivation of grain and
ethereal-oil crops
D. The rail transport is of major importance for intra-regional connections

(History) Point out the concept that is missed in the text of the Turnovo
Constitution: ,,Art. 54 All born in Bulgaria, also those born elsewhere by
parents Bulgarian  , count as  of the Bulgarian Principality. Art. 78 Initial
teaching is free and obligatory for all  of the Bulgarian Principality.”
A. residents
B. citizents
C. electors
D. voters

(History Quiz) Sofroniy Vrachanski started a family that plays a big role in
the history of the Bulgarian National Revival. What is its name?
A. Georgievi
B. Tapchileshtovi
C. Bogoridi
D. Palauzovi

Table 2: Example questions, one per subject, from our Bulgarian dataset. The
correct answer is marked in green.

Table 1 shows the distribution of questions per subject category, the length
(in words) for both the questions and the options (candidate answers), and the
vocabulary richness, measured in terms of unique words. The first part of the
table presents statistics about our dataset, while the second part is a
comparison to RACE (Lai et al., 2017).

We divided the Bulgarian questions into two groups based on the question’s
source. The first group (12th Grade Matriculation Exam) was collected from
twelfth grade matriculation exams created by the Ministry of Education of
Bulgaria in the period 2008–2019. Each exam contains thirty multiple-choice
questions with four possible answers per question. The second set of questions
(Online History Quizzes) are history-related and are collected from online
quizzes. While they are not created by educators, the questions are still
challenging and well formulated. Furthermore, we manually filtered out
questions with non-textual content (i.e., pictures, paintings, drawings,
etc.), ordering questions (i.e., order the historical events), and questions
involving calculations (i.e., how much $X$ we need to add to $Y$ to arrive at
$Z$).

Table 1 shows that history questions in general contain more words
($14.0$–$38.9$ on average), compared to other subjects ($8.9$–$12.8$ on
average). A tangible difference in length compared to other subjects is seen
for 12th grade History and PzHistory, due to the large number of quotes, and
document pieces contained in questions from these two groups. Also, the
average question length is $15.666\,160\,849\,772\,382$, which is longer
compared to the RACE dataset with $10.0$. On the other hand, the option
lengths per subject category in our dataset follow a narrower distribution.
They fall in the interval between $2.5$ and $2.9$ words on average, expect for
12th grade History, with $3.6$ words. Here, we note a significant difference
compared to the option lengths in RACE, which tend to be $2.4$ words longer on
average – $5.3$ for RACE vs. $2.9$ for ours.

Finally, we examine the vocabulary richness of the two datasets. The total
number of unique words is shown in the last column of Table 1 (Vocab Size).
For our dataset, there are two numbers per row: the first one shows statistics
based on the question–answer pairs only, while the second one, enclosed in
parentheses, measures the vocabulary size including the extracted passages by
the Context Retriever. The latter number is a magnitude estimate rather then a
concrete number, since its upper limit is the number of words in Wikipedia,
and it can vary for different retrieval strategies.

##  5 Experiments and Evaluation

###  5.1 BERT Fine-Tuning

We divide the fine-tuning into two groups of models (i) Multilingual BERT, and
(ii) Slavic BERT. Table 3 below presents the results in the multiple-choice
comprehension task on the dev dataset from RACE Lai et al. (2017).

#Epoch | RACE-M | RACE-H | Overall
---|---|---|---
BERT 1 | $64.206\,128\,133\,704\,73$ | $53.659\,233\,847\,913\,09$ | $56.728\,820\,429\,671\,67$
BERT 2 | $68.802\,228\,412\,256\,27$ | $57.575\,757\,575\,757\,58$ | $60.843\,129\,306\,850\,43$
BERT 3 | $69.150\,417\,827\,298\,05$ | $58.433\,390\,508\,862\,2$ | $61.552\,492\,906\,364\,01$
Slavic 2 | $53.551\,532\,033\,426\,19$ | $44.482\,561\,463\,693\,54$ | $47.122\,010\,539\,116\,34$
Slavic 3 | $57.381\,615\,598\,885\,79$ | $46.883\,933\,676\,386\,51$ | $49.939\,197\,405\,755\,98$
Table 3: Accuracy measured on the dev RACE dataset after each training epoch.

#### Multilingual BERT

As our initial model, we use BERTbase, Multilingual Cased which is pre-trained
on 104 languages, and has 12-layers, 768-hidden units per layer, 12-heads, and
a total of 110M parameters. We further fine-tune the model on RACE Lai et al.
(2017) for 3 epochs saving a checkpoint after each epoch. We use a batch size
of 8, a max sequence size of 320, and a learning rate of 1e-5.

#### Slavic BERT

The Slavic model444http://github.com/deepmipt/Slavic-BERT-NER was built using
transfer learning from the Multilingual BERT model to four Slavic languages:
Bulgarian, Czech, Polish, and Russian. In particular, the Multilingual BERT
model was fine-tuned on a stratified dataset of Russian news and Wikipedia
articles for the other languages. We use this pre-trained Slavic BERT model,
and we apply the same learning procedure as for Multilingual BERT.

###  5.2 Wikipedia Retrieval and Indexing

Here, we discuss the retrieval setup (see Section 3.1 for more details). We
use the Bulgarian dump of Wikipedia from 2019-04-20, with a total of 251,507
articles. We index each article title and body in plain text, which we call a
_passage_. We further apply additional processing for each field:

  * •

ngram: word-based 1–3 grams;

  * •

bg: lowercased, stop-words removed (from Lucene), and stemmed (Savoy, 2007);

  * •

none: bag-of-words index.

We ended up using a subset of four fields from all the possible analyzer-field
combinations, namely title.bg, passage, passage.bg, and passage.ngram. We
applied Bulgarian analysis on the _title_ field only as it tends to be short
and descriptive, and thus very sensitive to noise from stop-words, which is in
contrast to questions that are formed mostly of stop-words, e.g., _what_ ,
_where_ , _when_ , _how_.

For indexing the Wikipedia articles, we adopt two strategies: sliding window
and paragraph. In the window-based strategy, we define two types of splits:
small, containing 80-100 words, and large, of around 300 words. In order to
obtain indexing chunks, we define a window of size $K$, and a stride equal to
one forth of $K$. Hence, each $\frac{K}{4}$ characters, which is the size of
the stride, are contained into four different documents. The paragraph-based
strategy divides the article by splitting it using one or more successive
newline characters ([\n]+) as a delimiter. We avoid indexing entire documents
due to their extensive length, which can be far beyond the maximum length that
BERT can take as an input, i.e., 320 word pieces (see Section 3 for the more
details). Note that extra steps are needed in order to extract a proper
passage from the text. Moreover, the amount of facts in the Wikipedia articles
that are unrelated to our questions give rise to false positives since the
question is short and term-unspecific.

Finally, we use a list of top-$N$ hits for each candidate answer. Thus, we
have to execute an additional query for each question + option combination,
which may result in duplicated passages, thus introducing an implicit bias
towards the candidates they support. In order to mitigate this effect, during
the answer selection phase (see Section 3.3), we remove all duplicate entries,
keeping a single instance.

###  5.3 Experimental Results

Here, we discuss the accuracy of each model on the original English MRC task,
followed by experiments in zero-shot transfer to Bulgarian.

#### English Pre-training for MCRC.

Table 3 presents the change in accuracy on the original English comprehension
task, depending on the number of training epochs. In the table, “BERT” refers
to the Multilingual BERT model, while “Slavic” stands for BERT with Slavic
pre-training. We further fine-tune the models on the RACE dataset. Next, we
report their performance in terms of accuracy, following the notation from
(Lai et al., 2017). Note that the questions in RACE-H are more complex than
those in RACE-M. The latter has more word matching questions and fewer
reasoning questions. The final column in the table, Overall, shows the
accuracy calculated over all questions in the RACE testset. We train both
setups for three epochs and we report their performance after each epoch. We
can see a positive correlation between the number of epochs and the model’s
accuracy. We further see that the Slavic BERT performs far worse on both
RACE-M and RACE-H, which suggests that the change of weights of the model
towards Slavic languages has led to catastrophic forgetting of the learned
English syntax and semantics. Thus, it should be expected that the adaptation
to Slavic languages would yield decrease in performance for English. What
matters though is whether this helps when testing on Bulgarian, which we
explore next.

Setting | Accuracy
---|---
Random | 24.89
Train for 3 epochs | –
\+ window & title.bg & pass.ngram | $29.62$
\+ passage.bg & passage | $39.35$
– title.bg | $39.69$
\+ passage.bg^2 | $40.26$
\+ title.bg^2 | $40.30$
\+ bigger window | $36.54$
\+ paragraph split | $42.23$
\+ Slavic pre-training | $33.270\,034\,181\,541\,966$
Train for 1 epoch best | $40.258\,260\,539\,308\,77$
Train for 2 epochs best | $41.891\,378\,655\,526\,02$
Table 4: Accuracy on the Bulgarian testset: ablation study when sequentially
adding/removing different model components.

#### Zero-Shot Transfer.

Here, we assess the performance of our model when applied to Bulgarian
multiple-choice reading comprehension. Table 4 presents an ablation study for
various components. Each line denotes the type of the model, and the addition
(+) or the removal (–) of a characteristic from the setup in the previous
line. The first line shows the performance of a baseline model that chooses an
option uniformly at random from the list of candidate answers for the target
question. The following rows show the results for experiments conducted with a
model trained for three epochs on RACE (Lai et al., 2017).

Our basic model uses the following setup: Wikipedia pages indexed using a
small sliding window (400 characters, and stride of 100 characters), and
context retrieval over two fields: Bulgarian analyzed title (text.bg), and
word $n$-grams over the passage (passage.ngram). This setup yields 29.62%
accuracy, and it improves over the random baseline by 4.73% absolute. We can
think of it as a non-random baseline for further experiments. Next, we add two
more fields to the IR query: passage represented as a bag of words (named
passage), and Bulgarian analyzed (passage.bg), which improves the accuracy by
additional 10%, arriving at 39.35%. The following experiment shows that
removing the title.bg field does not change the overall accuracy, which makes
it an insignificant field for searching. Further, we add double weight on
passage.bg, (shown as ^2), which yields 1% absolute improvement.

From the experiments described above, we found the best combination of query
fields to be title.bulgarian^2, passage.ngram, passage, passage.bulgarian^2,
where the title has a minor contribution, and can be sacrificed for ease of
computations and storage. Fixing the best query fields, allowed us to evaluate
other indexing strategies, i.e., bigger window (size 1,600, stride 400) with
accuracy 36.54%, and paragraph splitting, with which we achieved our highest
accuracy of 42.23%. This is an improvement of almost 2.0% absolute over the
small sliding window, and 5.7% over the large one.

Next, we examined the impact of the Slavic BERT. Surprisingly, it yielded 9%
absolute drop in accuracy compared to the multi-lingual BERT. This suggests
that the latter already has enough knowledge about Bulgarian, and thus it does
not need further adaptation to Slavic languages.

Figure 2: Accuracy per question category based on the number of query results
per answer option.

Next, we study the impact of the number of fine-tuning epochs on the model’s
performance. We observe an increase in accuracy as the number of epochs grows,
which is in line with previously reported results for English tasks. While
this correlation is not as strong as for the original RACE task (see Table 3
for comparison), we still observe 1.6% and 0.34% absolute increase in accuracy
for epochs 2 and 3, respectively, compared to epoch 1. Note that we do not go
beyond three epochs, as previous work has suggested that 2-3 fine-tuning
epochs are enough Devlin et al. (2019), and after that, there is a risk of
catastrophic forgetting of what was learned at pre-training time (note that we
have already seen such forgetting with the Slavic BERT above).

We further study the impact of the size of the results list returned by the
retriever on the accuracy for the different categories. Figure 2 shows the
average accuracy for a given query size $S_{q}$ over all performed
experiments, where $S_{q}~{}\in~{}{\\{1,2,5,10,20\\}}$. We can see in Figure 2
that longer query result lists (i.e., containing more than 10 results) per
answer option worsen the accuracy for all categories, except for _biology_ ,
where we see a small peak at length 10, while still the best overall results
for this category is achieved for a result list of length 5. A single well-
formed maximum at length 2 is visible for _history_ and _philosophy_. With
these two categories being the biggest ones, the cap at the same number of
queries for the overall accuracy is not a surprise. The per-category results
for the experiments are discussed in more detail in Appendix A.

We can see that the highest accuracy is observed for _history_ , particularly
for online quizzes, which are not designed by educators and are more of a
word-matching nature rather then a reasoning one (see Table 2). Finally,
_geography_ appears to be the hardest category with only 38.73% accuracy: 3.5%
absolute difference compared to the second-worst category. The performance for
this subject is also affected differently by changes in query result length:
the peak is at lengths 5 and 10, while there is a drop for length 2. A further
study of the model’s behavior can be found in Appendix B.

##  6 Conclusion and Future Work

We studied the task of multiple-choice reading comprehension for low-resource
languages, using a newly collected Bulgarian corpus with 2,633 questions from
matriculation exams for twelfth grade in history and biology, and online exams
in history without explanatory contexts. In particular, we designed an end-to-
end approach, on top of a multilingual BERT model (Devlin et al., 2019), which
we fine-tuned on large-scale English reading comprehension corpora, and open-
domain commonsense knowledge sources (Wikipedia). Our main experiments
evaluated the model when applied to Bulgarian in a zero-shot fashion. The
experimental results found additional pre-training on the English RACE corpus
to be very helpful, while pre-training on Slavic languages to be harmful,
possibly due to catastrophic forgetting. Paragraph splitting, $n$-grams, stop-
word removal, and stemming further helped the context retriever to find better
evidence passages, and the overall model to achieve accuracy of up to
$42.233\,194\,075\,199\,39$%, which is well above the baselines of 24.89% and
29.62%.

In future work, we plan to make use of reading strategies (Sun et al., 2019b),
linked entities (Pan et al., 2018), concatenation and reformulation of
passages and questions (Simov et al., 2012; Clark et al., 2016; Ni et al.,
2019), as well as re-ranking of documents (Nogueira and Cho, 2019).

## Acknowledgments

We want to thank Desislava Tsvetkova and Anton Petkov for the useful
discussions and for their help with some of the experiments.

This research is partially supported by Project UNITe BG05M2OP001-1.001-0004
funded by the OP “Science and Education for Smart Growth” and co-funded by the
EU through the ESI Funds.

## References

  * Aharoni et al. (2019) Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019.  Massively multilingual neural machine translation.  In Proceedings of the Conference of the North American Chapter of ACL. Minneapolis, MN, USA, NAACL-HLT ’19, pages 3874–3884.
  * Artetxe and Schwenk (2018) Mikel Artetxe and Holger Schwenk. 2018.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  arXiv preprint arXiv:1812.10464 .
  * Asai et al. (2018) Akari Asai, Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2018.  Multilingual extractive reading comprehension by runtime machine translation.  arXiv preprint arXiv:1809.03275 .
  * Chen et al. (2017a) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017a.  Reading Wikipedia to answer open-domain questions.  In Proceedings of the Meeting of the Association for Computational Linguistics. Vancouver, Canada, ACL ’17, pages 1870–1879.
  * Chen et al. (2017b) Yun Chen, Yang Liu, Yong Cheng, and Victor O.K. Li. 2017b.  A teacher-student framework for zero-resource neural machine translation.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver, Canada, ACL ’17, pages 1925–1935.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? Try ARC, the AI2 reasoning challenge.  arXiv preprint arXiv:1803.05457 .
  * Clark et al. (2016) Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Turney, and Daniel Khashabi. 2016.  Combining retrieval, statistics, and inference to answer elementary science questions.  In Proceedings of the 13th AAAI Conference on Artificial Intelligence. Phoenix, AZ, USA, AAAI ’16, pages 2580–2586.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium, EMNLP ’18, pages 2475–2485.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019.  Transformer-XL: Attentive language models beyond a fixed-length context.  In Proceedings of the Meeting of the Association for Computational Linguistics. Florence, Italy, ACL ’19, pages 2978–2988.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Minneapolis, MN, USA, NAACL-HLT ’19, pages 4171–4186.
  * Firat et al. (2016) Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016.  Zero-resource translation with multi-lingual neural machine translation.  In Proc. of the Conference on Empirical Methods in Natural Language Processing. Austin, TX, USA, EMNLP ’16, pages 268–277.
  * Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018.  Learning word vectors for 157 languages.  In Proceedings of the Conference on Language Resources and Evaluation. Miyazaki, Japan, LREC ’18.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne, Australia, ACL ’18, pages 328–339.
  * Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017.  Google’s multilingual neural machine translation system: Enabling zero-shot translation.  Transactions of the Association for Computational Linguistics 5:339–351.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics. Vancouver, Canada, ACL ’17, pages 1601–1611.
  * Joty et al. (2017) Shafiq Joty, Preslav Nakov, Lluís Màrquez, and Israa Jaradat. 2017.  Cross-language learning with adversarial neural networks.  In Proc. of the Conference on Computational Natural Language Learning. Vancouver, Canada, CoNLL ’17, pages 226–237.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017.  RACE: Large-scale ReAding comprehension dataset from examinations.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark, EMNLP ’17, pages 785–794.
  * Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019.  Cross-lingual language model pretraining.  arXiv preprint arXiv:1901.07291 .
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? A new dataset for open book question answering.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium, EMNLP ’18, pages 2381–2391.
  * Nakov (2003) Preslav Nakov. 2003.  Building an inflectional stemmer for Bulgarian.  In Proceedings of the 4th International Conference Conference on Computer Systems and Technologies: E-Learning. Rousse, Bulgaria, CompSysTech ’03, pages 419–424.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset.  In Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches. Barcelona, Spain, CoCo@NIPS ’16.
  * Ni et al. (2019) Jianmo Ni, Chenguang Zhu, Weizhu Chen, and Julian McAuley. 2019.  Learning to attend on essential terms: An enhanced retriever-reader model for open-domain question answering.  In Proceedings of the Conference of the North American Chapter of ACL. Minneapolis, MN, USA, NAACL-HLT ’19, pages 335–344.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage re-ranking with BERT.  arXiv preprint arXiv:1901.04085 .
  * Pan et al. (2018) Xiaoman Pan, Kai Sun, Dian Yu, Heng Ji, and Dong Yu. 2018.  Improving question answering with external knowledge.  arXiv preprint:1902.00993 .
  * Peñas et al. (2012) Anselmo Peñas, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Richard Sutcliffe, Corina Forascu, Yassine Benajiba, and Petya Osenova. 2012.  Overview of QA4MRE at CLEF 2012: Question answering for machine reading evaluation.  In CLEF Working Note Papers. Rome, Italy, pages 1–24.
  * Peñas et al. (2014) Anselmo Peñas, Christina Unger, and Axel-Cyrille Ngonga Ngomo. 2014.  Overview of CLEF question answering track 2014.  In Information Access Evaluation. Multilinguality, Multimodality, and Interaction. pages 300–306.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics. New Orleans, LA, USA, NAACL-HLT ’18, pages 2227–2237.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training .
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners .
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In Proceedings of the Meeting of the Association for Computational Linguistics. Melbourne, Australia, ACL ’18, pages 784–789.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019.  CoQA: A conversational question answering challenge.  Transactions of the Association for Computational Linguistics 7:249–266.
  * Richardson et al. (2013) Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013.  MCTest: A challenge dataset for the open-domain machine comprehension of text.  In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle, WA, USA, EMNLP ’13, pages 193–203.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: BM25 and beyond.  Found. Trends Inf. Retr. 3(4):333–389.
  * Savoy (2007) Jacques Savoy. 2007.  Searching strategies for the Bulgarian language.  Inform. Retrieval 10(6):509–529.
  * Simov et al. (2012) Kiril Ivanov Simov, Petya Osenova, Georgi Georgiev, Valentin Zhikov, and Laura Tolosi. 2012.  Bulgarian question answering for machine reading.  In CLEF Working Note Papers. Rome, Italy.
  * Sun et al. (2019a) Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019a.  DREAM: A challenge data set and models for dialogue-based reading comprehension.  Transactions of the Association for Computational Linguistics 7:217–231.
  * Sun et al. (2019b) Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019b.  Improving machine reading comprehension with general reading strategies.  In Proceedings of the North American Chapter of ACL. Minneapolis, MN, USA, NAACL-HLT ’19, pages 2633–2643.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In Proceedings of the 27th Annual Conference on Neural Information Processing Systems. Montreal, Canada, NIPS ’14, pages 3104–3112.
  * Tay et al. (2018) Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018.  Multi-range reasoning for machine comprehension.  arXiv preprint arXiv:1803.09074 .
  * Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017.  NewsQA: A machine comprehension dataset.  In Proceedings of the 2nd Workshop on Representation Learning for NLP. Vancouver, Canada, RepL4NLP ’19, pages 191–200.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In Proceedings of the Annual Conference on Neural Information Processing Systems. Long Beach, CA, USA, NIPS ’17, pages 5998–6008.
  * Yang et al. (2019a) Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019a.  End-to-end open-domain question answering with BERTserini.  In Proceedings of the Conference of the North American Chapter of ACL. Minneapolis, MN, USA, NAACL-HLT ’19, pages 72–77.
  * Yang et al. (2019b) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019b.  XLNet: Generalized autoregressive pretraining for language understanding.  arXiv preprint arXiv:1906.08237 .

## Appendix

##  Appendix A Per-Category Results

Table 5 gives an overview, including per-category breakdown, of our parameter
tuning experiments. We present the results for some interesting experiments
rather then for a full grid search. The first row shows a random baseline for
each category. In the following rows, we compare different types of indexing:
first, we show the results for a small sliding window (400-character window,
and 100-character stride), followed by a big window (1,600-character window,
and 400-character stride), and finally for paragraph indexing. We use the same
notation as in Section 5. The last group in the table (Paragraph) shows the
best-performing model, where we mark in bold the highest accuracy for each
category. For completeness, we also show the accuracy when using the Slavic
BERT model for prediction, which yields a 10% drop on average compared to
using the Multilingual BERT, for each of the categories.

#docs | Overall | biology-12th | philosophy-12th | geography-12th | history-12th | history-quiz
---|---|---|---|---|---|---
Random
0 | $24.886\,191\,198\,786\,04$ | $26.086\,956\,521\,739\,13$ | $24.444\,444\,444\,444\,443$ | $24.183\,006\,535\,947\,71$ | $25.871\,559\,633\,027\,523$ | $24.029\,126\,213\,592\,23$
Window Small
title.bulgarian, passage.bulgarian
1 | $39.954\,424\,610\,710\,22$ | $40.274\,599\,542\,334\,094$ | $40.634\,920\,634\,920\,63$ | $34.967\,320\,261\,437\,905$ | $42.988\,929\,889\,298\,895$ | $41.990\,291\,262\,135\,92$
2 | $40.220\,281\,048\,233\,95$ | $40.274\,599\,542\,334\,094$ | $40.634\,920\,634\,920\,63$ | $35.947\,712\,418\,300\,654$ | $42.619\,926\,199\,261\,99$ | $42.718\,446\,601\,941\,75$
5 | $40.220\,281\,048\,233\,95$ | $38.901\,601\,830\,663\,616$ | $40.634\,920\,634\,920\,63$ | $38.071\,895\,424\,836\,605$ | $41.512\,915\,129\,151\,29$ | $42.475\,728\,155\,339\,81$
10 | $38.663\,121\,914\,166\,354$ | $40.503\,432\,494\,279\,174$ | $39.841\,269\,841\,269\,84$ | $35.457\,516\,339\,869\,28$ | $39.298\,892\,988\,929\,886$ | $38.834\,951\,456\,310\,68$
20 | $36.840\,106\,342\,575\,01$ | $37.528\,604\,118\,993\,14$ | $39.047\,619\,047\,619\,05$ | $33.823\,529\,411\,764\,7$ | $38.745\,387\,453\,874\,535$ | $34.708\,737\,864\,077\,67$
title.bulgarian, passage.ngram
1 | $28.940\,372\,199\,012\,533$ | $29.061\,784\,897\,025\,17$ | $32.063\,492\,063\,492\,06$ | $27.287\,581\,699\,346\,404$ | $27.490\,774\,907\,749\,078$ | $28.398\,058\,252\,427\,184$
2 | $29.092\,290\,163\,311\,812$ | $29.061\,784\,897\,025\,17$ | $33.333\,333\,333\,333\,336$ | $25.0$ | $28.782\,287\,822\,878\,228$ | $29.126\,213\,592\,233\,01$
5 | $29.054\,310\,672\,236\,99$ | $27.459\,954\,233\,409\,61$ | $32.063\,492\,063\,492\,06$ | $26.633\,986\,928\,104\,576$ | $30.627\,306\,273\,062\,732$ | $27.669\,902\,912\,621\,36$
10 | $29.624\,003\,038\,359\,287$ | $29.061\,784\,897\,025\,17$ | $32.539\,682\,539\,682\,54$ | $26.960\,784\,313\,725\,49$ | $30.073\,800\,738\,007\,38$ | $29.126\,213\,592\,233\,01$
20 | $29.434\,105\,582\,985\,19$ | $31.807\,780\,320\,366\,13$ | $32.698\,412\,698\,412\,696$ | $26.633\,986\,928\,104\,576$ | $28.597\,785\,977\,859\,78$ | $27.184\,466\,019\,417\,474$
title.bulgarian, passage.ngram, passage, passage.bulgarian
1 | $38.321\,306\,494\,492\,97$ | $38.215\,102\,974\,828\,376$ | $40.0$ | $34.477\,124\,183\,006\,54$ | $39.483\,394\,833\,948\,34$ | $40.048\,543\,689\,320\,39$
2 | $39.080\,896\,315\,989\,364$ | $37.070\,938\,215\,102\,98$ | $40.317\,460\,317\,460\,316$ | $34.477\,124\,183\,006\,54$ | $40.590\,405\,904\,059\,04$ | $44.174\,757\,281\,553\,4$
5 | $39.346\,752\,753\,513\,1$ | $40.961\,098\,398\,169\,334$ | $39.841\,269\,841\,269\,84$ | $34.640\,522\,875\,816\,99$ | $41.328\,413\,284\,132\,84$ | $41.262\,135\,922\,330\,096$
10 | $38.625\,142\,423\,091\,53$ | $40.503\,432\,494\,279\,174$ | $40.634\,920\,634\,920\,63$ | $33.496\,732\,026\,143\,79$ | $40.405\,904\,059\,040\,59$ | $38.834\,951\,456\,310\,68$
20 | $36.536\,270\,413\,976\,45$ | $38.672\,768\,878\,718\,536$ | $37.936\,507\,936\,507\,94$ | $31.372\,549\,019\,607\,842$ | $37.453\,874\,538\,745\,39$ | $38.592\,233\,009\,708\,735$
passage.ngram, passage, passage.bulgarian^2
1 | $39.688\,568\,173\,186\,48$ | $40.274\,599\,542\,334\,094$ | $40.634\,920\,634\,920\,63$ | $35.130\,718\,954\,248\,366$ | $42.066\,420\,664\,206\,64$ | $41.262\,135\,922\,330\,096$
2 | $40.258\,260\,539\,308\,77$ | $39.816\,933\,638\,443\,935$ | $40.952\,380\,952\,380\,95$ | $35.947\,712\,418\,300\,654$ | $42.619\,926\,199\,261\,99$ | $42.961\,165\,048\,543\,69$
5 | $39.574\,629\,699\,962\,02$ | $39.588\,100\,686\,498\,855$ | $39.365\,079\,365\,079\,37$ | $37.254\,901\,960\,784\,316$ | $40.959\,409\,594\,095\,94$ | $41.504\,854\,368\,932\,04$
10 | $38.701\,101\,405\,241\,17$ | $41.189\,931\,350\,114\,41$ | $39.523\,809\,523\,809\,526$ | $35.784\,313\,725\,490\,19$ | $39.298\,892\,988\,929\,886$ | $38.349\,514\,563\,106\,794$
20 | $37.143\,942\,271\,173\,565$ | $39.359\,267\,734\,553\,775$ | $37.777\,777\,777\,777\,78$ | $35.294\,117\,647\,058\,826$ | $38.376\,383\,763\,837\,64$ | $34.951\,456\,310\,679\,61$
title.bulgarian^2, passage.ngram, passage, passage.bulgarian^2
1 | $39.840\,486\,137\,485\,755$ | $40.274\,599\,542\,334\,094$ | $40.793\,650\,793\,650\,79$ | $35.130\,718\,954\,248\,366$ | $42.250\,922\,509\,225\,09$ | $41.747\,572\,815\,533\,98$
2 | $40.296\,240\,030\,383\,59$ | $40.274\,599\,542\,334\,094$ | $40.634\,920\,634\,920\,63$ | $36.111\,111\,111\,111\,114$ | $42.804\,428\,044\,280\,44$ | $42.718\,446\,601\,941\,75$
5 | $40.258\,260\,539\,308\,77$ | $39.130\,434\,782\,608\,695$ | $40.634\,920\,634\,920\,63$ | $38.398\,692\,810\,457\,52$ | $41.143\,911\,439\,114\,39$ | $42.475\,728\,155\,339\,81$
10 | $38.739\,080\,896\,315\,99$ | $40.503\,432\,494\,279\,174$ | $39.682\,539\,682\,539\,684$ | $35.620\,915\,032\,679\,74$ | $39.483\,394\,833\,948\,34$ | $39.077\,669\,902\,912\,625$
20 | $37.067\,983\,289\,023\,93$ | $37.757\,437\,070\,938\,22$ | $39.047\,619\,047\,619\,05$ | $34.640\,522\,875\,816\,99$ | $38.560\,885\,608\,856\,09$ | $34.951\,456\,310\,679\,61$
Window Big
title.bulgarian^2, passage.ngram, passage, passage.bulgarian^2
1 | $31.219\,141\,663\,501\,71$ | $28.375\,286\,041\,189\,93$ | $33.968\,253\,968\,253\,97$ | $29.411\,764\,705\,882\,35$ | $30.811\,808\,118\,081\,18$ | $33.252\,427\,184\,466\,02$
2 | $33.118\,116\,217\,242\,69$ | $31.578\,947\,368\,421\,05$ | $37.460\,317\,460\,317\,46$ | $31.209\,150\,326\,797\,385$ | $33.948\,339\,483\,394\,83$ | $29.854\,368\,932\,038\,835$
5 | $36.042\,537\,030\,003\,8$ | $35.697\,940\,503\,432\,49$ | $38.095\,238\,095\,238\,095$ | $33.823\,529\,411\,764\,7$ | $37.822\,878\,228\,782\,29$ | $34.223\,300\,970\,873\,79$
10 | $36.536\,270\,413\,976\,45$ | $37.299\,771\,167\,048\,06$ | $36.031\,746\,031\,746\,03$ | $33.986\,928\,104\,575\,16$ | $39.298\,892\,988\,929\,886$ | $36.650\,485\,436\,893\,206$
20 | $35.624\,762\,628\,180\,78$ | $34.553\,775\,743\,707\,09$ | $39.682\,539\,682\,539\,684$ | $31.045\,751\,633\,986\,928$ | $38.376\,383\,763\,837\,64$ | $33.737\,864\,077\,669\,904$
Paragraph
title.bulgarian^2, passage.ngram, passage, passage.bulgarian^2
1 | $41.815\,419\,673\,376\,375$ | $41.418\,764\,302\,059\,49$ | $42.063\,492\,063\,492\,06$ | $38.071\,895\,424\,836\,605$ | $40.959\,409\,594\,095\,94$ | $48.543\,689\,320\,388\,35$
2 | $42.233\,194\,075\,199\,39$ | $42.562\,929\,061\,784\,9$ | $43.174\,603\,174\,603\,18$ | $35.620\,915\,032\,679\,74$ | $42.988\,929\,889\,298\,895$ | $49.271\,844\,660\,194\,176$
5 | $41.587\,542\,726\,927\,46$ | $43.249\,427\,917\,620\,14$ | $40.317\,460\,317\,460\,316$ | $38.725\,490\,196\,078\,43$ | $40.036\,900\,369\,003\,69$ | $48.058\,252\,427\,184\,46$
10 | $39.460\,691\,226\,737\,56$ | $40.961\,098\,398\,169\,334$ | $38.412\,698\,412\,698\,41$ | $36.928\,104\,575\,163\,395$ | $39.852\,398\,523\,985\,24$ | $42.718\,446\,601\,941\,75$
20 | $37.523\,737\,181\,921\,76$ | $39.130\,434\,782\,608\,695$ | $37.619\,047\,619\,047\,62$ | $34.640\,522\,875\,816\,99$ | $38.560\,885\,608\,856\,09$ | $38.592\,233\,009\,708\,735$
Slavic BERT
1 | $33.194\,075\,199\,392\,33$ | $30.892\,448\,512\,585\,812$ | $33.174\,603\,174\,603\,18$ | $28.758\,169\,934\,640\,524$ | $32.287\,822\,878\,228\,78$ | $43.446\,601\,941\,747\,57$
2 | $33.270\,034\,181\,541\,966$ | $31.578\,947\,368\,421\,05$ | $31.904\,761\,904\,761\,905$ | $31.209\,150\,326\,797\,385$ | $35.239\,852\,398\,523\,986$ | $37.621\,359\,223\,300\,97$
5 | $31.143\,182\,681\,352\,07$ | $30.205\,949\,656\,750\,573$ | $30.158\,730\,158\,730\,158$ | $29.248\,366\,013\,071\,895$ | $30.996\,309\,963\,099\,63$ | $36.650\,485\,436\,893\,206$
10 | $30.421\,572\,350\,930\,496$ | $29.290\,617\,848\,970\,253$ | $29.682\,539\,682\,539\,684$ | $29.738\,562\,091\,503\,27$ | $31.918\,819\,188\,191\,883$ | $31.796\,116\,504\,854\,368$
20 | $29.661\,982\,529\,434\,106$ | $28.604\,118\,993\,135\,01$ | $29.365\,079\,365\,079\,364$ | $28.431\,372\,549\,019\,606$ | $32.103\,321\,033\,210\,335$ | $29.854\,368\,932\,038\,835$
Table 5: Evaluation results for the Bulgarian multiple-choice reading
comprehension task: comparison of various indexing and query strategies.

##  Appendix B Case Study

In Table 6, we present the retrieved evidence passages for the example
questions in Table 2: we omit the answers, and we only show the questions and
the contexts. Each example is separated by a double horizontal line, where the
first row is the question starting with “Q:”, and the following rows contain
passages returned by the retriever. For each context, we normalize the raw
scores from the comprehension model using Eq. 1 to obtain a probability
distribution. We then select an answer using $\operatorname*{arg\,max}$,
according to Eq. 2. In the table, we indicate the correctness of each
predicted answer using one of the following symbols before the question:

  * ✓

The question is answered correctly.

  * ✗

An incorrect answer has the highest score.

  * Two or more answers have the highest score.

We show the top retrieved result in order to illustrate the model scores over
different evidence passages and the quality of the articles. The queries are
formed by concatenating the question with an answer option, even though this
can lead to duplicate results since some answers can be quite similar or the
question’s terms could dominate the similarity score.

The questions in Table 6 are from five different categories: biology,
philosophy, geography, history, and online quizzes. Each of them has its own
specifics and gives us an opportunity to illustrate a different model
behavior.

The first question is from the biology domain, and we can see that the text is
very general, and so is the retrieved context. The latter talks about _hair_
rather than _coat_ , and the correct answer (D) _morphological adaptation_ is
not present in the retrieved text. On the other hand, all the terms are only
connected to it, and hence the model assigns high probability to this answer
option.

For the second question, from the philosophy domain, there are two related
contexts found. The first one is quite short, noisy, and it does not give much
information in general. The second paragraph manages to extract the definition
of _relativism_ and to give good supporting evidence for the correct answer,
namely that _there is no absolute good and evil_ (B). As a result, this option
is assigned high probability. Nevertheless, the incorrect answer _here is only
one moral law that is valid for all_ (A) is assigned an even higher
probability and it wins the voting.

In the third example, from the domain of geography, we see a large number of
possible contexts, due to the long and descriptive answers. We can make two
key observations: (i) the query is drawn in very different directions by the
answers, and (ii) there is no context for _Southwestern region_ , and thus, in
the second option, the result is for Russia, not for Bulgaria. The latter
passage pushes the probability mass to an option that talks about
transportation (D), which is incorrect. Fortunately, the forth context has an
almost full term overlap with the correct answer (B), and thus gets very high
probability assigned to it: 72%.

The fourth question, from the history domain, asks to point out a missing
concept, but the query is dominated by the question, and especially by
underscores, leading to a single hit, counting only symbols, without any
words. As expected, the model assigned uniform probability to all classes.

The last question, a history quiz, is a factoid one, and it lacks a reasoning
component, unlike the previous examples. The query returned a single direct
match. The retrieved passage contains the correct answer exactly: option
_Bogoridi_ (C). Thereby, the comprehension model assigns to it a very high
probability of 68%.

Context | $Pr_{A}$ | $Pr_{B}$ | $Pr_{C}$ | $Pr_{D}$
---|---|---|---|---
✓ Q: The thick coat of mammals in winter is an example of: |  |  |  |
1) The hair cover is a rare and rough bristle. In winter, soft and dense hair develops between them. Color ranges from dark brown to gray, individually and geographically diverse | $0.19$ | $0.19$ | $0.15$ | $0.47$
✗ Q: According to relativism in ethics: |  |  |  |
1) Moral relativism | $0.45$ | $0.24$ | $0.10$ | $0.21$
2) In ethics, relativism is opposed to absolutism. Whilst absolutism asserts the belief that there are universal ethical standards that are inflexible and absolute, relativism claims that ethical norms vary and differ from age to age and in different cultures and situations. It can also be called epistemological relativism - a denial of absolute standards of truth evaluation. | $0.28$ | $0.41$ | $0.09$ | $0.22$
✓ Q: Which of the assertions about the economic specialization of the Southwest region is true? |  |  |  |
1) Geographic and soil-climatic conditions are blessed for the development and cultivation of oil-bearing rose and other essential oil crops. | $0.12$ | $0.52$ | $0.28$ | $0.08$
2) Kirov has an airport of regional importance. Kirov is connected with rail transport with the cities of the Transsiberian highway (Moscow and Vladivostok). | $0.14$ | $0.27$ | $0.06$ | $0.53$
3) Dulovo has always been and remains the center of an agricultural area, famous for its grain production. The industrial sectors that still find their way into the city’s economy are primarily related to the primary processing of agricultural produce. There is also the seamless production that evolved into small businesses with relatively limited economic significance. | $0.25$ | $0.05$ | $0.67$ | $0.03$
4) In the glacial valleys and cirques and around the lakes in the highlands of Rila and Pirin, there are marshes and narrow-range glaciers (overlaps). | $0.10$ | $0.72$ | $0.08$ | $0.10$
Q: Point out the concept that is missed in the text of the Turnovo Constitution: … |  |  |  |
1)  | $0.26$ | $0.26$ | $0.26$ | $0.22$
✓ Q: Sofroniy Vrachanski sets up a genre that plays a big role in the history of the Bulgarian Revival. What is his name? |  |  |  |
1) Bogoridi is a Bulgarian Chorbadji genus from Kotel. Its founder is Bishop Sofronius Vrachanski (1739-1813). His descendants are: | $0.06$ | $0.16$ | $0.68$ | $0.10$
Table 6: Retrieved unique top-1 contexts for the example questions in Table 2.
The passages are retrieved using queries formed by concatenating a question
with an answer option.


# A Novel Procrustes Analysis Method to Quantify Multi-Joint Coordination of
the Upper Extremity after Stroke

Khadija F. Zaidi1 and Michelle Harris-Love2 *This work was supported by a
Dissertation Completion Grant awarded by George Mason University1Khadija F.
Zaidi is a Doctoral Candidate in the Computational Biomedical Engineering
concentration of the Department of Bioengineering Volgenau School of
Engineering, George Mason University, Fairfax, Virginia, USA
<EMAIL_ADDRESS>Harris-Love is with Anschutz Medical Campus,
University of Colorado, Aurora, Colorado, USA

###### Abstract

Upper extremity motor impairment affects about 80% of persons after strokes.
For stroke rehabilitation, upper limb kinematic assessments have increasingly
been used as primary or secondary outcome measures. Studying the upper
extremity provides a valuable tool for assessing limb coordination, mal-
adaptations, and recovery. There is currently no universal standardized scale
for categorizing multi-joint upper extremity movement. We propose a modified
Procrustes statistical shape method as a quantitative analysis that can
recognize segments of movement where multiple limb segments are coordinating
movement. Generalized Procrustes methods allow data points to be compared
across an array simultaneously rather than comparing them in pairs. Rather
than rely solely on discrete kinematic values to contrast movement, this
method allows evaluation of how movement progresses. The Procrustes analysis
of able-bodied movement showed that the hand and forearm segments moved in a
more coordinated manner during initiation. The shoulder and elbow become more
coordinated during movement completion. In impaired movement, this
coordination between the hand and forearm is disrupted. Potentially mal-
adaptive compensation occurs between the upper arm and forearm after movement
enters the deceleration phase. The utilization of Procrustes analysis may be a
step towards developing a comprehensive and universal quantitative tool that
does not require changes to existing treatments or increase patient burden.

Clinical relevance— This modified Procrustes Shape Analysis method can be
applied by clinicians to motion capture data from patients suffering upper
extremity movement deficits to objectively identify multi-joint coordination.

##  I INTRODUCTION

The human upper limb can move around seven degrees of freedom (DOF) and
demonstrates motor abundance, i.e. the ability to achieve a task in many
different ways. When the human upper extremity reaches towards a target at
arm’s length or beyond, the shoulder, elbow, and wrist joints are all involved
in coordinating the movement [1]. The neuromuscular control of reaching is
computationally complicated; however behavioral or kinematic analyses of the
upper extremity can be valuable tools for clinicians in evaluating movement in
individuals with neurological disorders (e.g. Stroke) [2, 3].

Upper extremity motor impairment affects about 80% of persons after strokes
and many activities of daily living. Therefore, being able to use the arm to
complete functional tasks is one of the top priorities for persons after
strokes, caregivers, and health care professionals. For stroke rehabilitation,
upper limb kinematic assessments have increasingly been used as primary or
secondary outcome measures. Clinicians may measure quality of movement through
goniometry, intertial movement units (IMU), ultrasound imaging, and a variety
of kinematic metrics. After a functional impairment, spasticity, restricted
range of motion, abnormal muscle synergies, and weakness can all affect
reaching [4, 5]. According to kinematic and kinetic descriptions, arm reaching
paths in the paretic arm become less smooth, and elbow and shoulder rotations
become less coordinated [6].

Studying the upper extremity also provides a valuable tool for assessing
recovery or retention of rehabilitative effects after the development of a
functional limitation. The ability to use the upper limb during functional
tasks may improve through compensatory strategies, in which individuals adapt
to complete a task in an entirely different way. The ability to distinguish
between mal-adaptive movements and compensation would help identify
interventions that can influence true recovery [7]. The current standards to
indicate coordinated movement include electromyography (EMG), clinical
observation, manual muscle testing, and screening tests. The tests most used
by clinicians score the upper extremity on the basis of achieving or not being
able to achieve certain tasks. These assessments do not yet provide enough
information about the strategies and mechanisms that underpin atypical
reaching, and are unable to distinguish between true recovery and
compensation.

Motion capture analysis has already been integrated into clinical practice as
a gold standard for kinematic analysis and is increasingly used clinically to
assess quality of movement following injury [8]. Current biomechanical
analysis methods include motion capture systems and accelerometers; however,
the use of such strategies is still not standard across clinical practice [9].
While there are many kinematic metrics that can assess quality of movement
such as velocity profiles, movement units, smoothness, path error, target
accuracy, there is currently no universal standardized scale for categorizing
multi-joint upper extremity movement, and current assessments provide little
information on how to tailor rehabilitation strategies to an individual
patient. Such models already exist for gait analysis and have been proven to
be beneficial.

In this paper we explore the Procrustes Analysis method and how it may be
applied to upper extremity kinematic analysis to indicate coordinated movement
between multiple limb segments. We aimed to:

  * •

Apply the Procrustes method as a quantitative analysis that can recognize
segments of movement where multiple limb segments are moving in a coordinated
fashion,

  * •

Demonstrate the potential to compare data from able-bodied adults and persons
who have developed functional limitations after stroke.

A quantitative tool that can be used across clinical practice without having
to alter current measurement modalities, convert large amounts of data, or
increase patient burden, presents a step towards developing a comprehensive
and universal upper extremity analysis.

##  II PROCRUSTES STATISTICAL SHAPE ANALYSIS

This study describes a novel application of Procrustes Shape Analysis to
multi-joint upper extremity kinematic data which can be used across any data-
set produced by clinical motion capture systems. Amusingly named after a Greek
myth of an innkeeper that would stretch or amputate the limbs of his hapless
victims, the Procrustes Method is a multivariate statistical technique of
comparing matrices of data points. Procrustes has been utilized since the
1970s in psycho-metrics and has become increasingly popular across
disciplines. Procrustes techniques are used in a variety of food research
areas, including sensory and consumer evaluations, by dividing outcomes into
matrices that can be compared to one another element by element [10].

Generalized Procrustes methods allow data points to be compared across an
array simultaneously rather than comparing them in pairs, which is
mathematically equivalent to fitting data to a group average. When applied to
upper extremity position data, Procrustes can be used to compare each position
across a reaching movement made by a persons after stroke to a neural intact
control. Current kinematic analyses compare movements by differentiating
between discrete metrics such as peak velocity, movement time, target error,
etc. The advantage of applying Procrustes analysis is that the overall
movement trajectory can be compared point-by-point. Procrustes also presents
the potential to monitor recovery by comparing trajectories of a patient over
time to assess if movements unfold similarly.

##  III METHODS

###  III-A Participants

Unimpaired adult participants $(n=2)$ and persons after stroke $(n=2)$ were
recruited to complete reach-to-target tasks while wearing clusters of motion
capture markers centered on the hand, forearm, and upper arm. All subjects
completed written consent paperwork as approved by the Institutional Review
Board of the National Rehabilitation Hospital in Washington, DC. All
identifying information was omitted from subject files. Subjects were excluded
in the instance of any other neurological disorders or if on medication to
prevent seizures. Inclusion criteria included the ability to complete a
reaching task and provide informed consent. Persons with stroke were
classified as exhibiting either mild or severe impairment based on an Upper
Extremity Fugel-Meyer (UEFM) score, where a score of 66 indicates all test
tasks can be performed by the subject. Subject information is shown in Tbl. I.
Able-bodied participants and persons with stroke were all right-dominant.

TABLE I: Demographics of Subjects with Stroke Sub | Months Since Stroke | UEFM | Paretic | Severity
---|---|---|---|---
1 | 31 | 31 | Right | Mild
2 | 51 | 11 | Right | Severe

Figure 1: Subject seated at reaching workspace with target placed medially at
80 % max reach capacity

A planar reach-to-target task was selected as a movement primitive that is a
sub-movement of many upper extremity functional movements. Each subject
completed two blocks of ten reaching movements using both paretic and non-
paretic arms for trajectory analysis. All individuals wore trunk restraints
and targets were placed well within arm’s reach to reduce trunk involvement
and shoulder elevation. Targets were placed at 80% maximal reach distance for
all twenty reaches to normalize with respect to the individual’s reach
capacity. The work space for reaching is depicted in Fig. 1. Subjects were
instructed to reach for the target at a comfortable pace in response to a ”go”
signal displayed in front of the subject. Curves for each arm were averaged to
produce a representative hand path trajectory. Curves were matched to control
subjects using the same arm for movements as each subject’s paretic arm. A
fourth order Butterworth filter was applied to the data collected at a sample
frequency of 300 Hz, in order to remain maximally sensitive to the desired
cutoff frequency of 50 Hz. Studying the standard deviations in orientation
angles over the course of movement allows a quantification of the interpatient
variability over the course of ten reaching trials per subject.

###  III-B Data Collection

In order to collect trajectory data for the three segments of the arm, an
Optotrak motion capture system was used along with small Infrared Emitting
Diode (IRED) marker clusters to represent the upper arm, forearm, and hand
[NDI Measurement Sciences] as seen in Fig. 2. Six degree of freedom Optotrak
cameras were mounted on the wall and the workspace in front of the subject was
digitized as the x-y plane. 7 mm diameter IRED markers were placed as three-
marker clusters. The current literature agrees on the placement of three
markers in each cluster in order to control for tissue stretch artifacts and
to accurately extrapolate the position of the arm. MATLAB was used to filter
reaching trajectory data collected from the above described marker clusters in
order to represent a clear displacement from the starting edge of the
workspace to the target and identify significant landmarks in the hand path
trajectory.

Figure 2: Clusters of three Infrared Emitting Diode markers centered at
subject’s hand, forearm, and upper arm segments

###  III-C Sliding Procrustes

To illustrate the output of the Procrustes method, consider two triangles in
two-dimensional space. The data-sets for each triangle consist of the three
points that construct each triangle. In a traditional Procrustes analysis, one
triangle is used as the reference configuration, while the second triangle is
scaled, translated, reflected, and/or rotated as necessary to achieve a
similar size and common center. When a data set is compared against a control
curve, a linear transformation matrix C, an orthogonal rotation and reflection
matrix T, a scaling factor b, and a dissimilarity index D are delivered that
would conform that data set to the control.

The able-bodied control trajectories were compared in part and in full to
trajectories with motor impairments in order to establish an index of
dissimilarity. In this study, the curves were not scaled, since capacity to
reach is specific to each subject. A novel modification was made to the
Procrustes method by analyzing subsets of the three-dimensional reaching path
data in increments of 35 consecutive data points, henceforth referred to as
Sliding Procrustes. By comparing reach paths in shorter sub-phases, movement
that is kinematically congruent between impaired and able-bodied movement can
be highlighted. The index of dissimilarity D, the sum of the squared
Procrustes distance between each corresponding element in both curves,
represents how dissimilar the two curves may be, and is normalized such that
it produces a value between 0 to 1, where 0 represents congruence between
curves and 1 represents complete dissimilarity.

In order to identify limb coherence, a dissimilarity index was produced
comparing whole curves of each limb segment, i.e. the hand, the forearm, and
the upper arm.

##  IV RESULTS

Figure 3: Procrustes comparison of dominant arm of able-bodied subjects
demonstrates the upper arm and forearm contributing to a coordinated movement
after peak velocity. The hand and forearm appear to coordinate during movement
initiation.

Fig. 3 shows the movement completed by the two able-bodied control subjects.
The Sliding Procrustes method showed that the hand and forearm segments moved
in a more coordinated manner during movement initiation. The shoulder and
elbow become more coordinated in the latter portion of movement before the
target is reached. Kinematically, these sections of movement are congruent,
indicating the limb segments move as one at these time points to achieve motor
goals.

Fig. 4 illustrates the output of the Sliding Procrustes analysis when applied
to the movement produced by persons with mild and severe impairment. The mild
impairment graphs indicate movement of the hand and forearm is similar during
movement completion, while the upper arm and forearm are similar earlier in
the movement. The severe impairment graphs in the bottom row show no
meaningful similarity in the hand and forearm movements. A potential
explanation for a lack of coordination may be due to the elbow joint
experiencing a diminished range of motion. Patients with more severe distal
impairments may incorporate more shoulder or trunk involvement in order to
achieve movements in the presence of muscle stiffness.

Figure 4: The paretic arm of the subject suffering severe impairment
demonstrates potentially mal-adaptive shoulder and elbow involvement early in
the movement.

Position data from the wrist, or the endpoint, was used to extrapolate peak
velocities (m/s), the time between movement initiation and peak velocity (ms),
and the time between peak velocity and movement completion (ms). These
kinematic measurements are shown in Tbl. II.

TABLE II: Peak Velocity and Movement Units for Subjects with Stroke Sub | PV (m/s) | Time to PV (ms) | Time after PV (ms)
---|---|---|---
1 | $0.42\pm 0.07$ | $450.48\pm 119.17$ | $1143.60\pm 544.97$
2 | $0.43\pm 0.03$ | $214.29\pm 96.80$ | $736.45\pm 161.19$

After the reach paths were compared through the Sliding Procrustes method, the
temporal locations of lowest dissimilarity were highlighted graphically. Fig.
3 shows a complete similarity of movement between the upper arm and forearm,
and the forearm and hand, towards the completion of movement and after the
peak velocity is achieved. As the target for all subjects was placed at 80 %
of maximum reach capacity, synergistic movement of the elbow and shoulder can
be present during precise movements or error correction, which are known to
occur in the latter part of movement.

The severe movement case showed the sub-movements that were most similar
between limb segments occurred for longer duration and earlier during the
movement. The mild movement case showed coordinated movement for shorter
durations and towards the completion of movement. The time points of discrete
kinematic metrics can be compared against the segments where Procrustes shows
trajectories with low dissimilarity. Peak velocity is achieved by the mild
impairment subject during the upper arm and forearm co-movement. In the severe
impairment case, the peak velocity is achieved before the upper arm and
forearm begin moving similarly. The dissimilarity indices were calculated
between the upper arm and forearm, the forearm and hand, and the upper arm and
hand. An index of 1 represents the highest comparative dissimilarity.
Dissimilarity ratios for the dominant limb of a control subject, as well as
ratios for the mild and severe impairment subjects are contained in Tbl. III.

TABLE III: Dissimilarity Indices of Whole Curve of Upper Arm (UA), Forearm (FA), and Hand (H) | UA/FA Index | FA/H Index | UA/H Index
---|---|---|---
Control 1 | 0.0013 | 0.0022 | 0.0057
Control 2 | 0.0012 | 0.0007 | 0.0023
Subject 1 | 0.0039 | 0.0174 | 0.0336
Subject 2 | 0.0073 | 0.1761 | 0.215

The two control subjects demonstrate the greatest dissimilarity in movement
occurs between the upper arm and hand during reach-to-target. In contrast, any
potential evidence of coordinated movement is between the upper arm and
forearm, or the forearm and the hand. The mild impairment subject shows an
increased dissimilarity in how limb segments modulate movement, with
impairment affecting coordination between the elbow and shoulder.

##  V DISCUSSION

Though kinetic measures of joint ranges of motion and coordination have been
identified as more effective descriptors of impairment than spasticity scores,
there is currently no single method of organizing functional movement into a
structured objective framework. The Sliding Procrustes method provides a
standardized analysis that can be applied to any clinically produced set of
endpoint position data. In the preliminary data, this method identifies the
time points at which multiple segments of the upper extremity display
congruous movement in relation to when peak velocity is achieved. The
dissimilarity ratios support the clinical findings of inter-joint coordination
becoming disrupted when movement is impaired. Using the cluster marker set to
identify the orientation and movement of the upper arm, forearm, and the hand,
the Sliding Procrustes approach can be applied to individual segments of the
upper extremity. When the Sliding Procrustes analysis is applied to individual
limb segments, it is seen in the control case that the hand and forearm assume
the same reaching progression earlier during the movement, while the upper arm
and forearm become similar later during the movement.

Rather than decompose movement into the impulse control, sub-corrections, and
limb target control phases, the Sliding Procrustes method was used to compare
every subject curve segment of 35 time-points against every segment of the
control subjects. The MATLAB script was custom-written to compare every
segment and sort by the output dissimilarity index, to find the segments
between control and impaired movement that were most congruous. In the mild
impairment movement curve, the initiation behavior of movement is extremely
similar to that of the control curve. This may imply spinal inputs being
responsible for the initiation of a strongly pre-learned behavior, and for
these inputs to remain preserved despite stroke pathology. Dissimilarity in
such a case would result from difficulty during online error regulation. This
supports the theory of initiation and pre-planning control mechanisms either
being preserved or possibly recovered post-stroke. In the severe impairment
movement curve, similarity does not occur at the initiation of movement. As
expected, the deceleration of movement, when the shoulder and elbow joints are
known to coordinate to complete the reaching task, is also not similar to
neurally intact movement. Due to severity of impairment, it was expected there
would be no significantly similar partial curve that would match the control
case. As shown by this preliminary data, the Sliding Procrustes approach is
limited in its ability to explain how a particular limb segment contributes to
the similarity of partial curves, whether through reduced range of motion or
mal-adaptive compensation.

This method may allow for further examination of which segments may be
modulated through a synergy; i.e. exhibit similar partial curves at the same
time increments. This approach represents an overlap between the trajectory
analysis and joint analysis, as it is neither a purely kinematic nor dynamic
metric. It is hoped with the inclusion of more subjects, it will be further
examined how and when the individual limb segments coordinate for optimized
movement to emerge. Future studies might couple the Sliding Procrustes output
with simultaneous electromyography (EMG) of the agonist and antagonist muscles
to validate whether the congruent sub-movements stem from synergistic and
coordinated movement.

## ACKNOWLEDGMENT

K.F. Zaidi thanks all that participated in this study. Thanks are due
specifically to Dr. Qi Wei, Dr. Quentin Sanders, Dr. Rachael Harrington, and
the MedStar National Rehabilitation Hospital research department.

## References

  * [1] Brown, J. & Gilleard, W. Transition from slow to ballistic movement: development of triphasic electromyogram patterns. European Journal Of Applied Physiology And Occupational Physiology. 63, 381-386 (1991,11)
  * [2] Schwarz, A., Bhagubai, M., Nies, S., Held, J., Veltink, P., Buurke, J. & Luft, A. Characterization of stroke-related upper limb motor impairments across various upper limb activities by use of kinematic core set measures. Journal Of NeuroEngineering And Rehabilitation. 19, 2 (2022,12)
  * [3] Öhberg, F., Bäcklund, T., Sundström, N. & Grip, H. Portable Sensors Add Reliable Kinematic Measures to the Assessment of Upper Extremity Function. Sensors. 19, 1241 (2019,3)
  * [4] Takatoku, N. & Fujiwara, M. Control of Arm Movements for Quick Change of Movement Direction. Journal Of Motor Behavior. 46, 25-32 (2014,1)
  * [5] Chiovetto, E., et al. “Tri-Dimensional and Triphasic Muscle Organization of Whole-Body Pointing Movements.” Neuroscience, vol. 170, no. 4, Nov. 2010, pp. 1223–38. DOI.org (Crossref)
  * [6] McCrea, P., Eng, J. & Hodgson, A. Biomechanics of reaching: clinical implications for individuals with acquired brain injury. Disability And Rehabilitation. 24, 534-541 (2002,1)
  * [7] Saes, M., Mohamed Refai, M., Beijnum, B., Bussmann, J., Jansma, E., Veltink, P., Buurke, J., Wegen, E., Meskers, C., Krakauer, J. & Kwakkel, G. Quantifying Quality of Reaching Movements Longitudinally Post-Stroke: A Systematic Review. Neurorehabilitation And Neural Repair. 36, 183-207 (2022,3)
  * [8] Montoya, D., Barria, P., Cifuentes, C., Aycardi, L., Morís, A., Aguilar, R., Azorín, J. & Múnera, M. Biomechanical Assessment of Post-Stroke Patients’ Upper Limb before and after Rehabilitation Therapy Based on FES and VR. Sensors. 22, 2693 (2022,3)
  * [9] Collins, K., Kennedy, N., Clark, A. & Pomeroy, V. Kinematic Components of the Reach-to-Target Movement After Stroke for Focused Rehabilitation Interventions: Systematic Review and Meta-Analysis. Frontiers In Neurology. 9 pp. 472 (2018,6)
  * [10] Gower, J. C., and Garmt B. Dijksterhuis. Procrustes Problems. Oxford University Press, 2004.