11institutetext: TH Köln, Germany
11email<EMAIL_ADDRESS>22institutetext: Universität Duisburg-
Essen, Germany
22email<EMAIL_ADDRESS>
# Validating Simulations of User Query Variants

Timo Breuer 11 Norbert Fuhr 22 Philipp Schaer 11

###### Abstract

System-oriented IR evaluations are limited to rather abstract understandings
of real user behavior. As a solution, simulating user interactions provides a
cost-efficient way to support system-oriented experiments with more realistic
directives when no interaction logs are available. While there are several
user models for simulated clicks or result list interactions, very few
attempts have been made towards query simulations, and it has not been
investigated if these can reproduce properties of real queries. In this work,
we validate simulated user query variants with the help of TREC test
collections in reference to real user queries that were made for the
corresponding topics. Besides, we introduce a simple yet effective method that
gives better reproductions of real queries than the established methods. Our
evaluation framework validates the simulations regarding the retrieval
performance, reproducibility of topic score distributions, shared task
utility, effort and effect, and query term similarity when compared with real
user query variants. While the retrieval effectiveness and statistical
properties of the topic score distributions as well as economic aspects are
close to that of real queries, it is still challenging to simulate exact term
matches and later query reformulations.

###### Keywords:

Query Simulation Dynamic Test Collection Reproducibility

##  1 Introduction

In accordance with the Cranfield paradigm, the underlying user model of
system-oriented IR evaluations is an abstract representation of real search
behavior. The simplified understanding of users is limited to a single query
and the examination of the result list in its entirety. Real search behavior
is more complex: searching is normally an iterative process with query
reformulations, and not every search result is examined but rather picked out
after judging its snippet text. To compensate for this shortcoming, it is
common practice to include (logged) user interactions in the evaluation
process. Industrial research is often supported by large datasets of user
interactions that, unfortunately, cannot be shared publicly, e.g., due to
privacy concerns [14]. Carterette et al. address the lack of user interaction
data available to academic research by introducing the concept of Dynamic Test
Collections [12]. Their framework expands test collections with simulated
interactions comprising the entire sequence of interactions, including the
simulation of queries, clicks, dwell times, and session abandonment.

Our work can be seen in the light of Dynamic Test Collections, but with a
special focus on simulating user query variants (UQVs). While previous work on
simulating interactions either focused on the completeness of interaction
sequences [12, 34, 47], click interactions [13], or stopping rules [34, 37],
work on simulating queries is underrepresented [20]. To the best of our
knowledge, the degree to which query simulators reproduce real user queries
has not yet been analyzed with TREC test collections. As opposed to previous
work in this regard, it is not our primary goal to generate the most effective
queries but rather to validate simulated queries, since the query formulation
is one of the first user interactions with the search system and as such it is
a critical component for any subsequent simulated interactions like clicks and
others. More specifically, our evaluations answer (RQ1) How do real user
queries relate to simulated queries made from topic texts and known-items in
terms of retrieval effectiveness? and (RQ2) To which degree do simulated
queries reproduce real queries provided that only resources of the test
collection are considered for the query simulation?

Our contributions are as follows. (1) We introduce an evaluation framework
that is used to analyze to which extent simulations reproduce real queries and
that reveals current limitations, (2) we compare and analyze conventional
query simulation methods that do not rely on large-scale interaction logs, (3)
we propose a new simulation method and hereby show that the parameterized
query reformulation behavior results in a better approximation of real queries
and resembles those of specific users, (4) we publish the code of the
experiments and provide the simulated queries for follow-up studies.

##  2 Related Work

Carterette et al. introduced Dynamic Test Collections [12] by enriching test
collections with simulated user interactions. Their outlined interaction
sequences included the simulation of queries, clicks, dwell times, and session
abandonment. Even though they implemented some specific simulators as part of
their experiments, they intended to provide a general framework that covers
all elements of user interactions which can also be implemented with various
methods. More recently, similar frameworks were introduced by Pääkkönen et al.
as Common Interaction Model [37] and Zhang et al. [47].

Most of the current methods for query simulations follow a two-stage approach
including the term candidate generation and the query modification strategy.
Usually, the term candidates are derived from a language model. Jordan et al.
introduced Controlled Query Generation (CQG) [28] that exploited the relative
entropy of a language model for query term generation. Azzopardi et al.
applied CQG when generating queries for known-item search [3, 4]. In a similar
vein, Berendsen et al. used annotations to group documents [9] and Huurnik et
al. simulated queries for purchased items [24]. When query term candidates are
available, there exist some commonly used query modification strategies [2, 5,
28, 29], which were also applied in follow-up studies [34, 35, 41] and
followed a principled way resulting in controlled query reformulation patterns
(cf. Section 3.1).

If large-scale user logs are available, different approaches propose, for
instance, learning to rewrite queries [22], model syntactic and semantic
changes between query reformulations [23], or replace old query terms with new
phrases with the help of the point-wise mutual information [27]. In contrast
to these examples, the query simulations analyzed in this study do not rely on
large-scale user logs but make use of test collections, i.e., topics and
relevance judgments.

As part of follow-up studies related to the TREC Session Track, Guan et al.
improved session search results by introducing the Query Change Model (QCM)
[19, 44] according to which the session search is modeled as a Markov Decision
Process that considers transitions between states, i.e., queries and other
interactions, to improve search results for query reformulations. Van Gysel et
al. found that QCM is especially effective for longer sessions while being on
par with term-frequency based approaches for shorter sessions [21]. Our query
simulation method draws inspiration from QCM, but generates queries instead of
improving retrieval results throughout a session.

Simulated UQVs contribute to more diverse and more realistic user-oriented
directives as part of system evaluations. Besides the actual simulation of
session search, applications for simulated queries are manifold. For instance,
UQVs enhance the pooling process [36], make rank fusion approaches possible
[8], are used for query performance prediction [18], or assist users with
query suggestions that improve the recall [42]. In this work, we compare
simulated to real UQVs.

##  3 Approach

In this section, we introduce the analyzed approaches for query simulations
(3.1) featuring conventional methods of term candidate generation and query
modification strategies and our new method. Furthermore, the evaluation
framework (3.2) and details about the datasets and implementations (3.3) are
described.

###  3.1 Query simulation

####  3.1.1 Term candidate generation

Simulating queries based on topics of test collections most likely complies
with exploitation search tasks [32], where users normally have a very concrete
understanding of their information needs. Provided that real users have read
the topic, they are very likely to include key terms of the topic texts when
formulating queries. As a simplified implementation, the TREC Topic Searcher
(TTS) considers only terms of the set
$T_{\mathrm{topic}}=\\{t_{1},...,t_{n}\\}$ composed of the topic’s title,
description and narrative with $t_{1},...,t_{n}$ being the term sequence in
the concatenated text. For upper bound performance estimates, we simulate a
Known-item Searcher (KIS). Here, we assume the simulated users to be familiar
with the document collection. When reading the topics, they recall key terms
of the relevant documents in the collection and use these as their query
terms. In this case, the term candidates
$T_{\mathrm{rel}}=\\{t_{1},...,t_{n}\\}$ are derived from a language model
based on CQG by Jordan et al. [28] according to
$P(t|D_{\mathrm{rel}})=(1-\lambda)P_{\mathrm{topic}}(t|D_{\mathrm{rel}})+\lambda
P_{\mathrm{background}}(t)$, where the topic model
$P_{\mathrm{topic}}(t|D_{\mathrm{rel}})$ is made from the relevant documents
$D_{\mathrm{rel}}$ for a given topic, while the background model
$P_{\mathrm{background}}(t)$ is derived from the vocabulary of the entire
corpus. $\lambda$ is used to model the influence of the background model, and
it is set to $0.4$ to be consistent with previous work [16, 28]. In this case,
$t_{1},...,t_{n}$ are ordered by the decreasing term probabilities of the
underlying language model.

####  3.1.2 Query modification strategy

We make use of the query generation techniques proposed by Baskaya et al. [5],
that were also used in previous simulation studies [28, 34, 35, 37, 41]. More
specifically, the following strategies are considered and used in combination
with the term candidates of $T_{\mathrm{topic}}$ and $T_{\mathrm{rel}}$: the
strategy S1 outputs single term queries $q_{i}$ following the ordering of term
candidates ($q_{1}=\\{t_{1}\\};q_{2}=\\{t_{2}\\};q_{3}=\\{t_{3}\\};...$); S2
keeps the first candidate term fixed and composes query strings by replacing
the second term for reformulations
($q_{1}=\\{t_{1},t_{2}\\};q_{2}=\\{t_{1},t_{3}\\};q_{3}=\\{t_{1},t_{4}\\};...$);
S$2^{\prime}$ is similar to S2, but keeps two candidate terms fixed
($q_{1}=\\{t_{1},t_{2},t_{3}\\};q_{2}=\\{t_{1},t_{2},t_{4}\\};q_{3}=\\{t_{1},t_{2},t_{5}\\};...$);
S3 starts with a single term query and incrementally adds query terms for
reformulations
($q_{1}=\\{t_{1}\\};q_{2}=\\{t_{1},t_{2}\\};q_{3}=\\{t_{1},t_{2},t_{3}\\};...$);
S$3^{\prime}$ is similar to S3, but starts with two candidate terms
($q_{1}=\\{t_{1},t_{2},t_{3}\\};q_{2}=\\{t_{1},t_{2},t_{3},t_{4}\\};q_{3}=\\{t_{1},t_{2},t_{3},t_{4},t_{5}\\};...$).
In total, we analyze ten different query simulators that result from the two
term candidate generators that are combined with five query modification
strategies, denoted as TTSS1-S$3^{\prime}$ and KISS1-S$3^{\prime}$,
respectively. We hypothesize, that the system performance of real queries
should range somewhere between those queries of the naive approach of TTS and
those queries of KIS.

####  3.1.3 Controlled Query Generation combined with Query Change Model

Compared to the previous query simulators, this approach adds an additional
scoring stage for the generated query string candidates. These candidates are
generated by considering every possible combination of n-grams made from a
term set. The corresponding terms are either taken from $T_{\mathrm{rel}}$ or
$T_{\mathrm{topic+rel}}=(T_{\mathrm{topic}}\cap
T_{\mathrm{rel}})\cup(T_{\mathrm{rel}}\setminus T_{\mathrm{topic}})_{k}$,
whereas $(T_{\mathrm{topic}}\cap T_{\mathrm{rel}})$ contains topic terms in
$T_{\mathrm{rel}}$ and $(T_{\mathrm{rel}}\setminus T_{\mathrm{topic}})_{k}$
denotes the top $k$ terms of $T_{\mathrm{rel}}$ that are not in the topic
text. In this regard, $k$ models the user’s vocabulary and domain knowledge.
Having a set of different query string candidates, we rank the queries by
$\frac{\sum_{j=1}^{|q|}\Theta_{j}}{|q|}$, which is the sum over all query
terms normalized by the query length $|q|$, whereas $\Theta_{j}$ is a term-
dependent score inspired by QCM [19, 44] and is implemented as follows.

$\Theta_{j}=\begin{cases}\alpha(1-P(t_{j}|D_{\mathrm{rel}})),&t_{j}\in
q_{\mathrm{title}}\\\ 1-\beta P(t_{j}|D_{\mathrm{rel}}),&t_{j}\in+\Delta
q\land t_{j}\in T_{\mathrm{topic}}\\\ \epsilon\
\mathrm{idf}(t_{j}),&t_{j}\in+\Delta q\land t_{j}\notin T_{\mathrm{topic}}\\\
-\delta P(t_{j}|D_{\mathrm{rel}}),&t_{j}\in-\Delta q\\\ \end{cases}$ (1)

whereas $q_{\mathrm{title}}$ is the set of topic title terms and $+/-\Delta q$
denotes added or removed terms of a query reformulation that is made in
reference to the previously simulated query, except for the first query
formulation $q_{1}$ for which the topic title is used as a reference. In our
experiments, we analyze 3-,4-,5-gram term candidates and analyze three
different parametrizations of the simulators, which are defined as follows.
First, we analyze the strategy S4
($\alpha=2.2,\beta=0.2,\epsilon=0.05,\delta=0.6$), which tends to prefer topic
terms and mostly keeps terms of previous queries. Second, we analyze the
strategy S$4^{\prime}$ ($\alpha=2.2,\beta=0.2,\epsilon=0.25,\delta=0.1$),
which mostly keeps terms of previous queries, but tends to include terms that
are not in the topic text. Finally, we analyze the strategy
S$4^{\prime\prime}$ ($\alpha=0.2,\beta=0.2,\epsilon=0.025,\delta=0.5$), which
tends to stick to the topic terms, but does not necessarily keep terms of
previous query formulations. In sum, we analyze six different instantiations
of these simulators, which are either based on $T_{\mathrm{rel}}$ (denoted as
KISS4-S$4^{\prime\prime}$), or based on $T_{\mathrm{topic+rel}}$ with $k=4$
(denoted as TTSS4-S$4^{\prime\prime}$).

###  3.2 Evaluation framework

In the following, we outline our evaluation framework used to validate the
simulations in reference to real queries in different aspects. It includes
evaluation of the average retrieval performance, shared task utility, effort
and effect, and query term similarity between simulated and real queries.

####  3.2.1 Retrieval performance

As shown by Tague and Nelson, simulated queries fall behind real queries in
terms of retrieval performance [40]. For this reason, we evaluate the Average
Retrieval Performance (ARP) as it is common practice in system-oriented IR
experiments. The ARP is determined by the average of a measure over all topics
in a test collection. Beyond comparing the averaged means of different
queries, we propose a more in-depth analysis of the topic score distributions.
Recently, the Root Mean Square Error (RMSE) and paired t-tests were introduced
as reproducibility measures [10]. The RMSE measures the closeness between the
topic score distributions, and low errors indicate a good reproduction. When
using t-tests as a reproducibility measure, low p-values result from diverging
score distributions and indicate a higher probability of failing the
reproduction.

####  3.2.2 Shared task utility

According to Huurnik et al. [24], the ARP of the simulated queries alone is
not an appropriate indicator of how well the simulations resemble the real
queries since useful query simulators should identify the best system. As
proposed by Huurnik et al., we analyze how the simulated queries reproduce
relative system orderings by comparing them with the help of Kendall’s $\tau$
as it is common practice as part of shared task evaluations [43]. We compare
the simulated and real queries by determining how well the ordering of systems
with different parametrizations (and different retrieval performance) can be
reproduced by simulated queries.

####  3.2.3 Effort and effect

In order to account for a more user-oriented evaluation, we simulate sessions
and evaluate them with regards to the effort (number of queries) that has to
be made and the resulting effects (cumulated gain). First, we simulate
sessions using ten simulated queries and an increasing number of documents per
query and evaluate the results by the sDCG measure [26], whereas the cumulated
gain is discounted for each result and query. Second, we evaluate the
simulation quality from another more economical point of view. Azzopardi
applies economic theory to the retrieval process [2] and demonstrates that for
a pre-defined level of cumulated gain, query reformulations can be compensated
by browsing depth (or vice versa browsing depth by more query reformulations).
Furthermore, he illustrates this relationship with isoquants - a visualization
technique used in microeconomics. Thus, we evaluate the closeness between
isoquants of simulated and real queries by the Mean Squared Logarithmic Error
(MSLE).

####  3.2.4 Query term similarity

It is not the primary goal of this study to simulate query strings with exact
term matches. Instead, simulated UQVs should result in diverse query strings
for a fixed information need (topic). Nonetheless, it is worth analyzing the
term overlap between the simulated and real queries. As Liu et al. [31] or
Mackenzie and Moffat [33] propose, we determine the Jaccard similarity between
the sets of unique terms made from the query reformulations. When compared
with the other evaluations, the term similarities add more insights about the
simulated UQVs. For instance, if it is possible to simulate query
reformulations that adequately relate to the properties of real queries, but
with other terms.

###  3.3 Datasets and implementation details

In our experimental setup, we use the user query variant (UQV) dataset
provided by Benham and Culpepper
[6]111https://culpepper.io/publications/robust-uqv.txt.gz. Given the topic
texts, eight users formulated up to ten query variants for each topic. Each
user formulated at least one query for each topic, and the fifth user (denoted
as UQV5) formulated ten queries for each topic. More details about the query
collection process are provided by Benham et al. [7]. Accordingly, we evaluate
the system runs with The New York Times Annotated Corpus and the topics of
TREC Common Core 2017 [1]. As part of our experiments, we exploit the
interactive search possibilities of the Pyserini toolkit [30]. We index the
Core17 test collection with the help of Anserini [45] and the default indexing
options as provided in the regression
guide222https://github.com/castorini/anserini/blob/master/docs/regressions-
core17.md. Unless stated otherwise, all results were retrieved with the BM25
method and Anserini’s default parameters ($b=0.4$, $k=0.9$). We evaluate the
results with the repro_eval toolkit [11] that is a dedicated reproducibility
framework featuring bindings to trec_eval measures. The source code of the
experiments and the simulated queries are available in a public GitHub
repository333github https://github.com/irgroup/ecir2022-uqv-sim.

##  4 Experimental results

####  4.0.1 Retrieval performance

Regarding RQ1, we validate the retrieval performance of real (UQV) and
simulated (TTS/KIS) queries. Table 0.A.1 shows the ARP including nDCG and AP
scores that are determined by averaging results with 1000 documents per topic
and P@10 scores over all queries, the first444S1 and S3, as well as S2 and
S$3^{\prime}$, do not differ when averaging over the first queries., or the
best query of a topic. Our assumptions are confirmed. The retrieval
performance of real queries ranges between that of the TTSS1-S$3^{\prime}$ and
KISS1-S$3^{\prime}$ simulators. Especially, the performance of the
TTSS1-S$3^{\prime}$ queries stays below that of real queries. For instance,
the average nDCG scores of the UQV queries range between $0.3787$ and
$0.4980$, whereas the maximum score of the TTSS1-S$3^{\prime}$ queries is
$0.3499$ and the nDCG scores of KISS$2^{\prime}$-S$3^{\prime}$ lie above those
of UQV. Similarly, the nDCG scores averaged over the first UQV queries reach
$0.3979$ at a minimum, whereas the maximum score of the TTSS1-S$3^{\prime}$
queries is $0.3895$. When averaging over the best queries, most nDCG scores of
TTS fall into the range of real queries, but there is also a higher
probability of finding a good performing query since more TTS than UQV queries
are available. Except for single term queries (S1), all KIS scores outperform
the UQV queries when averaging over the best queries. With regard to the
simulated queries based on the TTSS4-S$4^{\prime\prime}$ approach, most of the
nDCG, P@10, and AP scores fall into the range of the real queries, while
KISS4-S$4^{\prime\prime}$ queries outperform UQV queries. Thus, we have a
specific focus on TTSS4-S$4^{\prime\prime}$.

Figure 0.A.2 shows the RMSEnDCG between queries with conventional query
modification strategies (TTSS1-S$3^{\prime}$/KISS1-S$3^{\prime}$) and the real
queries (UQV). Especially for the TTS queries, the strategy S$2^{\prime}$ has
the lowest RMSE scores and acceptable scores for the KIS queries. In the
following experiments, we primarily use the strategy S$2^{\prime}$ for both
the TTS and KIS queries since their term length complies with the typical
length of real queries [25] and they serve as estimates of lower and upper
bound retrieval performance. Additionally, we evaluate the
TTSS4-S$4^{\prime\prime}$ queries with the help of the RMSE and simulations in
reference to the ten queries per topic of UQV5. For each query reformulation,
100 documents are retrieved and contribute to the final ranking list of a
topic if a previous query has not retrieved them. Figure 1 shows the RMSE
instantiated with P@1000, nDCG, and AP along with an increasing number of
documents retrieved with ten queries. For all measures, the error increases
when more documents per query are retrieved. With regard to P@1000 and nDCG,
the TTSS$2^{\prime}$ and KISS$2^{\prime}$ queries have the largest error,
while KISS$2^{\prime}$ has a lower RMSEAP than TTSS$4^{\prime}$. For all
measures, the TTSS4-S$4^{\prime\prime}$ queries have the lowest error, which
means they are the best approximation of UQV5 among all analyzed query
simulations.

Figure 1: RMSE instantiated with P@1000, nDCG, and AP

Finally, we compare the topic score distributions of the simulated queries and
all UQV queries by paired t-tests555Applying the Bonferroni correction adjusts
the alpha level to $\alpha=\frac{0.05}{64}\approx 0.0008$ (considering eight
users and eight query simulators for an alpha level of 0.05).. Since some
users formulated no more than one query per topic, we limit our evaluations to
the first query of each simulator. It means that each of the p-values shown in
Figure 0.A.2 is determined by t-tests with nDCG score distributions that
result from 50 UQV and 50 simulated queries. The TTSS$2^{\prime}$ queries have
the highest p-values when compared with UQV{2,3,8}. These results align with
the ARP scores reported in Table 0.A.1. The nDCG scores of UQV2 (0.4096), UQV3
(0.3979), and UQV8 (0.4046) are the most similar to the nDCG score of TTSS3
(0.3895) in comparison to other simulators. In contrast, the p-values of
KISS$2^{\prime}$ queries are low for all UQV queries, which complies with the
ARP scores in Table 0.A.1. The KISS$2^{\prime}$ scores averaged over the first
queries are substantially higher compared to the UQV scores (e.g.,
nDCG(KISS$2^{\prime}$)=0.5474 compared to the best UQV query with
nDCG(UQV7)=0.4980). The UQV{1,4,5,6,8} queries have comparably higher p-values
with the TTS{S4,S$4^{\prime\prime}$} queries which align with similar ARP
scores. Interestingly, the t-test with UQV7 and TTSS$4^{\prime}$ results in
the highest overall p-value of 0.9901 and similarly high p-values with
KISS4-S$4^{\prime\prime}$. This lets us assume that the corresponding user of
the UQV7 queries diverged from the terms in the topic texts and had some prior
knowledge about adequate queries for at least some of the topics. In sum, not
only the ARP can be reproduced with the simulated TTSS4-S$4^{\prime\prime}$
and KISS4-S$4^{\prime\prime}$ queries, but also statistical properties of the
topic score distributions.

####  4.0.2 Shared task utility

Regarding RQ2, we validate to which degree the simulated queries reproduce
properties of the real queries in several regards. First, we evaluate if the
simulated queries can preserve the relative system orderings. To be consistent
with Huurnik et al., we evaluate five systems and different parametrizations
($\mu=50,250,500,1250,2500,5000$) of the query likelihood model with Dirichlet
smoothing (QLD) [46], but other retrieval methods and variations thereof can
be reasonable as well. For each query formulation $q_{i}$, we determine the
correlation by Kendall’s $\tau$ averaged over all topics (cf. Figure 2 (left))
in comparison to the UQV5 queries. The TTSS$2^{\prime}$ queries do not
preserve the relative system ordering. Especially for the first five query
reformulations, there is a low correlation with the relative system orderings
of the real queries. Interestingly, the KISS$2^{\prime}$ queries result in
acceptable Kendall’s $\tau$ scores [43], while the scores beyond the sixth
query formulation show low correlations. Similarly, the
TTSS4-S$4^{\prime\prime}$ queries correlate with the system orderings of UQV5
queries fairly well, even reaching the maximum score of 1.0. Beyond the sixth
query reformulation, the correlation falls off. While it is out of this
study’s scope to reach any deﬁnitive conclusions, we assume that this is
related to query drifts - an issue that is also known from term expansions as
part of pseudo-relevance feedback [15, 38].

Figure 2: Kendall’s $\tau$ between system orderings of query reformulations in
reference to UQV5 (left). Jaccard similarity between unique terms of the
queries (right).

####  4.0.3 Effort and effect

Since most of the experiments validated single queries only, we simulate
search sessions and evaluate these by sDCG (instantiated with b=2, bq=4). We
compare sessions with 3, 5, or 10 queries and an increasing number of
documents per query. Figure 3 (top) compares the queries of UQV5 (made by a
single user [7]) to ten simulated queries of TTSS$2^{\prime}$,
KISS$2^{\prime}$, and TTSS4-S$4^{\prime\prime}$. As expected, the cumulative
gain increases faster when more queries per session are used. Likewise, the
TTSS$2^{\prime}$ and KISS$2^{\prime}$ queries deliver lower and upper bound
limits, respectively. In between, there are the cumulative gains by the UQV5
and TTSS4-S$4^{\prime\prime}$ queries. These results show that it is possible
to fine-tune and to reproduce the cumulative gain close to that of real
queries, in this particular case with TTSS$4^{\prime\prime}$.

Figure 3 (bottom) shows the isoquants and illustrates how many documents have
to be examined by a simulated user to reach pre-defined levels of nDCG (0.3,
0.4, 0.5). More queries compensate browsing depth, and as expected, the least
documents have to be examined with KISS$2^{\prime}$ queries and the most with
TTSS$2^{\prime}$ queries. The TTSS$2^{\prime}$ isoquants lie above the others,
which can be explained by the poorer retrieval performance as already shown in
Table 0.A.1. As shown by the MSLE, the TTSS4 isoquant has the lowest error for
all values of nDCG. Again, we see a better approximation of the UQV5 isoquant
with the TTSS4-S$4^{\prime\prime}$ strategies and that it is possible to
reproduce economic properties through parameterizing the query reformulation
behavior.

MSLE
---
nDCG | 0.3 | 0.4 | 0.5
TTSS$2^{\prime}$ | 0.3371 | 0.4279 | 0.6568
KISS$2^{\prime}$ | 0.0949 | 0.1837 | 0.3987
TTSS4 | 0.0059 | 0.0323 | 0.0444
TTSS$4^{\prime}$ | 0.0509 | 0.0758 | 0.1550
TTSS$4^{\prime\prime}$ | 0.0713 | 0.0807 | 0.0791

Figure 3: Simulations with 3, 5, or 10 queries per session evaluated by sDCG
(top). Isoquants and MSLE between simulations and UQV5 with fixed nDCG
(bottom).

####  4.0.4 Query term similarities

Figure 2 (right) shows the Jaccard similarities between the concatenated query
strings. More specifically, only normalized unique terms are compared, and
depending on the number of available queries for a specific topic, we include
an equal number of simulated queries to avoid low Jaccard similarities when
less than ten UQV queries are available. As the results show, the highest
similarities are between the simulated queries. While the similarities between
conventional strategies S1 to S$3^{\prime}$ and the strategies S4 to
S$4^{\prime\prime}$ are rather low for the TTS queries, there are higher
similarities for the KIS queries. Compared to UQV and TTS queries, the KIS
queries have the lowest similarities, which indicates that descriptive terms
of relevant documents are very different from those used in real queries and
the topic texts. Interestingly, the UQV{2,3,8} queries do not have a
remarkably high Jaccard similarity with TTSS$2^{\prime}$ queries, despite the
high p-values that are shown in Figure 0.A.2. This shows that it is possible
to simulate UQVs with different query terms than in the real queries, but with
comparable statistical properties as indicated by the p-values even with the
rather naive approach of TTSS$2^{\prime}$. There are slightly higher
similarities between KIS queries and the TTSS4-S$4^{\prime\prime}$ queries. In
particular, there is a higher similarity between TTSS$4^{\prime}$ and the KIS
queries since the simulator is parameterized to diverge from the topic terms.
Overall, we conclude that the analyzed simulation methods do not result in
query strings that exactly match the terms of real queries in this specific
UQV dataset [6].

##  5 Discussion

Referring to our research questions posed earlier, we answer them as follows.

####  5.0.1 RQ1 How do real user queries relate to simulated queries made from
topic texts and known-items in terms of retrieval effectiveness?

It is possible to use the TTSS1-S$3^{\prime}$ and KISS1-S$3^{\prime}$ queries,
which follow conventional simulation methods, as lower and upper bound
estimates between which the retrieval performance of real user query variants
(UVQ1-8) ranges. Simulations based on our new method (TTSS$4^{\prime\prime}$)
provide better approximations of real query effectiveness, and the
parametrization allows the simulation of different query formulation behaviors
and a retrieval performance better resembling real queries.

####  5.0.2 RQ2 To which degree do simulated queries reproduce real queries
provided that only resources of the test collection are considered for the
query simulation?

Our experiments show that the simulated TTSS4-S$4^{\prime\prime}$ queries
reproduce the real UQV queries reasonably well in several regards. Beyond a
similar ARP, they also reproduce statistical properties of the topic score
distributions as shown by the RMSE and p-values. Furthermore, it is shown that
the simulated queries also reproduce economic aspects of the real queries as
evaluated with the sDCG experiments and the isoquants that compare tradeoffs
between the number of query reformulations and the browsing depth for a fixed
level of gain. Furthermore, when evaluating the shared task utility, the
queries of our new parameterized simulation approach preserve the relative
system orderings up to the fifth reformulation, while the correlations fall
off for later reformulations. We assume that this is related to topic drifts,
and further analysis in this direction is required. Finally, even though it is
not the primary goal to simulate exact term matches with UQVs, the analysis of
the query term similarity showed that there is only a slight overlap between
terms of simulated and real queries, and a more dedicated approach is required
to reproduce exact term matches.

##  6 Conclusion

In this work, we present an evaluation framework and a new method for
simulated user query variants. Our experiments showed that the retrieval
performance of real queries ranges between that of simulated queries from
conventional methods based on topic texts and known-items. As a better
approximation of user queries, we introduce a simulation method that allows
parameterizing the query reformulation behavior and thus better reproduces
real queries from specific users. One limitation of our simulations is the
exclusion of relevance feedback from previous search results. Users normally
include terms of documents or snippets they consider as relevant [17, 39] in
their query reformulations. Likewise, the experiments neglect click
simulations. We leave it for future work to complement and analyze simulations
in this regard.

Table 0.A.1: Average retrieval performance over $q$ queries | All queries | First queries | Best queries
---|---|---|---
| $q$ | nDCG | P@10 | AP | $q$ | nDCG | P@10 | AP | $q$ | nDCG | P@10 | AP
UQV1 | 150 |  . 3787  |  . 4507  |  . 1581  | 50 |  . 4293  |  . 5040  |  . 2003  | 50 |  . 4969  |  . 6320  |  . 2429
UQV2 | 52 |  . 4221  |  . 5058  |  . 2020  | 50 |  . 4096  |  . 4880  |  . 1894  | 50 |  . 4103  |  . 4900  |  . 1896
UQV3 | 68 |  . 3922  |  . 4353  |  . 1780  | 50 |  . 3979  |  . 4560  |  . 1813  | 50 |  . 4117  |  . 4800  |  . 1878
UQV4 | 123 |  . 4126  |  . 4894  |  . 1888  | 50 |  . 4469  |  . 5220  |  . 2099  | 50 |  . 5146  |  . 6300  |  . 2644
UQV5 | 500 |  . 3922  |  . 4330  |  . 1649  | 50 |  . 4447  |  . 4920  |  . 2043  | 50 |  . 5353  |  . 7240  |  . 2807
UQV6 | 136 |  . 4030  |  . 4713  |  . 1843  | 50 |  . 4488  |  . 5080  |  . 2197  | 50 |  . 4980  |  . 5980  |  . 2515
UQV7 | 50 |  . 4980  |  . 5720  |  . 2418  | 50 |  . 4980  |  . 5720  |  . 2418  | 50 |  . 4980  |  . 5720  |  . 2418
UQV8 | 156 |  . 3814  |  . 4545  |  . 1645  | 50 |  . 4046  |  . 4500  |  . 1799  | 50 |  . 4556  |  . 5620  |  . 2193
TTSS1 | 500 |  . 0479  |  . 0306  |  . 0127  | 50 |  . 1705  |  . 1280  |  . 0541  | 50 |  . 3066  |  . 2360  |  . 0971
TTSS2 | 500 |  . 1964  |  . 1716  |  . 0688  | 50 |  . 3592  |  . 3900  |  . 1604  | 50 |  . 4391  |  . 5100  |  . 2097
TTSS$2^{\prime}$ | 500 |  . 3387  |  . 3426  |  . 1413  | 50 |  . 3895  |  . 4020  |  . 1821  | 50 |  . 4639  |  . 5940  |  . 2283
TTSS3 | 500 |  . 3323  |  . 3632  |  . 1388  | 50 |  . 1705  |  . 1280  |  . 0541  | 50 |  . 4776  |  . 6080  |  . 2383
TTSS$3^{\prime}$ | 500 |  . 3499  |  . 3874  |  . 1474  | 50 |  . 3592  |  . 3900  |  . 1604  | 50 |  . 4709  |  . 6060  |  . 2311
TTSS4 | 500 |  . 4493  |  . 5168  |  . 2088  | 50 |  . 4409  |  . 4920  |  . 2072  | 50 |  . 5945  |  . 7620  |  . 3282
TTSS$4^{\prime}$ | 500 |  . 4788  |  . 5626  |  . 2288  | 50 |  . 4976  |  . 5940  |  . 2429  | 50 |  . 6207  |  . 8040  |  . 3554
TTSS$4^{\prime\prime}$ | 500 |  . 3780  |  . 4224  |  . 1644  | 50 |  . 4393  |  . 4860  |  . 2065  | 50 |  . 5812  |  . 7680  |  . 3222
KISS1 | 500 |  . 1334  |  . 1044  |  . 0314  | 50 |  . 2836  |  . 2040  |  . 0813  | 50 |  . 4087  |  . 4400  |  . 1492
KISS2 | 500 |  . 3969  |  . 3972  |  . 1615  | 50 |  . 5096  |  . 5400  |  . 2535  | 50 |  . 5988  |  . 7460  |  . 3429
KISS$2^{\prime}$ | 500 |  . 5114  |  . 5666  |  . 2507  | 50 |  . 5474  |  . 6220  |  . 2870  | 50 |  . 6336  |  . 7980  |  . 3762
KISS3 | 500 |  . 5598  |  . 6336  |  . 3009  | 50 |  . 2836  |  . 2040  |  . 0813  | 50 |  . 6907  |  . 8620  |  . 4299
KISS$3^{\prime}$ | 500 |  . 5941  |  . 6882  |  . 3285  | 50 |  . 5096  |  . 5400  |  . 2535  | 50 |  . 6922  |  . 8620  |  . 4337
KISS4 | 500 |  . 5216  |  . 5976  |  . 2604  | 50 |  . 5146  |  . 5960  |  . 2630  | 50 |  . 6461  |  . 8200  |  . 3902
KISS$4^{\prime}$ | 500 |  . 5008  |  . 5888  |  . 2416  | 50 |  . 5033  |  . 5980  |  . 2400  | 50 |  . 6269  |  . 8080  |  . 3703
KISS$4^{\prime\prime}$ | 500 |  . 4859  |  . 5584  |  . 2293  | 50 |  . 5191  |  . 6020  |  . 2644  | 50 |  . 6401  |  . 8360  |  . 3781

Figure 0.A.1: RMSE between TTSS1-S$3^{\prime}$ and KISS1-S$3^{\prime}$ queries
and the UQV queries.

Figure 0.A.2: p-values of paired t-tests between UQV and simulated queries.

## References

  * [1] Allan, J., Harman, D., Kanoulas, E., Li, D., Gysel, C.V., Voorhees, E.M.: TREC 2017 common core track overview. In: Proc. TREC (2017)
  * [2] Azzopardi, L.: The economics in interactive information retrieval. In: Proc. SIGIR. pp. 15–24 (2011)
  * [3] Azzopardi, L., de Rijke, M.: Automatic construction of known-item finding test beds. In: Efthimiadis, E.N., Dumais, S.T., Hawking, D., Järvelin, K. (eds.) Proc. SIGIR. pp. 603–604 (2006)
  * [4] Azzopardi, L., de Rijke, M., Balog, K.: Building simulated queries for known-item topics: an analysis using six european languages. In: Proc. SIGIR. pp. 455–462 (2007)
  * [5] Baskaya, F., Keskustalo, H., Järvelin, K.: Time drives interaction: simulating sessions in diverse searching environments. In: Proc. SIGIR. pp. 105–114 (2012)
  * [6] Benham, R., Culpepper, J.S.: Risk-reward trade-offs in rank fusion. In: Proc. ADCS. pp. 1:1–1:8 (2017)
  * [7] Benham, R., Gallagher, L., Mackenzie, J.M., Damessie, T.T., Chen, R., Scholer, F., Moffat, A., Culpepper, J.S.: RMIT at the 2017 TREC CORE track. In: Proc. TREC (2017)
  * [8] Benham, R., Mackenzie, J.M., Moffat, A., Culpepper, J.S.: Boosting search performance using query variations. ACM Trans. Inf. Syst. 37(4), 41:1–41:25 (2019)
  * [9] Berendsen, R., Tsagkias, M., de Rijke, M., Meij, E.: Generating pseudo test collections for learning to rank scientific articles. In: Proc. CLEF. pp. 42–53 (2012)
  * [10] Breuer, T., Ferro, N., Fuhr, N., Maistro, M., Sakai, T., Schaer, P., Soboroff, I.: How to measure the reproducibility of system-oriented IR experiments. In: Proc. SIGIR. pp. 349–358 (2020)
  * [11] Breuer, T., Ferro, N., Maistro, M., Schaer, P.: repro_eval: A python interface to reproducibility measures of system-oriented IR experiments. In: Proc. ECIR. pp. 481–486 (2021)
  * [12] Carterette, B., Bah, A., Zengin, M.: Dynamic test collections for retrieval evaluation. In: Proc. ICTIR. pp. 91–100. ACM (2015)
  * [13] Chuklin, A., Markov, I., de Rijke, M.: Click Models for Web Search. Synthesis Lectures on Information Concepts, Retrieval, and Services, Morgan & Claypool Publishers (2015)
  * [14] Craswell, N., Campos, D., Mitra, B., Yilmaz, E., Billerbeck, B.: ORCAS: 20 million clicked query-document pairs for analyzing search. In: Proc. CIKM. pp. 2983–2989 (2020)
  * [15] Croft, W.B., Harper, D.J.: Using probabilistic models of document retrieval without relevance information. J. Documentation 35(4), 285–295 (1979)
  * [16] Cronen-Townsend, S., Zhou, Y., Croft, W.B.: Predicting query performance. In: Proc. SIGIR. pp. 299–306 (2002)
  * [17] Eickhoff, C., Teevan, J., White, R., Dumais, S.T.: Lessons from the journey: a query log analysis of within-session learning. In: Proc. WSDM. pp. 223–232 (2014)
  * [18] Faggioli, G., Zendel, O., Culpepper, J.S., Ferro, N., Scholer, F.: An enhanced evaluation framework for query performance prediction. In: Proc. ECIR. pp. 115–129 (2021)
  * [19] Guan, D., Zhang, S., Yang, H.: Utilizing query change for session search. In: Proc. SIGIR. pp. 453–462 (2013)
  * [20] Günther, S., Hagen, M.: Assessing query suggestions for search session simulation. Proc. Sim4IR (2021), http://ceur-ws.org/Vol-2911/paper6.pdf
  * [21] Gysel, C.V., Kanoulas, E., de Rijke, M.: Lexical query modeling in session search. In: Proc. ICTIR. pp. 69–72 (2016)
  * [22] He, Y., Tang, J., Ouyang, H., Kang, C., Yin, D., Chang, Y.: Learning to rewrite queries. In: Proc. CIKM. pp. 1443–1452 (2016)
  * [23] Herdagdelen, A., Ciaramita, M., Mahler, D., Holmqvist, M., Hall, K.B., Riezler, S., Alfonseca, E.: Generalized syntactic and semantic models of query reformulation. In: Proc. SIGIR. pp. 283–290 (2010)
  * [24] Huurnink, B., Hofmann, K., de Rijke, M., Bron, M.: Validating query simulators: An experiment using commercial searches and purchases. In: Proc. CLEF. pp. 40–51 (2010)
  * [25] Jansen, B.J., Booth, D.L., Spink, A.: Patterns of query reformulation during web searching. J. Assoc. Inf. Sci. Technol. 60(7), 1358–1371 (2009)
  * [26] Järvelin, K., Price, S.L., Delcambre, L.M.L., Nielsen, M.L.: Discounted cumulated gain based evaluation of multiple-query IR sessions. In: Proc. ECIR. pp. 4–15 (2008)
  * [27] Jones, R., Rey, B., Madani, O., Greiner, W.: Generating query substitutions. In: Proc. WWW. pp. 387–396 (2006)
  * [28] Jordan, C., Watters, C.R., Gao, Q.: Using controlled query generation to evaluate blind relevance feedback algorithms. In: Proc. JCDL. pp. 286–295 (2006)
  * [29] Keskustalo, H., Järvelin, K., Pirkola, A., Sharma, T., Lykke, M.: Test collection-based IR evaluation needs extension toward sessions - A case of extremely short queries. In: Proc. AIRS. pp. 63–74 (2009)
  * [30] Lin, J., Ma, X., Lin, S., Yang, J., Pradeep, R., Nogueira, R.: Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In: Proc. SIGIR. pp. 2356–2362. ACM (2021)
  * [31] Liu, B., Craswell, N., Lu, X., Kurland, O., Culpepper, J.S.: A comparative analysis of human and automatic query variants. In: Proc. SIGIR. pp. 47–50 (2019)
  * [32] Liu, J., Sarkar, S., Shah, C.: Identifying and predicting the states of complex search tasks. In: Proc. CHIIR. pp. 193–202 (2020)
  * [33] Mackenzie, J., Moffat, A.: Modality effects when simulating user querying tasks. In: Proc. ICTIR. pp. 197–201 (2021)
  * [34] Maxwell, D., Azzopardi, L.: Agents, simulated users and humans: An analysis of performance and behaviour. In: Proc. CIKM. pp. 731–740. ACM (2016)
  * [35] Maxwell, D., Azzopardi, L.: Simulating interactive information retrieval: Simiir: A framework for the simulation of interaction. In: Proc. SIGIR. pp. 1141–1144. ACM (2016)
  * [36] Moffat, A., Scholer, F., Thomas, P., Bailey, P.: Pooled evaluation over query variations: Users are as diverse as systems. In: Proc. CIKM. pp. 1759–1762 (2015)
  * [37] Pääkkönen, T., Kekäläinen, J., Keskustalo, H., Azzopardi, L., Maxwell, D., Järvelin, K.: Validating simulated interaction for retrieval evaluation. Inf. Retr. J. 20(4), 338–362 (2017)
  * [38] Ruthven, I., Lalmas, M.: A survey on the use of relevance feedback for information access systems. Knowl. Eng. Rev. 18(2), 95–145 (2003)
  * [39] Sloan, M., Yang, H., Wang, J.: A term-based methodology for query reformulation understanding. Inf. Retr. J. 18(2), 145–165 (2015)
  * [40] Tague, J., Nelson, M.J.: Simulation of user judgments in bibliographic retrieval systems. In: Proc. SIGIR. pp. 66–71 (1981)
  * [41] Verberne, S., Sappelli, M., Järvelin, K., Kraaij, W.: User simulations for interactive search: Evaluating personalized query suggestion. In: Proc. ECIR. pp. 678–690 (2015)
  * [42] Verberne, S., Sappelli, M., Kraaij, W.: Query term suggestion in academic search. In: Proc. ECIR. pp. 560–566 (2014)
  * [43] Voorhees, E.M.: Variations in relevance judgments and the measurement of retrieval effectiveness. In: Proc. SIGIR. pp. 315–323 (1998)
  * [44] Yang, H., Guan, D., Zhang, S.: The query change model: Modeling session search as a markov decision process. ACM Trans. Inf. Syst. 33(4), 20:1–20:33 (2015)
  * [45] Yang, P., Fang, H., Lin, J.: Anserini: Reproducible ranking baselines using lucene. ACM J. Data Inf. Qual. 10(4), 16:1–16:20 (2018)
  * [46] Zhai, C., Lafferty, J.D.: A study of smoothing methods for language models applied to ad hoc information retrieval. In: Proc. SIGIR. pp. 334–342 (2001)
  * [47] Zhang, Y., Liu, X., Zhai, C.: Information retrieval evaluation as search simulation: A general formal framework for IR evaluation. In: Proc. ICTIR. pp. 193–200 (2017)


# First Attempt at Building Parallel Corpora for Machine Translation of
Northeast India’s Very Low-Resource Languages

Atnafu Lambebo Tonja1,2,3, Melkamu Mersha 1, Ananya Kalita4,
Olga Kolesnikova 2 and Jugal Kalita 1

1 University of Colorado, Colorado Springs, USA 2 Instituto Politécnico
Nacional, Mexico city, Mexico,
3 Lelapa AI, South Africa, 4 Palmer Ridge High School, Monument, Colorado, USA

###### Abstract

This paper presents the creation of initial bilingual corpora for thirteen
very low-resource languages of India, all from Northeast India. It also
presents the results of initial translation efforts in these languages. It
creates the first-ever parallel corpora for these languages and provides
initial benchmark neural machine translation results for these languages. We
intend to extend these corpora to include a large number of low-resource
Indian languages and integrate the effort with our prior work with African and
American-Indian languages to create corpora covering a large number of
languages from across the world.

Figure 1: The languages under consideration fall under the Sino-Tibetan family
and two subfamilies, Tani and Tibeto-Burman.

##  1 Introduction

In the last few years, there have been significant advancements in deep
learning approaches, such as the development of transformers Vaswani et al.
(2017). These advancements have greatly improved machine translation (MT), a
core natural language processing (NLP) task. Regarding coverage and
translation quality, MT has shown remarkable improvements Wang et al. (2021).
Most models and methods for high-resource languages do not perform well in
low-resource settings. Low-resource languages have also suffered from
inadequate language technology designs Costa-jussà et al. (2022); Tonja et al.
(2023a); King (2015); Joshi et al. (2019); Tonja et al. (2022, 2023b); Yigezu
et al. (2021). Creating effective methods for natural language tasks is
challenging, with extremely limited resources and little to no available data.
The problem becomes worse without a parallel dataset for a vast number of
world’s languages Joshi et al. (2020); Ranathunga et al. (2023); Adebara and
Abdul-Mageed (2022).

Ethnologue111www.ethnologue.com enumerates 7,618 extant human languages in the
world, of which it estimates 3,072 are endangered. India has 30 languages
spoken by more than a million native speakers, 92 additional languages spoken
by more than 10,000 people, and 1599 other languages. The terms language and
dialect may have been conflated in Census of India
reports222https://en.wikipedia.org/wiki/Languages_of_India. Of these, 197
languages are endangered Dash et al. (2022). The languages with which work in
this paper are a small fraction of such languages.

This paper makes a selection of low-resource languages of India, all from
Northeast India, with the intention of developing resources to facilitate
neural machine translation. We choose Northeast India because this area has a
very high concentration of very low-resource and/or endangered languages, and
the area being remote from mainland India has exacerbated scholarly neglect.

##  2 Related Work

Dash (2020) reviewed the issues languages with a small number of speakers face
in India. Based on extensive research, the author surmised that the amount of
published work on low-resource and endangered languages of India, especially
in the context of technology and, in particular, natural language processing,
is miniscule.

Chauhan et al. (2021) provided a monolingual corpus of Kangri (a language
spoken by 1.1 million people in Himachal Pradesh and Punjab) with 1.8M words,
as well bitext Hindi-Kangri corpus of 27 thousand words. For NMT, they used a
model that learns to map word embedding from one language to another in an
unsupervised manner Artetxe et al. (2018). Only cursory details are given
regarding the architecture of the neural model, although translation metrics
are given in terms of BLEU and Meteor scores. The corpora are available on
Github333https://github.com/chauhanshweta/Kangri_corpus.

Acharya (2021) discussed a proposal for building a digital archive to collect
and preserve textual, audio, and video documentation of twelve Munda
languages, a sub-family of Austro-Asiatic languages. Of twenty Munda
languages, spoken by between just 20 to 7M people, UNESCO has identified
twelve as endangered. The authors proposed to use advanced technologies like
artificial intelligence in the design of the archive. However, it is not clear
what has been achieved.

To the authors’ knowledge, there is no prior computational work of any kinda
in the very low-resource and endangered languages of Northeast India.

##  3 Languages Under Consideration

The Sino-Tibetan language family includes more than 400 modern languages
spoken in China, India, Burma, and Nepal. It is one of the most diverse
language families in the world, with 1.4 billion speakers, including Chinese,
Tibetan, and Burmese. Based on a phylogenetic study of 50 ancient and modern
Sino-Tibetan languages, scholars have recently concluded that the Sino-Tibetan
languages originated in North China around 7,200 years ago Sagart et al.
(2019). Various classification schemes have been proposed for the Sino-Tibetan
family of languages Matisoff (2003, 2015); Driem (2001). Our discussion
follows Matisoff’s classification that divides Sino-Tibetan languages into a
number of sub-families at various levels.

Language | ISO Code |  Family | Speakers |  Location |  Corpus Domain |  Corpus Size
---|---|---|---|---|---|---
Adi | adi |  Tani | 150K |  Arunachal Pradesh |  Religious |  29301
Angami | njm |  Tibeto-Burman | 150K |  Nagaland |  " |  30017
Ao | njo |  Tibeto-Burman | 607K |  Nagaland |  " |  29121
Apatani | apt |  Tani | 45K |  Arunachal Pradesh |  " |  7185
Chokri | nri |  Tibeto-Burman | 111K |  Nagaland |  " |  7821
Dimasa | dis |  Tibeto-Burman | 137K |  Assam, Nagaland |  " |  10275
Karbi | mjw |  Tibeto-Burman | 2.5M |  Assam, Meghalaya, Arunachai Pradesh |  " |  7185
Kokborok | trp |  Tibeto-Burman | 1M |  Tripura, Assam, Mizoram, Myanmar, Bangladesh |  " |  29298
Konyak | nbe |  Tibeto-Burman | 246K |  Nagaland, Myanmar |  " |  28518
Mising | mrg |  Tani | 629K |  Assam |  " |  7825
Paite | pck |  Tibeto-Burman | 1M |  Manipur, Mizoram, Assam, Myanmar |  " |  29615
Tangkhul | nmf |  Tibeto-Burman | 140K |  Manipur, Nagaland |  " |  28324
Thado | tcz |  Tibeto-Burman | 350K |  Manipur, Nagaland, Assam, Mizoram |  " |  29004
Table 1: Languages Under Consideration. Accurate population count is difficult
to obtain and varies substantially among sources, corpus size shows the number
of parallel sentences.

Table 1 provides a list of languages we work with in this paper. No detailed
description of the languages is given in this paper due to lack of space. The
languages are from Northeast India, which is linguistically very diverse and
has a large number of very low-resource languages that are vulnerable or
endangered. The languages we have chosen all belong to the Sino-Tibetan
family. Under this class, there are two sub-families called Tibeto-Burman and
Taani, among other sub-families. All our languages come from these two sub-
families (see Figure 1).

##  4 Dataset

###  4.1 Dataset Collection

We obtained datasets in 13 Indian languages from religious domains through a
Bible-related website444https://www.bible.com/. To extract the Bible data from
these websites, we utilized a web crawler that identified the structure of web
documents, including pages, books, and phrases, for each article. Python
libraries like requests, regular expressions (R), and Beautiful Soup (BS) were
used to extract article content and analyze website structure from a given
URL. Extensive research did not discover any additional publicly available
texts in these languages.

###  4.2 Sentence Alignment

We gathered the limited corpora for various languages and aligned each Indian
language sentence with a corresponding sentence in English to create a dataset
for the MT experiment. We followed the heuristic alignment method outlined by
the Tonja et al. (2023c) to align the sentences.

###  4.3 Dataset Pre-processing

We aligned the texts of Indian languages with their corresponding translations
in English. Before splitting the corpus, we pre-processed by removing numbers,
special characters, and sentences that contain less than five words. We
divided the pre-processed corpus into training, development, and test sets in
a 70:10:20 ratio for the baseline experiments. Detailed information on the
selected languages, language families, domain, and dataset size can be found
in Table 1.

Figure 2: Benchmark translation results for transformer and fine-tuned
approaches in both directions (English/low-resource Indian languages).

Model | en-xx |
---|---|---
adi | apt | dis | mjw | mrg | nbe | njm | njo | nmf | nri | pck | tcz | trp | Avg.
| Bleu Score |
Transformer | 1.62 | 2 | 4.33 | 2.17 | 5.12 | 2.91 | 4.26 | 6.32 | 2.81 | 0.66 | 10.23 | 4.35 | 9.18 | 4.30
m2m100-fine-tuned | 6.06 | 15.49 | 13.82 | 15.91 | 19.23 | 12.31 | 14.07 | 18.76 | 10.88 | 5.10 | 23.03 | 13.28 | 13.16 | 13.93

Table 2: Benchmark translation results from English to low-resource Indian
languages

Model | xx-en |
---|---|---
adi | apt | dis | mjw | mrg | nbe | njm | njo | nmf | nri | pck | tcz | trp | Avg.
| Bleu Score |
Transformer | 5.09 | 5.29 | 5.11 | 5.3 | 4.02 | 7.32 | 9.8 | 11.06 | 6.64 | 4.51 | 11.54 | 11.49 | 6.88 | 7.63
m2m100-fine-tuned | 17.62 | 14.11 | 17.42 | 24.52 | 24.03 | 23.27 | 26.44 | 28.37 | 22.61 | 17.12 | 28.16 | 31.03 | 20.90 | 22.67

Table 3: Benchmark translation results from low-resource Indian Languages to
English language

##  5 Baseline Experiments and Results

###  5.1 Experiments

To evaluate the usability of the newly collected corpus, we trained bi-
directional MT models that can translate Indian languages to/from English
using (1) transformer and (2) fine-tuning multilingual machine translation
model.

1) Transformer \- is a type of neural network architecture first introduced in
the paper Attention Is All You Need Vaswani et al. (2017). Transformers are
state-of-the-art approaches widely used in NLP tasks such as MT, text
summarization, and sentiment analysis. We trained transformers from scratch
for this experiment.

2) Fine-tuning involves using a pre-trained MT model and adapting it to a
specific translation task, such as translating between a particular language
pair or in a specific domain Lakew et al. (2019). We used M2M100-48 a
multilingual encoder-decoder (seq-to-seq) model trained for many-to-many
multilingual translation Fan et al. (2020). We used a model with 48M
parameters due to computing resource limitations.

###  5.2 Results

We used Sacrebleu Post (2018) evaluation metrics to evaluate translation
models. Tables 2, 3 and Figure 2 show the translation results in both
directions (to/from English - from/to low-resource Indian languages)

####  5.2.1 Translating from English to low-resource Indian languages

In Table 2, we present the translation results from English to low-resource
Indian languages. We observed that fine-tuning the m2m100 model performs
better than using a transformer trained from scratch. The transformer model’s
performance also varies significantly (0.66 – 10.23 spBLEU) depending on the
language and corpus size. This indicates that a bilingual translation model
trained from scratch performs poorly for low-resource language training
compared to fine-tuning multilingual translation models. Fine-tuning the
multilingual model produced better results than the model built from scratch
for English to Indian language translation.

####  5.2.2 Translating from low-resource Indian languages to English

Table 3 displays the results of using English as the target language to
translate low-resource Indian languages. As is evident from the results, the
fine-tuned model outperforms the transformer model significantly when
translating from Indian languages to English. However, when it comes to
translating similar languages to English, the transformer model shows an
improvement compared to Table 2. It is worth noting that the fine-tuned model
exhibits better Bleu scores while translating to English than when translating
to low-resource Indian languages. The results indicate that languages with
larger datasets tend to perform better. Therefore, both models exhibit
improved performance while translating from low-resource Indian languages to
English, whereas the model struggles to translate from English to low-resource
Indian languages.

##  6 Conclusions and Future Work

This paper presents the first, albeit limited size, parallel corpus for 13
low-resource Sino-Tibetan Indian languages paired with English and discusses
the benchmark results for the translation of language pairs to/from low-
resource Indian languages from/to English. We evaluated the usability of the
collected corpus by using transformer and fine-tuning multilingual translation
model. From our results fine-tuning multilingual model outperforms transformer
model trained from scratch in both translation directions.

In the future, we aim to increase corpus sizes of these low-resource languages
by extracting text from scanned documents if/where available and evaluate
additional machine translation approaches to improve performance. We intend to
increase the number of languages substantially by first incorporating all low-
resource languages of India for which a Bible translation exists. We also plan
to find language communities in social media platforms such as Facebook and
attempt to gather additional bitext documents and evaluate the quality of
translations with native speakers.

## References

  * Acharya (2021) Kaushik Acharya. 2021.  KaushikAcharya at SemEval-2021 task 9: Candidate generation for fact verification over tables.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 1271–1275, Online. Association for Computational Linguistics.
  * Adebara and Abdul-Mageed (2022) Ife Adebara and Muhammad Abdul-Mageed. 2022.  Towards afrocentric NLP for African languages: Where we are and where we can go.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3814–3841, Dublin, Ireland. Association for Computational Linguistics.
  * Artetxe et al. (2018) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018.  A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings.  _arXiv preprint arXiv:1805.06297_.
  * Chauhan et al. (2021) Shweta Chauhan, Shefali Saxena, and Philemon Daniel. 2021.  Monolingual and parallel corpora for kangri low resource language.  _arXiv preprint arXiv:2103.11596_.
  * Costa-jussà et al. (2022) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.  No language left behind: Scaling human-centered machine translation.  _arXiv preprint arXiv:2207.04672_.
  * Dash (2020) Rajendra Kumar Dash. 2020.  Revitalizing endangered languages in india: Can public-private partnership (ppp) work.  In _2nd International Conference on Social Sciences in the 21st Century, Seminar Paper_.
  * Dash et al. (2022) Sarthak Dash, Sugato Bagchi, Nandana Mihindukulasooriya, and Alfio Gliozzo. 2022.  Permutation invariant strategy using transformer encoders for table understanding.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 788–800, Seattle, United States. Association for Computational Linguistics.
  * Driem (2001) George van Driem. 2001.  Languages of the himalayas: an ethnolinguistic handbook of the greater himalayan region: containing an introduction to the symbiotic theory of language.  _(No Title)_.
  * Fan et al. (2020) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020.  Beyond english-centric multilingual machine translation.  _arXiv preprint_.
  * Joshi et al. (2019) Pratik Joshi, Christain Barnes, Sebastin Santy, Simran Khanuja, Sanket Shah, Anirudh Srinivasan, Satwik Bhattamishra, Sunayana Sitaram, Monojit Choudhury, and Kalika Bali. 2019.  Unsung challenges of building and deploying language technologies for low resource language communities.  _arXiv preprint arXiv:1912.03457_.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the nlp world.  _arXiv preprint arXiv:2004.09095_.
  * King (2015) Benjamin Philip King. 2015.  _Practical Natural Language Processing for Low-Resource Languages._ Ph.D. thesis.
  * Lakew et al. (2019) Surafel M Lakew, Alina Karakanta, Marcello Federico, Matteo Negri, and Marco Turchi. 2019.  Adapting multilingual neural machine translation to unseen languages.  _arXiv preprint arXiv:1910.13998_.
  * Matisoff (2003) James A Matisoff. 2003.  _Handbook of Proto-Tibeto-Burman: system and philosophy of Sino-Tibetan reconstruction_.  Univ of California Press.
  * Matisoff (2015) James A Matisoff. 2015.  _The Sino-Tibetan etymological dictionary and thesaurus_.  Regents of the University of California.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Ranathunga et al. (2023) Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. 2023.  Neural machine translation for low-resource languages: A survey.  _ACM Computing Surveys_ , 55(11):1–37.
  * Sagart et al. (2019) Laurent Sagart, Guillaume Jacques, Yunfan Lai, Robin J Ryder, Valentin Thouzeau, Simon J Greenhill, and Johann-Mattis List. 2019.  Dated language phylogenies shed light on the ancestry of sino-tibetan.  _Proceedings of the National Academy of Sciences_ , 116(21):10317–10322.
  * Tonja et al. (2023a) Atnafu Lambebo Tonja, Tadesse Destaw Belay, Israel Abebe Azime, Abinew Ali Ayele, Moges Ahmed Mehamed, Olga Kolesnikova, and Seid Muhie Yimam. 2023a.  Natural language processing in ethiopian languages: Current state, challenges, and opportunities.  _arXiv preprint arXiv:2303.14406_.
  * Tonja et al. (2022) Atnafu Lambebo Tonja, Olga Kolesnikova, Muhammad Arif, Alexander Gelbukh, and Grigori Sidorov. 2022.  Improving neural machine translation for low resource languages using mixed training: The case of ethiopian languages.  In _Mexican International Conference on Artificial Intelligence_ , pages 30–40. Springer.
  * Tonja et al. (2023b) Atnafu Lambebo Tonja, Olga Kolesnikova, Alexander Gelbukh, and Grigori Sidorov. 2023b.  Low-resource neural machine translation improvement using source-side monolingual data.  _Applied Sciences_ , 13(2):1201.
  * Tonja et al. (2023c) Atnafu Lambebo Tonja, Christian Maldonado-Sifuentes, David Alejandro Mendoza Castillo, Olga Kolesnikova, Noé Castro-Sánchez, Grigori Sidorov, and Alexander Gelbukh. 2023c.  Parallel corpus for indigenous language translation: Spanish-mazatec and spanish-mixtec.  _arXiv preprint arXiv:2305.17404_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2021) Haifeng Wang, Hua Wu, Zhongjun He, Liang Huang, and Kenneth Ward Church. 2021.  Progress in machine translation.  _Engineering_.
  * Yigezu et al. (2021) Mesay Gemeda Yigezu, Michael Melese Woldeyohannis, and Atnafu Lambebo Tonja. 2021.  Multilingual neural machine translation for low resourced languages: Ometo-english.  In _2021 International Conference on Information and Communication Technology for Development for Africa (ICT4DA)_ , pages 89–94. IEEE.