# Knowledge Base Question Answering for Space Debris Queries

Paul Darm1
<EMAIL_ADDRESS>Antonio Valerio Miceli-Barone2
<EMAIL_ADDRESS>Shay B. Cohen2
<EMAIL_ADDRESS>Annalisa Riccardi1
<EMAIL_ADDRESS>1 Mechanical and Aerospace Engineering,
University of Strathclyde
2 School of Informatics, University of Edinburgh University

###### Abstract

Space agencies execute complex satellite operations that need to be supported
by the technical knowledge contained in their extensive information systems.
Knowledge bases (KB) are an effective way of storing and accessing such
information at scale. In this work we present a system, developed for the
European Space Agency (ESA), that can answer complex natural language queries,
to support engineers in accessing the information contained in a KB that
models the orbital space debris environment. Our system is based on a pipeline
which first generates a sequence of basic database operations, called a
sketch, from a natural language question, then specializes the sketch into a
concrete query program with mentions of entities, attributes and relations,
and finally executes the program against the database. This pipeline
decomposition approach enables us to train the system by leveraging out-of-
domain data and semi-synthetic data generated by GPT-3, thus reducing
overfitting and shortcut learning even with limited amount of in-domain
training data. Our code can be found at https://github.com/PaulDrm/DISCOSQA.

##  1 Introduction

Space debris are uncontrolled artificial objects in space that are left in
orbit during either normal operations or due to malfunctions. Collisions
involving space debris can generate secondary debris which can cause more
collisions, potentially leading to a runaway effect known as “Kessler
Syndrome” (Kessler and Cour-Palais, 1978; Kessler et al., 2010), which in the
worst-case scenario could make large ranges of orbits unusable for space
operations for multiple generations.

Question: | What is the inclination of the orbit of Hubble?
---|---
Sketch: |  FindRelateQueryAttr
Arguments: |  HubbleOrbitinclination
Question: | How many rocket debris objects have re-entered
| Earth’s atmosphere before 2019?
Sketch: |  FindAllFilterYearFilterConceptCount
Arguments: |  Reentry,2019,$<$RocketDebrisObjects

Figure 1: Two representative queries for DISCOS and their decomposition
according to the Program Induction method.

Therefore, space agencies have established departments responsible for
cataloging the space debris environment, which can be used for space traffic
management, collision avoidance, re-entry analysis, and raising public
awareness of the problem.111https://tinyurl.com/44tc24d4

The European Space Agency (ESA) has catalogued over 40,000 trackable and
unidentified objects in its DISCOS (Database and Information System
Characterizing Objects in Space) Knowledge Base (KB) (Klinkrad, 1991; Flohrer
et al., 2013). Accessing this information efficiently often requires technical
expertise in query languages and familiarity with the specific schema of
DISCOS, which may fall outside the skillset of the engineers searching for
relevant information in the database. In this project, we developed a question
answering system for the DISCOS KB. This deployed prototype enables ESA
engineers to query the database with complex natural language (English)
questions, improving their ability to make informed decisions regarding space
debris.

Recent breakthroughs in open question answering have been achieved using large
language models that have been fine-tuned as dialog assistants, such as
ChatGPT.222https://chat.openai.com These models, however, are black boxes that
store knowledge implicitly in their parameters which makes it hard to
guarantee that their answers are supported by explicit evidence, understand
their failures and update them when the supporting facts change. In contrast,
parsing a question into a query program and then executing it on an explicit
KB is guaranteed to provide a factual correct answer provided the KB and query
program are correct. Our approach is particularly useful for applications such
as satellite operations where accuracy and reliability are critical.

The main challenge for this project was that no training set or example
questions were available for the DISCOS KB. This issue, combined with the
large amount of unique and diverse objects in the database, precluded a
straightforward application of common supervised learning techniques. Although
possible strategies for solving this task, such as direct semantic parsing of
the query with seq2seq models, were identified in the literature, they suffer
from problems with compositional generalization Herzig et al. (2021); Furrer
et al. (2020). Furthermore, very little work has been done on generalizing to
KB element components that were never seen during training Cao et al. (2022b);
Das et al. (2021a); Huang et al. (2021).

To overcome these challenges, we apply and adapt a methodology from the
literature called Program Transfer (Cao et al., 2022b) to significantly reduce
the required dataset for adequate generalization over the complete DISCOS KB.
This is a two-step approach. For each user query first a program sketch is
predicted, consisting of a sequence of query functions where the arguments are
either variables or placeholders, then the representation of the query is
compared to the representations of the KB entities, in order to fill out the
placeholders with arguments relevant to the query text. The underlying query
language of this approach is called Knowledge-oriented-Programming-Language
(KoPL) for which two representative example questions are shown together with
their decomposition into sketch and arguments in Figure 1.

We also conduct a data collection study with domain experts, and we apply a
data augmentation pipeline leveraging the underlying ontology of the KB and
prompting a Large Language Model (LLM) to generate automatically more training
examples. The architecture was retrained with different domain-specific LMs
and baselines to determine the benefits of using a domain-specific pre-trained
encoder.

The main contributions of this paper are:

  * •

Applying and adapting a methodology described in the literature for complex
knowledge base question answering (CKBQA) on a novel industry-relevant
database, with a large and dynamic set of unique entities;

  * •

Collecting a new dataset on this database from domain-experts and leveraging
the in-context learning capability of LLMs for data augmentation on it;

  * •

Evaluating the use of domain-specific LMs as different encoders on our curated
dataset;

  * •

Demonstrating the effectiveness of the approach by achieving comparable
results to general-purpose LLMs

##  2 Related Work

#### Low-resource CKBQA

Pre-trained language models have demonstrated state-of-the-art performance in
semantic parsing for complex question answering on KBs where the same logic
compounds are contained in both the training and validation sets Furrer et al.
(2020). However, they struggle with compositional generalization, where the
“compounds” (combinations) of components are diverse between training and
validation, even if all components (entity, relation, program filters) have
been seen during training (Herzig et al., 2021). Das et al. (2021b) explored
retrieval-based methods to pick the top $n$ similar examples from the training
set and use them as additional input for the prediction. In theory, this would
make it possible to reason over changes on the KB by only adding new examples
to the training set without the need of retraining the whole model. Another
approach is adapting the architecture of language models to incorporate the
structure of a KB directly for the prediction. For example, Huang et al.
(2021) ranked FreeBase KB entities by using an EleasticSearch search API to
identify these entities. When generating the query program, instead of
entities a special token is predicted, which in the post-processing step get
replaced by the top ranked entity identified by ElasticSearch. Although,
achieving good results, it is unclear how this would translate to queries with
multiple entities and also has the typical limitations of ElasticSearch.
Another method is the Program Induction and Program Transfer method, where a
sequence of _functions_ , or a _sketch_ , is generated from the input query. A
single function here stands for a basic logic operation on the KB. The premise
is that the sketch is mostly dependent on the formulation of the input query
and less dependent on the KB, therefore the training on a source domain can
transfer to inference on a target domain. In a second step, the particular
inputs that each function in the sketch receives are identified from the
elements of the KB through comparing their representations with the one of the
model at the specific function. During training, the goal is to create
sophisticated representations for the components of the KB as well as for the
query that can generalize to components which were not seen during training
Cao et al. (2022b).

#### Domain-specific Language Models

Using self-supervised pre-trained language models is the de-facto standard
approach in modern natural language processing (NLP). These models are trained
on large volumes of text, learning representations that can generalize over
natural language variations and capture long-term dependencies between the
input tokens. During fine-tuning, these learned features and representations
commonly lead to improved results on the downstream task. While the interplay
between the amount of in-domain data, model capacity and training regime is
complex Zhao et al. (2022), as a general rule, training these models on in-
domain task-related text improves the performance of this task (Maheshwari et
al., 2021; Berquand et al., 2021; Arnold et al., 2022; Joshi et al., 2023). An
alternative approach involves modifying the pre-training objective according
to the domain. In the context of question-answering with tabular data, it was
explored how a language model could function during pre-training as a SQL-
query excecutor, predicting the results of an automated created SQL query on
the corresponding concatenated table, to elicit an understanding of the
underlining dependencies in tables. This approach resulted in improved
performance on related downstream tasks (Liu et al., 2022). For KBs, together
with the standard Masked-Language-Modelling loss an architecture called Kepler
was tested that also minimizes a contrastive loss on related KB triples, where
both the correct triples and randomly perturbed incorrect triples are scored,
and the loss penalizes scoring the perturbed triples higher than the correct
ones (Wang et al., 2021a).

##  3 Dataset Collection and Use

We used two sources of data for the study. The first one is the original
dataset which first introduced the KoPL-format for general-domain knowledge-
base question answering (described in §3.1). The second one is for our
particular domain with information about objects in space (§3.2). We also
describe in §3.3 how we further collected training data for fine-tuning a
language model, and how we augmented this dataset with new question-answer
pairs (§3.4).

###  3.1 The KQA Pro Dataset

The KQA pro dataset Cao et al. (2022a) is a large scale dataset for complex
question answering over a knowledge base. It contains over 120,000 diverse
questions for an subset of entities from the Freebase KB and their associated
relations, attributes and concepts. The reasoning process to arrive at a
solution is provided in the form of a Knowledge-oriented-Programming-Language
(KoPL), which was designed specifically for this dataset. The question-program
pairs were automatically generated by randomly sampling the extracted KB and
using novel compositional templates to create a canonical form of the question
and associated answer. To increase ambiguity these questions were then
paraphrased and controlled by Amazon Mechanical Turk workers Cao et al.
(2022a).

###  3.2 The DISCOS Database

The ESA Database and Information System Characterising Objects in Space
(DISCOS)333https://discosweb.esoc.esa.int/ is a regularly updated source for
information about launches, launch vehicles, objects, spacecraft registration
numbers, mission-specific information (mass, mission objective, operator), and
most importantly orbital and fragmentation data histories for all trackable as
well as unclassified objects. With currently over 40,000 objects being
tracked, this tracking provides rich information for ESA offices monitoring
and managing space debris, collision avoidance, re-entry analyses, and
contingency support. Other actors, such as research institutes, government
organisations, or industrial companies from ESA Member states can apply for an
account to access the information provided by DISCOS free of charge. A
comparison between the the DISCOS KB and the KB used for the KQA Pro dataset
can be seen in Table 1

Dataset | #Entities | #Relations | #Concepts
---|---|---|---
KQA Pro | 16,960 | 363 | 794
DISCOS | 73,354 | 32 | 39

Table 1: Entity, relation, and concept counts for the KQA Pro and DISCOS KBs

###  3.3 Data Collection

As there was no question-program-answer training set available on the DISCOS
KB, potential queries had to be collected from domain-experts via a simple
user interface. The interface allowed domain experts to input queries of
interest, along with their username and feedback, see Appendix E. Based on the
domain experts’ feedback, a manually labeled baseline dataset of around 102
question-KoPL program pairs was created.

###  3.4 Data Augmentation

To extend the limited baseline dataset and increase diversity of potential
queries, we augmented the dataset by creating paraphrases of the questions,
shown to add robustness to question answering systems Fader et al. (2013);
Narayan et al. (2016). For each unique program sketch, the schema of the
ontology was used to alter the arguments of the single functions in that
sketch. For example, for the program Find(‘Saturn V’) $\rightarrow$
QueryAttr(‘mass’), the concept of the Find function argument (Saturn V) was
identified as LaunchVehicle. Subsequently, the ontology was queried for other
entities of the same concepts and also for associated attributes to substitute
the argument for the QueryAttr function.

In order to generate appropriate questions for each “augmented” program, we
used the few-shot and in-context learning capabilities of GPT-3 (Brown et al.,
2020). A prompt was curated, consisting of question-program pairs from the
manually labeled dataset, so that GPT-3 could generate a question for an
unlabelled augmented program. The sampling included all programs from the
manual dataset with the same sketch as the augmented one, as well as using
examples with the same relation type, which showed great abilities to generate
a correct question. Additionally, an instruction section was added to the
prompt, consisting mainly of a list of expansions for acronyms commonly used
in the KB’s ontology. This ensured that acronyms were expanded correctly and
reduced the likelihood of hallucinations by the LLM for these acronyms.
Examples of generated questions with their corresponding programs can be found
in Appendix D. The prompt schema can be found in Appendix A.

The benefits of an automatically generated dataset include cost-effective data
sample generation, while also ensuring a balanced distribution of complex and
simple queries as well as common (ex: Saturn V, Hubble Space Telescope) and
uncommon arguments (ex: L-186, PSLV-Q). With the use of LLMs, the generated
questions also have already slightly different semantics and syntactic
structure as the generation process is inherently statistical and can be
adjusted with parameters such as the temperature. LLMs can also leverage their
stored world knowledge, e.g. we observed the entity name "" to be
automatically translated into the english "Tianzhou 4" designation. More
examples can be seen in Appendix D. However, it is important to note that the
question generation is not foolproof and could be further optimized e.g.
through additional prompt engineering or a subsequent data cleaning procedure.

##  4 Methodology and Model Architecture

We describe in §4.1 the problem that our model aims to solve. In §4.2, we
describe what modification we applied to the methodology from Cao et al.
(2022b).

###  4.1 Problem Definition

The task is defined in the following way, given a natural language question
$Q$ we want to predict a program $y$ that traverses the knowledge base $K$ and
produces an answer $A$ for $Q$. This means:

$A=y(Q,K),\quad K=\\{E,R,C,A\\},$

where ${E,R,C,A}$ represent respectively the mutually disjoint sets of
entities, relations, concepts and attributes in $K$. More specifically for a
set entities in the training set $E_{t}$, the task is to be able to generalise
to the set of $E_{v}$, which were unseen during training, with $E=E_{t}\cup
E_{v}$, $E_{t}\cap E_{v}=\emptyset$. Therefore, we apply a program induction
and transfer methodology, predicting for $Q$ a program, a tuple of actions,

$y(Q,K)=(o^{1}(arg^{1}),\dots,o^{t}(arg^{t})),$ $o^{i}\in O,\forall
i=1,\ldots,t;\quad arg^{t}\in E\cup R\cup C\cup A,$

where $O$ is a set predefined basic functions executable on the KB, with each
of them taking one disjoint set as inputs from the pool of arguments from the
KB. In the first step, the sketch $[o^{1},\dots,o^{t}]\in O^{t}$ is generated
by encoding the question $Q$ with a pre-trained language model and using its
representation as the starting point for a GRU-based decoder with attention
mechanism (Bahdanau et al., 2015; Cao et al., 2022b).

The input arguments for each function, $o^{1},\ldots,o^{t}$, are chosen in a
second step by calculating the probability between the encoded representations
of the $i$-th candidate $R_{i}^{t}$ in the KB at position $t$ in the sequence,
and the representation of the decoder at position $t$, $h^{t}$. The
probability is computed as:

$p(arg^{t}|R_{i}^{t},h^{t})=\textrm{softmax}(R^{t}_{i},h^{t}),$

and is followed by choosing the most likely candidate as the input argument
for the function.

We refer the reader to Cao et al. (2022b) for the details. In addition to the
standard BERT model we also train RoBERTA-based (Liu et al., 2019) domain-
specific encoders (see Appendix B).

###  4.2 Enhancements to Program Transfer

At the beginning, the prediction for entities, relations, and operations
($\\{<,>,=\\}$) had to be re-implemented as it was not available in the
associated source code.444https://github.com/thu-keg/programtransfer

The standard methodology of comparing the representations to all inputs during
training posed another challenge for entities, as during the training step the
gradients for all them would need to be stored, which were exceeding the
available virtual random access memory on the system.

As a result, instead of comparing the representation at each sequence position
to all entities in the KB, only the subset of entities in the current batch
are compared with each other.555Note that these entities are part of the
training set entities, hence are all known apriori. In addition, random
samples from the complete entity set where selected and added to the batch to
also give signals to entities not occurring in the training set.

More formally, let $X^{(j)}$ be the $j$th batch from the training set, $E$ be
the overall set of entities, and $X^{(j)}_{E}$ be the set of entities of the
training batch. At each training batch $j$, we use the set of entities
$e^{(j)}$ to compare the probability of their representations against the
candidate input argument in batch $j$, as defined by:

$e^{(j)}=X^{(j)}_{E}\cup f^{(j)},$ (1)

where $f^{(j)}=\\{e_{1},e_{2},\dots,e_{n}\\}$ are $n$ randomly sampled
entities from $E$ without replacement. Furthermore, for extracting time values
from queries we use the SUTime library Chang and Manning (2012), whereas for
extracting numerical patterns we use a simple regex schema. In addition, for
parsing the predicted inputs into a form than can be read by the KoPL engine a
parser function had to be written as well as a method for assigning the
dependencies between the single basic functions. Other small adaptation of the
original approach included the addition of a normalization layer between the
linear layers for the prediction of the single arguments as well as masking-
out padding tokens from the loss calculation of the function generation
component. A schematic overview of the pipeline can be seen in Appendix C.

##  5 Experiments

We next describe our dataset (§5.1), the way we trained our models (§5.2) and
our results (§5.3).

###  5.1 The DISCOS-Questions-Programs Dataset

The creation of the training and validation datasets involved the following
steps: First, all the unique sequences of functions were extracted from the
manually labeled dataset. Then, ten augmented programs were generated for each
unique program in the manually labeled dataset by substituting the inputs and
generating questions as described in the data augmentation methodology. It is
noteworthy that the entities included in the manually labeled dataset were not
considered as candidates for the data augmentation. The questions for the
augmented examples were generated using the large language model code-
davinci-002.666https://platform.openai.com/playground Through trial and error,
a temperature of 0.75 was used for generation as it produced diverse examples
while still capturing the meaning of the associated KoPL-program. Finally, the
augmented dataset was split into training and validation datasets, where the
original manually labeled dataset and 0.05% of randomly sampled examples from
the augmented dataset were used as the validation set and the rest of the
augmented dataset was used for training. In a subsequent filtering step,
programs that appeared in the validation set were filtered from the training
dataset. The resulting DISCOS-Questions-Programs (DQP) dataset consists of 905
samples for training and 151 samples for validation.

###  5.2 Model Training

The preliminary results indicated that training directly on the DQP dataset
did not lead to convergence in entity prediction. We then proceeded to first
train on the out-of-domain original KQA Pro dataset and then further train on
the DQP dataset. The hyperparameters for pretraining on the KQA pro dataset
were adopted from the original paper. For training the DQP dataset, the
hyperparameters were left unchanged, except for an adapted learning rate for
the decoder, which was set to $10^{-4}$ instead of $10^{-3}$. For the
experiments, different domain-adapted models were used as the encoder and then
compared to each other as well as baseline models. For more information about
the domain-adapted models see Appendix B.

###  5.3 Results

The analysis of different models was challenging as multiple components
(functions, entities, relations, etc.) need to be predicted to arrive at the
full program that can be run against the KB. Therefore, the analysis was
divided into two parts. Firstly, the accuracy at the lowest validation loss
for each component was compared separately between all the different trained
models, providing an overview of each model’s best predictive performance for
each component. The results can be seen in Table 2. Although no model
consistently outperformed the others on all components, CosmicRoBERTa Berquand
et al. (2021) achieved the highest performance on four out of six accuracies.

The validation loss curves showed that during training, the validation
accuracy can drop for one component while it rises for another. The validation
loss curves can be found in Appendix F.

For deployment of a single model, it is necessary to identify a checkpoint
where the model predicts accurately across all components. To obtain a more
holistic view of the performance, the models were also compared by summing up
the normalized validation losses for each component.

Accuracy | BERT | Kepler | RoBERTa | CR
---|---|---|---|---
Function | 0.797 | 0.812 | 0.796 | 0.826
Entity | 0.874 | 0.887 | 0.907 | 0.927
Attribute | 0.955 | 0.948 | 0.948 | 0.948
Relation | 0.938 | 0.983 | 0.983 | 1
Concept | 0.872 | 0.896 | 0.92 | 0.89
Operations | 1 | 1 | 1 | 1
Table 2: Accuracy at lowest validation loss for each respective component. CR
stands for CosmicRoBERTa. The best score for each component is highlighted in
bold, second best in italics.

The results of summing up the normalized validation losses with equal weights
are shown in Table 3. On average, the RoBERTa-base model has the lowest
validation loss, followed by the Kepler model. However, no model consistently
outperforms the others in terms of prediction accuracy. From an application
perspective, the most important metrics are the accuracy in predicting
functions and entities. To obtain the correct answer, the most crucial step is
to predict the correct sequence of functions, and for multi-hop queries, it is
essential to identify the correct starting entity. In addition, the number of
unique entities is magnitudes higher than the number of unique attributes or
relations in the KB, which makes identifying the right entity more difficult.
Among the models considered, CosmicRoBERTa stands out as the model that
performs well both in predicting functions and entities. Unfortunately, the
Kepler model only sporadically showed improvements over RoBERTa-base. The
reason for this could be the very limited pre-training corpus, which as a
result was significantly smaller than the one from CosmicRoBERTa As a result,
for the purpose of deploying a single model, the decision was made to choose
CosmicRoBERTa.

Accuracy | BERT | Kepler | RoBERTa | CR
---|---|---|---|---
Min. sum valid loss | 0.21 | 0.171 | 0.11 | 0.252
Function | 0.79 | 0.734 | 0.775 | 0.789
Entity | 0.874 | 0.887 | 0.894 | 0.927
Attribute | 0.935 | 0.948 | 0.941 | 0.915
Relation | 0.978 | 0.95 | 0.969 | 0.969
Concept | 0.853 | 0.8841 | 0.908 | 0.927
Operations | 1 | 0.97 | 1 | 0.94

Table 3: Accuracy at the lowest validation loss summed over all compoennts.
The best score for each component is highlighted in bold, second best in
italics. CR stands for CosmicRoBERTa.

The results of the prediction are overall very impressive, with high accuracy
scores over all components. It is especially worth highlighting that from over
40,000 entities in the database, only 400 appear in the training and
validation set and only 1 of 71 entities in the validation set also appear in
the training set. Despite this, some models achieve an accuracy of over 90% in
predicting entities, demonstrating their strong ability to generalize to
entities not seen during training, which was a critical user requirement for
our system.

In addition, we benchmarked our method to recently released general purpose
models such as ChatGPT777https://openai.com/blog/chatgpt and GPT-4 OpenAI
(2023). For each model, training set examples were randomly added to the
prompt until the respective model’s context limit is reached. Then the models
were prompted to generate the right program for a question from the validation
set. Our methodology has an overall accuracy of predicting the right program
completely of 48%, which is higher than the performance of around 25% of
ChatGPT and comparable with the performance of GPT-4 of around 50%. A detailed
comparison can be found in Table 4. This further demonstrates the efficiency
of our methodology as it can be run at a fraction of the necessary compute as
well as locally on consumer-grade hardware.

| CosmicRoBERTa | GPT-4 | ChatGPT-3.5
---|---|---|---
Functions | 0.79 | 0.79 | 0.516
Entities | 0.93 | 0.86 | 0.41
Relations | 0.97 | 0.87 | 0.32
Concepts | 0.93 | 0.82 | 0.61
Attributes | 0.92 | 0.84 | 0.72
Overall | 0.48 | 0.5 | 0.25

Table 4: Accuracy of deployed model (CosmicRoBERTa) versus general purpose
models ChatGPT-3.5 and GPT-4. The best score for each component is highlighted
in bold.

##  6 Conclusion

We developed a system for ESA to address the challenge of answering complex
natural language questions on their DISCOS KB. The main obstacles included a
lack of training data, the diversity and regular updates of the database
entries, and the need for an economically feasible solution. The program
transfer for complex KBQA methodology was selected for its potential to reduce
the amount of required training samples through transfer learning and its
capability to potentially generalize for examples which were never seen during
training. A data collection study was conducted with domain experts, which was
then used to augment the data through leveraging the underlying ontology of
the KB and prompting a large language model to generate fitting questions. The
architecture was retrained with different domain-specific models and baselines
to determine the benefits of using a domain-specific pre-trained encoder.
Although the results were mixed, the best performance was achieved by
CosmicRoBERTa, a pre-trained model on a space domain corpus. With an accuracy
of over 90% of predicting the right entity on the validation set over the vast
pool of candidate entities, the method demonstrates its strong ability to
predict the correct input arguments for unseen examples. This is further
demonstrated in the comparison with general-purpose models such as ChatGPT or
GPT-4, where our method achieved competitive results. Therefore, this approach
has the potential to be extended to other databases and query languages in the
future, especially in scenarios where there are few to no training examples.

## Limitations

The study has several clear limitations. Firstly, the training and validation
datasets used in this study are still relatively small. A larger dataset would
give more robust results for comparing different encoders. Additionally, the
experiments were only conducted on the ability to generalize to unseen
entities and not on the ability to generalize to unseen sketch types, which is
also of key importance when addressing low resource CKBQA.

Moreover, the methodology used in this study relies on annotated question-
program pairs, which are expensive to collect. Learning only from question-
answer pairs or even a question with an indicated difficulty based on whether
the model was able to answer the question, could be more easier to collect.
While the models achieve high accuracy on most of the knowledge base
components, overfitting can occur at different stages during training, leading
to high accuracy for one component at one training step but poor accuracy for
another component at another step. In the future, revising the training
procedure or the model setup may help address this issue.

## Ethics Statement

In any safety-critical context like spacecraft operations, there is an
inherent risk associated with the use of automatic methods supporting human
operators. The transparency of the predicted programs could mitigate this
issue as it allows even for an engineer with limited knowledge about the
underlying query method to interpret the program to some degree. In any case,
the developed systems might support human analysis and decision making by
decreasing workload, but cannot replace it. As mentioned before the DISCOS KB
can be accessed after creating a user account. We plan on publishing the
created question-program paris and trained models online in accordance with
ESA’s guidelines.

## Acknowledgments

This study was funded by the European Space Agency (ESA) under the Intelligent
Operational Assistant project with contract No.AO/1-10776/21/D/SR. This work
was also supported by the UKRI Research Node on Trustworthy Autonomous Systems
Governance and Regulation (grant EP/V026607/1) which provided additional
funding for Antonio Valerio Miceli-Barone. We thank Evridiki Ntagiou (ESA) and
Paulo Leitao (Vision Space) for their roles as project leader and industrial
partner, respectively. We are grateful for Callum Wilson’s help (University of
Strathclyde) in creating the GraphDB KG from the original DISCOSweb API, which
was used to create the DISCOS KB. We also thank Sean Memery and Maria Luque
Anguita (University of Edinburgh) for their contribution to the KEPLER data
collection, and everyone who took part in the data collection for the first
iteration of the DISCOSWeb-questions dataset. We also thank Marcio Fonseca and
the reviewers for useful feedback on the paper.

## References

  * Arnold et al. (2022) Alexandre Arnold, Fares Ernez, Catherine Kobus, and Marion-Cécile Martin. 2022\.  Extraction d’informations de messages aéronautiques (NOTAMs) avec des modèles de langue appris de façon auto-supervisée (information extraction from aeronautical messages ).  In _Actes de la 29e Conférence sur le Traitement Automatique des Langues Naturelles. Volume 1 : conférence principale_ , pages 335–344, Avignon, France. ATALA.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Berquand et al. (2021) Audrey Berquand, Paul Darm, and Annalisa Riccardi. 2021.  Spacetransformers: Language modeling for space systems.  _IEEE Access_ , 9:133111–133122.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Cao et al. (2022a) Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. 2022a.  KQA pro: A dataset with explicit compositional programs for complex question answering over knowledge base.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6101–6119, Dublin, Ireland. Association for Computational Linguistics.
  * Cao et al. (2022b) Shulin Cao, Jiaxin Shi, Zijun Yao, Xin Lv, Jifan Yu, Lei Hou, Juanzi Li, Zhiyuan Liu, and Jinghui Xiao. 2022b.  Program transfer for answering complex questions over knowledge bases.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8128–8140, Dublin, Ireland. Association for Computational Linguistics.
  * Chang and Manning (2012) Angel X. Chang and Christopher Manning. 2012.  SUTime: A library for recognizing and normalizing time expressions.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 3735–3740, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Das et al. (2021a) Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021a.  Case-based reasoning for natural language queries over knowledge bases.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9594–9611, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Das et al. (2021b) Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021b.  Case-based reasoning for natural language queries over knowledge bases.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9594–9611, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Fader et al. (2013) Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2013.  Paraphrase-driven learning for open question answering.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1608–1618, Sofia, Bulgaria. Association for Computational Linguistics.
  * Flohrer et al. (2013) T Flohrer, S Lemmens, B Bastida Virgili, H Krag, H Klinkrad, E Parrilla, N Sanchez, J Oliveira, and F Pina. 2013.  Discos-current status and future developments.  In _Proceedings of the 6th European Conference on Space Debris_ , volume 723, pages 38–44.
  * Furrer et al. (2020) Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schärli. 2020.  Compositional generalization in semantic parsing: Pre-training vs. specialized architectures.
  * Herzig et al. (2021) Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021.  Unlocking compositional generalization in pre-trained models using intermediate representations.  _ArXiv preprint_ , abs/2104.07478.
  * Huang et al. (2021) Xin Huang, Jung-Jae Kim, and Bowei Zou. 2021.  Unseen entity handling in complex question answering over knowledge base via language generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 547–557, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Joshi et al. (2023) Harshit Joshi, Abishai Ebenezer, José Cambronero, Sumit Gulwani, Aditya Kanade, Vu Le, Ivan Radiček, and Gust Verbruggen. 2023.  Flame: A small language model for spreadsheet formulas.
  * Kessler and Cour-Palais (1978) Donald J. Kessler and Burton G. Cour-Palais. 1978.  Collision frequency of artificial satellites: The creation of a debris belt.  _Journal of Geophysical Research: Space Physics_ , 83(A6):2637–2646.
  * Kessler et al. (2010) Donald J Kessler, Nicholas L Johnson, JC Liou, and Mark Matney. 2010.  The kessler syndrome: implications to future space operations.  _Advances in the Astronautical Sciences_ , 137(8):2010.
  * Klinkrad (1991) H. Klinkrad. 1991.  Discos - esa’s database and information system characterising objects in space.  _Advances in Space Research - ADV SPACE RES_ , 11:43–52.
  * Liu et al. (2022) Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022.  TAPEX: table pre-training via learning a neural SQL executor.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _ArXiv preprint_ , abs/1907.11692.
  * Maheshwari et al. (2021) Himanshu Maheshwari, Bhavyajeet Singh, and Vasudeva Varma. 2021.  SciBERT sentence representation for citation context classification.  In _Proceedings of the Second Workshop on Scholarly Document Processing_ , pages 130–133, Online. Association for Computational Linguistics.
  * Narayan et al. (2016) Shashi Narayan, Siva Reddy, and Shay B. Cohen. 2016.  Paraphrase generation from latent-variable PCFGs for semantic parsing.  In _Proceedings of the 9th International Natural Language Generation conference_ , pages 153–162, Edinburgh, UK. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.
  * Wang et al. (2021a) Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021a.  KEPLER: A unified model for knowledge embedding and pre-trained language representation.  _Transactions of the Association for Computational Linguistics_ , 9:176–194.
  * Wang et al. (2021b) Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021b.  KEPLER: A unified model for knowledge embedding and pre-trained language representation.  _Transactions of the Association for Computational Linguistics_ , 9:176–194.
  * Zhao et al. (2022) Zheng Zhao, Yftah Ziser, and Shay B Cohen. 2022.  Understanding domain learning in language models through subpopulation analysis.  In _Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_.

##  Appendix A Instructions for Generating Questions

"Here is a list of knowledge graph query programs in JSON format, each with
its corresponding question in English. Acronyms are expanded with the
following dictionary: ["GEO": "Geostationary Orbit", "IGO":"Inclined
Geosynchronous Orbit", "EGO":"Extended Geostationary Orbit", "NSO":"Navigation
Satellites Orbit", "GTO":"GEO Transfer Orbit", "MEO":"Medium Earth Orbit",
"GHO":"GEO-superGEO Crossing Orbits", "LEO":"Low Earth Orbit", "HAO":"High
Altitude Earth Orbit", "MGO":"MEO-GEO Crossing Orbits", "HEO":"Highly
Eccentric Earth Orbit", "LMO":"LEO-MEO Crossing Orbits", "UFO":"Undefined
Orbit","ESO":"Escape Orbits"] Program: {program} Question: {question}
(Repeated until maximum prompt limit is reached)

##  Appendix B Domain-specific Language Models

#### CosmicRoBERTa

A domain-specific language model from the SpaceTransformer family trained with
basic masked-language-modelling on a corpus consisting of 75M words Berquand
et al. (2021).888https://huggingface.co/icelab/cosmicroberta

#### Kepler

We trained our own domain-specific language model by appending the KB pre-
training objective as described by Wang et al. (2021b). The in-domain text
data for the Knowledge-augmented LM was specifically mined to be closely
related to the topic of DISCOS. To achieve this, we collected scientific
papers about space debris and documents from ESA about their internal mission
operation procedures. Additionally, we added the online available dataset from
SpaceTransformers999https://pureportal.strath.ac.uk/en/datasets/dataset-of-
space-systems-corpora-thesis-data to the training data.

The complete dataset comprises approximately 17.6 million words, with around
70% used for training and the remainder for validation. To predict triples, we
converted our database into a set of (head, relation, tail) triples, where
head and tail are entities represented by their English name or description,
and relation is a relation represented by a unique token for the relation
type.

The KG triplet datasets consist of approximately 640,000 triples, which are
split into 636,000 for training, 2,000 for validation, and 2,000 for testing.
These triples represent approximately 59,000 entities and 32 relations. We
trained our model using the available Kepler
implementation.101010https://github.com/THU-KEG/KEPLER

##  Appendix C Pipeline Components

Figure 2: Overview over components in query processing pipeline.

##  Appendix D Example Augmented Programs and Generated Questions

⬇ [{"function":"FindAll", "inputs":[], "dependencies":[]},
{"function":"FilterConcept", "inputs":["LMO"], "dependencies":[0]},
{"function":"Relate", "inputs":["orbit"], "dependencies":[1]},
{"function":"FilterNum", "inputs":["depth", "0.3", "="], "dependencies":[2]},
{"function":"FilterConcept", "inputs":["UnknownObjClass"],
"dependencies":[3]}, {"function":"Count", "inputs":[], "dependencies":[4]}]
Question: How many objects are there that are currently in the LEO-MEO
crossing orbits and are classified as unknown and have a depth of 0.3 m?

⬇ {"function":"Find", "inputs":["State Remote Sensing Center"],
"dependencies":[]}, {"function":"Relate", "inputs":["host_country"],
"dependencies":[0]}, {"function":"FilterConcept", "inputs":["Entity"],
"dependencies":[1]}, {"function":"What", "inputs":[], "dependencies":[2]}]
Question: "Which operators are based in the host country of the State Remote
Sensing Center?"

⬇ {"function":"FindAll", "inputs":[], "dependencies":[]},
{"function":"FilterDate", "inputs":["epoch", "2022-04-08", "="],
"dependencies":[0]}, {"function": "FilterConcept", "inputs":["Launch"],
"dependencies":[1]}, {"function": "Count", "inputs":[], "dependencies":[2]}]
Question: "How many launches are planned for 8th of April
2022?"111111Interestingly the generated question implies that 8th of April
2022 lies in the future, which is in accordance with OpenAI’s training set
cut-off in September 2021

##  Appendix E Interface for Data Collection and Access to System

For implementing a simple user interface, the popular Python library Streamlit
was used. Besides providing the input query the user can also provide
feedback, which potentially could be used to improve the model by identifying
right or wrong answers and adding them to the training set. Another button
allows the user to generate an answer for a question from the validation
dataset to get a better feeling of what questions could be answered.

##  Appendix F Validation Loss Curves

Figure 3: Validation loss and Accuracy over number of training batches for
each component and each tested model.


# Tracking Typological Traits of Uralic Languages
in Distributed Language Representations

Johannes Bjerva
Department of Computer Science
University of Copenhagen
Denmark
<EMAIL_ADDRESS>
Isabelle Augenstein
Department of Computer Science
University of Copenhagen
Denmark
<EMAIL_ADDRESS>

###### Abstract

Although linguistic typology has a long history, computational approaches have
only recently gained popularity. The use of distributed representations in
computational linguistics has also become increasingly popular. A recent
development is to learn distributed representations of language, such that
typologically similar languages are spatially close to one another. Although
empirical successes have been shown for such language representations, they
have not been subjected to much typological probing. In this paper, we first
look at whether this type of language representations are empirically useful
for model transfer between Uralic languages in deep neural networks. We then
investigate which typological features are encoded in these representations by
attempting to predict features in the World Atlas of Language Structures, at
various stages of fine-tuning of the representations. We focus on Uralic
languages, and find that some typological traits can be automatically inferred
with accuracies well above a strong baseline.

## Tiivistelmä

Vaikka kielitypologialla on pitkä historia, laskentamenetelmät ovat vasta
viime aikoina saavuttaneet suosiota. Myös hajautettujen esitysten käyttö
laskennallisessa kielitieteessä on tullut suositummaksi. Viimeaikainen kehitys
on hajautetun kieliedustuksen oppiminen, kuten että samanlaiset kielet ovat
lähellä toisiaan. Vaikka empiirisiä tuloksia onkin saavutettu, ei niille ole
tehty paljoakaan typologista tutkimusta. Tässä artikkelissa tutkitaan ensin,
ovatko tämänlaiset kieliedustukset empiirisesti käyttökelpoisia, kun kyseessä
on uralilaisten kielten ”model transfer” syvissä neuroverkoissa. Tutkimme
myös, mitä typologisia piirteitä voimme löytää kieliedustuksissa, yrittämällä
ennustaa ominaisuuksia jotka saamme World Atlas of Language Structures:n
kautta. Keskitymme uralilaisiin kieliin ja löydämme, että jotkin typologiset
piirteet voidaan automaattisesti päätellä selvästi vahvan perustason
yläpuolelle.

##  1 Introduction

For more than two and a half centuries, linguistic typologists have studied
languages with respect to their structural and functional properties, thereby
implicitly classifying languages as being more or less similar to one another,
by virtue of such properties (Haspelmath, 2001; Velupillai, 2012). Although
typology has a long history (Herder, 1772; Gabelentz, 1891; Greenberg, 1960,
1974; Dahl, 1985; Comrie, 1989; Haspelmath, 2001; Croft, 2002), computational
approaches have only recently gained popularity (Dunn et al., 2011; Wälchli,
2014; Östling, 2015; Bjerva and Börstell, 2016; Deri and Knight, 2016;
Cotterell and Eisner, 2017; Peters et al., 2017; Asgari and Schütze, 2017;
Malaviya et al., 2017). One part of traditional typological research can be
seen as assigning sparse explicit feature vectors to languages, for instance
manually encoded in databases such as the World Atlas of Language Structures
(WALS, Dryer and Haspelmath, 2013). A recent development which can be seen as
analogous to this, is the process of learning distributed language
representations in the form of dense real-valued vectors, often referred to as
language embeddings (Tsvetkov et al., 2016; Östling and Tiedemann, 2017;
Malaviya et al., 2017). These language embeddings encode typological
properties of language, reminiscent of the sparse features in WALS, or even of
parameters in Chomsky’s Principles and Parameters framework (Chomsky, 1993;
Chomsky and Lasnik, 1993; Chomsky, 2014).

In this paper, we investigate the usefulness of explicitly modelling
similarities between languages in deep neural networks using language
embeddings. To do so, we view NLP tasks for multiple Uralic languages as
different aspects of the same problem and model them in one model using
multilingual transfer in a multi-task learning model. Multilingual models
frequently follow a hard parameter sharing regime, where all hidden layers of
a neural network are shared between languages, with the language either being
implicitly coded in the input string (Johnson et al., 2017), given as a
language ID in a one-hot encoding (Ammar et al., 2016), or as a language
embedding (Östling and Tiedemann, 2017). In this paper, we both explore
multilingual modelling of Uralic languages, and probe the language embeddings
obtained from such modelling in order to gain novel insights about typological
traits of Uralic languages. We aim to answer the following three research
questions (RQs).

  1. RQ 1

To what extent is model transfer between Uralic languages for PoS tagging
mutually beneficial?

  2. RQ 2

Are distributed language representations useful for model transfer between
Uralic languages?

  3. RQ 3

Can we observe any explicit typological properties encoded in these
distributed language representations when considering Uralic languages?

##  2 Data

###  2.1 Distributed language representations

There are several methods for obtaining distributed language representations
by training a recurrent neural language model (Mikolov et al., 2010)
simultaneously for different languages (Tsvetkov et al., 2016; Östling and
Tiedemann, 2017). In these recurrent multilingual language models with long
short-term memory cells (LSTM, Hochreiter and Schmidhuber, 1997), languages
are embedded into a $n$-dimensional space. In order for multilingual parameter
sharing to be successful in this setting, the neural network is encouraged to
use the language embeddings to encode features of language. Other work has
explored learning language embeddings in the context of neural machine
translation (Malaviya et al., 2017). In this work, we explore the embeddings
trained by Östling and Tiedemann (2017), both in their original state, and by
further tuning them for PoS tagging.

###  2.2 Part-of-speech tagging

We use PoS annotations from version 2 of the Universal Dependencies (Nivre et
al., 2016). We focus on the four Uralic languages present in the UD, namely
Finnish (based on the Turku Dependency Treebank, Pyysalo et al., 2015),
Estonian (Muischnek et al., 2016), Hungarian (based on the Hungarian
Dependency Treebank, Vincze et al., 2010), and North Sámi (Sheyanova and
Tyers, 2017). As we are mainly interested in observing the language
embeddings, we down-sample all training sets to 1500 sentences (approximate
number of sentences in the Hungarian data), so as to minimise any size-based
effects.

###  2.3 Typological data

In the experiments for RQ3, we attempt to predict typological features. We
extract the features we aim to predict from WALS (Dryer and Haspelmath, 2013).
We consider features which are encoded for all four Uralic languages in our
sample.

##  3 Method and experiments

We approach the task of PoS tagging using a fairly standard bi-directional
LSTM architecture, based on Plank et al. (2016). The system is implemented
using DyNet (Neubig et al., 2017). We train using the Adam optimisation
algorithm (Kingma and Ba, 2014) over a maximum of 10 epochs, using early
stopping. We make two modifications to the bi-LSTM architecture of Plank et
al. (2016). First of all, we do not use any atomic embedded word
representations, but rather use only character-based word representations.
This choice was made so as to encourage the model not to rely on language-
specific vocabulary. Additionally, we concatenate a pre-trained language
embedding to each word representation. That is to say, in the original bi-LSTM
formulation of Plank et al. (2016), each word $w$ is represented as
$\vec{w}+LSTM_{c}(w)$, where $\vec{w}$ is an embedded word representation, and
$LSTM_{c}(w)$ is the final states of a character bi-LSTM running over the
characters in a word. In our formulation, each word $w$ in language $l$ is
represented as $LSTM_{c}(w)+\vec{l}$, where $LSTM_{c}(w)$ is defined as
before, and $\vec{l}$ is an embedded language representation. We use a two-
layer deep bi-LSTM, with 100 units in each layer. The character embeddings
used also have 100 dimensions. We update the language representations,
$\vec{l}$, during training. The language representations are 64-dimensional,
and are initialised using the language embeddings from Östling and Tiedemann
(2017). All PoS tagging results reported are the average of five runs, each
with different initialisation seeds, so as to minimise random effects in our
results.

###  3.1 Model transfer between Uralic languages

The aim of these experiments is to provide insight into RQ 1 and RQ 2. We
first train a monolingual model for each of the four Uralic languages. This
model is then evaluated on all four languages, to investigate how successful
model transfer between pairs of languages is. Results are shown in Figure 1.
Comparing results within each language shows that transfer between Finnish and
Estonian is the most successful. This can be expected considering that these
are the two most closely related languages in the sample, as both are Finnic
languages. Model transfer both to and from the more distantly related
languages Hungarian and North Sámi is less successful. There is little-to-no
difference in this monolingual condition with respect to whether or not
language embeddings are used. As a baseline, we include transfer results when
training on Spanish, which we consider a proxy of a distantly related
languages. Transferring from Spanish is significantly worse ($p<0.05$) than
transferring from a Uralic language in all settings.

Figure 1: Monolingual PoS training. The x-axes denote the training languages,
and the y-axes denote the PoS tagging accuracy on the test language at hand.

Next, we train a bilingual model for each Uralic language. Each model is
trained on the target language in addition to one other Uralic language.
Results are shown in Figure 2. Again, transfer between the two Finnic
languages is the most successful. Here we can also observe a strong effect of
whether or not language embeddings are incorporated in the neural
architecture. Including language embeddings allows for both of the Finnic
languages to benefit significantly ($p<0.05$) from the transfer setting, as
compared to the monolingual setting. No significant differences are observed
for other language pairs.

Figure 2: Bilingual PoS training. The x-axes denote the added training
languages (in addition to the target language), and the y-axes denote the PoS
tagging accuracy on the test language at hand.

###  3.2 Predicting typological features with language embeddings

Having observed that language embeddings are beneficial for model transfer
between Uralic languages, we turn to the typological experiments probing these
embeddings. The aim of these experiments is to provide insight into RQ 3. We
investigate typological features from WALS (Dryer and Haspelmath, 2013),
focussing on those which have been encoded for the languages included in the
UD.

We first train the same neural network architecture as for the previous
experiments on all languages in UD version 2. Observing the language
embeddings from various epochs of training permits tracking the typological
traits encoded in the distributed language representations as they are fine-
tuned. In order to answer the research question, we train a simple linear
classifier to predict typological traits based on the embeddings. Concretely,
we train a logistic regression model, which takes as input a language
embedding $\vec{l}_{e}$ from a given epoch of training, $e$, and outputs the
typological class a language belongs to (as coded in WALS). When $e$ is 0,
this indicates the pre-trained language embeddings as obtained from Östling
and Tiedemann (2017). Increasing $e$ indicates the number of epochs of PoS
tagging during which the language embedding has been updated. All results are
the mean of three-fold cross-validation. We are mainly interested in observing
two things: i) Which typological traits do language embeddings encode?; ii) To
what extent can we track the changes in these language embeddings over the
course of fine-tuning for the task of PoS tagging?.

We train the neural network model over five epochs, and investigate
differences of classification accuracies of typological properties as compared
to pre-trained embeddings. A baseline reference is also included, which is
defined as the most frequently occurring typological trait within each
category. In these experiments, we disregard typological categories which are
rare in the observed sample (i.e. of which we have one or zero examples).
Looking at classification accuracy of WALS features, we can see four emerging
patterns:

  1. 1.

The feature is pre-encoded;

  2. 2.

The feature is encoded by fine-tuning;

  3. 3.

The feature is not pre-encoded;

  4. 4.

The feature encoding is lost by fine-tuning.

One example per category is given in Figure 3. Two features based on word-
ordering can be seen as belonging in the categories of features which are
either pre-encoded or which become encoded during training. The fine-tuned
embeddings do not encode the feature for whether pronominal subjects are
expressed, or the feature for whether a predicate nominal has a zero copula.

Figure 3: Predicting typological features in WALS. The x-axes denote number of
epochs the language embeddings have been fine-tuned for. The y-axe denotes
classification accuracy for the typological feature at hand.

####  3.2.1 Predicting Uralic typological features

Finally, we attempt to predict typological features for the four Uralic
languages included in our sample, as shown in Figure 4. Similarly to the
larger language sample in Figure 3, the Uralic language embeddings also both
gain typological information in some respects, and lose information in other
respects. For instance, the pre-trained embeddings are not able to predict
ordering of adpositions and noun phrase in the Uralic languages, whereas
training on PoS tagging for two epochs adds this information.

Figure 4: Predicting typological features in Uralic languages. The x-axes
denote number of epochs the language embeddings have been fine-tuned for. The
y-axes denote classification accuracy for the typological feature at hand.

##  4 Discussion

###  4.1 Language embeddings for Uralic model transfer

In the monolingual transfer setting, we observed that transferring from more
closely-related languages was relatively beneficial. This is expected, as the
more similar two languages are, the easier it ought to be for the model to
directly apply what it learns from one language to the other. Concretely, we
observed that transferring between the two Finnic languages in our sample,
Finnish and Estonian, worked relatively well. We further observed that
including language embeddings in this setting had little-to-no effect on the
results. This can be explained by the fact that the language embedding used is
the same throughout the training phase, as only one language is used, hence
the network likely uses this embedding to a very low extent.

In bilingual settings, omitting the language embeddings results in a severe
drop in tagging accuracy in most cases. This is likely because that treating
our sample of languages as being the same language introduces a large amount
of confusion into the model. This is further corroborated by the fact that
treating the two Finnic languages in this manner results in a relatively small
drop in accuracy.

Including language embeddings allows for the model transfer setting to be
beneficial for the more closely related languages. This bodes well for the
low-resource case of many Uralic languages in particular, and possibly for
low-resource NLP in general. In the cases of the more distantly related
language pairings, including language embeddings does not result in any
significant drop in accuracy. This indicates that using language embeddings at
least allows for learning a more compact model without any significant losses
to performance.

###  4.2 Language embeddings for Uralic typology

Interestingly, the language embeddings are not only a manner for the neural
network to identify which language it is dealing with, but are also used to
encode language similarities and typological features. To contrast, the neural
network could have learned something akin to a one-hot encoding of each
language, in which case the languages could easily have been told apart, but
classification of typological features would have been constantly at baseline
level.

Another interesting finding is the fact that we can track the typological
traits in the distributed language representations as they are fine-tuned for
the task at hand. This has the potential to yield insight on two levels, of
interest both to the more engineering-oriented NLP community, as well as the
more linguistically oriented CL community. A more in-depth analysis of these
embeddings can both show what a neural network is learning to model, in
particular. Additionally, these embeddings can be used to glean novel insights
and answer typological research questions for languages which, e.g., do not
have certain features encoded in WALS.

In the specific case of Uralic languages, as considered in this paper, the
typological insights we gained are, necessarily, ones that are already known
for these languages. This is due to the fact that we simply evaluated our
method on the features present for the Uralic languages in WALS. It is
nonetheless encouraging for this line of research that we, e.g., could predict
WALS feature 86A (Order of Genitive and Noun) based solely on these
embeddings, and training a very simple classifier on a sample consisting
exclusively of non-Uralic languages.

##  5 Conclusions and future work

We investigated model transfer between the four Uralic languages Finnish,
Estonian, Hungarian and North Sámi, in PoS tagging, focussing on the effects
of using language embeddings. We found that model transfer is successful
between these languages, with the main benefits found between the two Finnic
languages (Finnish and Estonian), when using language embeddings. We then
turned to an investigation of the typological features encoded in the language
embeddings, and found that certain features are encoded. Furthermore, we found
that the typological features encoded change when fine-tuning the embeddings.
In future work, we will look more closely at how the encoding of typological
traits in distributed language representations changes depending on the task
on which they are trained.

## Acknowledgements

The authors would like to thank Iina Alho for help with translating the
abstract to Finnish. We would also like to thank Robert Östling for giving us
access to pre-trained language embeddings.

## References

  * Ammar et al. (2016) Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah Smith. 2016\.  Many languages, one parser.  Transactions of the Association of Computational Linguistics 4:431–444.  https://aclanthology.info/pdf/Q/Q16/Q16-1031.pdf.
  * Asgari and Schütze (2017) Ehsaneddin Asgari and Hinrich Schütze. 2017.  Past, present, future: A computational investigation of the typology of tense in 1000 languages.  In EMNLP. Association for Computational Linguistics, pages 113–124.
  * Bjerva and Börstell (2016) Johannes Bjerva and Carl Börstell. 2016.  Morphological Complexity Influences Verb-Object Order in Swedish Sign Language.  In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC). The COLING 2016 Organizing Committee, pages 137–141.
  * Chomsky (1993) Noam Chomsky. 1993.  Lectures on government and binding: The Pisa lectures.  9\. Walter de Gruyter.
  * Chomsky (2014) Noam Chomsky. 2014.  The minimalist program.  MIT press.
  * Chomsky and Lasnik (1993) Noam Chomsky and Howard Lasnik. 1993.  The theory of principles and parameters.
  * Comrie (1989) Bernard Comrie. 1989.  Language universals and linguistic typology: Syntax and morphology.  University of Chicago press.
  * Cotterell and Eisner (2017) Ryan Cotterell and Jason Eisner. 2017.  Probabilistic typology: Deep generative models of vowel inventories.  In ACL. Association for Computational Linguistics, pages 1182–1192.
  * Croft (2002) William Croft. 2002.  Typology and universals.  Cambridge University Press.
  * Dahl (1985) Östen Dahl. 1985.  Tense and Aspect Systems.  Basil Blackwell Ltd., NewYork.
  * Deri and Knight (2016) Aliya Deri and Kevin Knight. 2016.  Grapheme-to-phoneme models for (almost) any language.  In ACL. Association for Computational Linguistics, pages 399–408.
  * Dryer and Haspelmath (2013) Matthew S. Dryer and Martin Haspelmath, editors. 2013.  WALS Online.  Max Planck Institute for Evolutionary Anthropology, Leipzig.  http://wals.info/.
  * Dunn et al. (2011) Michael Dunn, Simon J Greenhill, Stephen C Levinson, and Russell D Gray. 2011.  Evolved structure of language shows lineage-specific trends in word-order universals.  Nature 473(7345):79–82.
  * Gabelentz (1891) Georg von der Gabelentz. 1891.  Die Sprachwissenschaft, ihre Aufgaben, Methoden und bisherigen Ergebnisse.  Leipzig.
  * Greenberg (1974) Joseph Greenberg. 1974.  Language typology: A historical and analytic overview, volume 184\.  Walter de Gruyter.
  * Greenberg (1960) Joseph H Greenberg. 1960.  A quantitative approach to the morphological typology of language.  International journal of American linguistics 26(3):178–194.
  * Haspelmath (2001) Martin Haspelmath. 2001.  Language typology and language universals: An international handbook, volume 20.  Walter de Gruyter.
  * Herder (1772) J. Herder. 1772.  Abhandlung über den Ursprung der Sprache.  Berlin: Christian Friedrich Voß.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  Neural computation 9(8):1735–1780.
  * Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017.  Google’s multilingual neural machine translation system: Enabling zero-shot translation.  Transactions of the Association of Computational Linguistics 5:339–351.  http://aclweb.org/anthology/Q17-1024.
  * Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980 .
  * Malaviya et al. (2017) Chaitanya Malaviya, Graham Neubig, and Patrick Littell. 2017.  Learning language representations for typology prediction.  In EMNLP. Association for Computational Linguistics, pages 2519–2525.  http://aclweb.org/anthology/D17-1267.
  * Mikolov et al. (2010) Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010.  Recurrent neural network based language model.  In Interspeech. volume 2, page 3.
  * Muischnek et al. (2016) Kadri Muischnek, Kaili Müürisep, and Tiina Puolakainen. 2016.  Estonian dependency treebank: from constraint grammar tagset to universal dependencies.  In LREC.
  * Neubig et al. (2017) Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017.  Dynet: The dynamic neural network toolkit.  arXiv preprint arXiv:1701.03980 .
  * Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016.  Universal dependencies v1: A multilingual treebank collection.  In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016).
  * Östling (2015) Robert Östling. 2015.  Word order typology through multilingual word alignment.  In The 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. pages 205–211.
  * Östling and Tiedemann (2017) Robert Östling and Jörg Tiedemann. 2017.  Continuous multilinguality with language vectors.  In EACL. Association for Computational Linguistics, pages 644–649.  http://aclanthology.coli.uni-saarland.de/pdf/E/E17/E17-2102.pdf.
  * Peters et al. (2017) Ben Peters, Jon Dehdari, and Josef van Genabith. 2017.  Massively multilingual neural grapheme-to-phoneme conversion.  In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems. pages 19–26.
  * Plank et al. (2016) Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016.  Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss.  In ACL. Association for Computational Linguistics, pages 412–418.  https://doi.org/10.18653/v1/P16-2067.
  * Pyysalo et al. (2015) Sampo Pyysalo, Jenna Kanerva, Anna Missilä, Veronika Laippala, and Filip Ginter. 2015.  Universal dependencies for finnish.  In NoDaLiDa. Linköping University Electronic Press, 109, pages 163–172.
  * Sheyanova and Tyers (2017) Mariya Sheyanova and Francis M. Tyers. 2017.  Annotation schemes in north sámi dependency parsing.  In Proceedings of the 3rd International Workshop for Computational Linguistics of Uralic Languages. pages 66–75.
  * Tsvetkov et al. (2016) Yulia Tsvetkov, Sunayana Sitaram, Manaal Faruqui, Guillaume Lample, Patrick Littell, David Mortensen, Alan W Black, Lori Levin, and Chris Dyer. 2016.  Polyglot neural language models: A case study in cross-lingual phonetic representation learning.  In NAACL-HLT. Association for Computational Linguistics, pages 1357–1366.  https://doi.org/10.18653/v1/N16-1161.
  * Velupillai (2012) Viveka Velupillai. 2012.  An introduction to linguistic typology.  John Benjamins Publishing.
  * Vincze et al. (2010) Veronika Vincze, Dóra Szauter, Attila Almási, György Móra, Zoltán Alexin, and János Csirik. 2010.  Hungarian dependency treebank.  In LREC.
  * Wälchli (2014) Bernhard Wälchli. 2014.  Algorithmic typology and going from known to similar unknown categories within and across languages.  Aggregating Dialectology, Typology, and Register Analysis: Linguistic Variation in Text and Speech 28:355.