# LLM-Assisted Rule Based Machine Translation for Low/No-Resource Languages

Jared Coleman
Bhaskar Krishnamachari
Khalil Iskarous
University of Southern California
<EMAIL_ADDRESS>
&Ruben Rosales
<EMAIL_ADDRESS>
###### Abstract

We propose a new paradigm for machine translation that is particularly useful
for no-resource languages (those without any publicly available bilingual or
monolingual corpora): LLM-RBMT (LLM-Assisted Rule Based Machine Translation).
Using the LLM-RBMT paradigm, we design the first language
education/revitalization-oriented machine translator for Owens Valley Paiute
(OVP), a critically endangered Indigenous American language for which there is
virtually no publicly available data. We present a detailed evaluation of the
translator’s components: a rule-based sentence builder, an OVP to English
translator, and an English to OVP translator. We also discuss the potential of
the paradigm, its limitations, and the many avenues for future research that
it opens up.

LLM-Assisted Rule Based Machine Translation for Low/No-Resource Languages

Jared Coleman Bhaskar Krishnamachari Khalil Iskarous University of Southern
California<EMAIL_ADDRESS>Ruben Rosales
<EMAIL_ADDRESS>

##  1 Introduction

Large Language Models like OpenAI’s GPT series OpenAI (2023) have shown
remarkable capability at an impressively wide range of natural language tasks
Bubeck et al. (2023) including machine translation Hendy et al. (2023). These
models work because they are trained on vast amounts of natural language data,
primarily from the internet OpenAI (2023). For languages that are low-resource
(languages for which there is little publicly available data) or no-resource
(languages for which there is no publicly available data), models like these
do not perform well on their own Chowdhery et al. (2022); Robinson et al.
(2023). There have been many efforts in improving machine translation for low-
resource languages (see Ranathunga et al. (2023) for a comprehensive survey),
but no-resource languages have received much less attention in the literature.

Over past decades, researchers and community leaders have led many remarkable
efforts in endangered language revitalization Coronel-Molina and McCarty
(2016); Taylor and Kochem (2022) and reclamation Baird (2016). In this paper,
we begin to explore how the impressive general-purpose language skills of LLMs
might be helpful for these kinds of efforts by introducing a new paradigm for
low/no-resource machine translation: LLM-RBMT (LLM-Assisted Rule-Based Machine
Translation). The intuition behind our approach is that, given the correct
tools, humans are good at translating simple sentences even for languages they
do not know. For example, a common task for students in a language class is to
translate sentences given a set of words and rules. Given a conjugation table
and a set of verbs a motivated student could probably translate a list of
simple subject-verb (e.g., “I eat”, “you sing”, etc.) sentences with extremely
high accuracy. Of course, they are limited to only translating specific kinds
of sentences with a limited vocabulary. Still, the idea is interesting: if we
provide enough context to an LLM like ChatGPT, which has been shown to exhibit
human-level performance on many natural language tasks, we may not need it to
actually know (exhibit general fluency in) the target language we want to
translate to/from. While rule-based translation can likely never achieve the
quality of modern ML translators for high-resource languages, this technique
has a few potential advantages for low/no-resource languages. First and most
importantly, it requires no parallel corpus. Also, it is amenable to partial
translation. Humans (and, as we’ll show in this paper, LLMs) are capable of
giving partial translations when they don’t have all of the necessary
vocabulary. For example, a Spanish student that doesn’t know the word for
“dog” might still be able to translate the rest of the sentence “the dog ate
the apple yesterday” by saying “el [dog] comió la manzana ayer”.

In this paper, we leverage LLMs (specifically, gpt-3.5-turbo and gpt-4 from
OpenAI’s GPT series) to break down natural language sentences into structured
simple sentences compatible with hand-crafted rule-based translators. We also
use them to turn sentence structured information (in JSON format) into natural
language sentences. Using this approach, the LLM never interacts directly with
the target language. Rather, we rely on the LLM to tell us how to use the
simple, rule-based translators to provide a translation as close as possible
to the user’s original input. We use this technique to build and evaluate two
machine translation tools for Owens Valley Paiute (OVP, also called Eastern
Mono or Monache in linguistics literature, ISO 639-3 mnr SIL International
(2024)), a critically endangered Indigenous American language in the Uto-
Aztecan language family Moseley (2010). The first is a selection-based OVP to
English simple sentence translator and the second is an English to OVP
translator that uses available vocabulary to construct translations of
arbitrary user input sentences. The translators are oriented toward language
teaching and revitalization. They are not designed to be general-purpose
translators, but rather as tools to help no-resource language learners express
ideas using simple sentence constructions.

###  1.1 Contributions

The main contributions of this work are111The code behind all contributions is
open-source: https://github.com/kubishi/kubishi_sentences:

  1. 1.

An extensible OVP sentence builder for constructing valid subject-verb and
subject-verb-object sentences in Owens Valley Paiute.

  2. 2.

An LLM-assisted OVP to English translator that translates OVP sentence builder
sentences to English with high accuracy.

  3. 3.

An LLM-assisted English to OVP translator that translates English sentences to
Owens Valley Paiute using the sentence-builder and OVP to English translation
tools.

  4. 4.

A novel methodology for the design and evaluation of no-resource language
translators.

The rest of this paper is organized as follows. We discuss related work in
Section 2. We present the sentence building tool and OVP to English
translation system in Section 3. Then, in Section 4, we present the English to
OVP translation system and report results on translation quality for different
types of input sentences using embeddings models to measure semantic
similarity. We conclude the paper with a summary of contributions and
discussion of future research directions in Section 5.

##  2 Related Work

The landscape of low-resource machine translation is vast and constantly
growing. A comprehensive survey on this subject is provided by Ranathunga et
al. (2023), which outlines the current techniques, guidance for selecting
which techniques to use for a given language, and future directions for
research. Of particular interest within this survey is the examination of
unsupervised neural machine translation. While there exists promising research
on constructing translators from minimal corpora, these methods invariably
require some natural language data and thus have limited applicability to no-
resource languages (such as OVP). The survey also discusses data augmentation
strategies, including word or phrase replacement-based augmentation and Back-
Translation-based Data Augmentation, both of which could potentially be
integrated with some of the solutions presented in this paper (in particular
the sentence builder and OVP to English translator to be presented in Section
3). Such an exploration is an interesting topic for future work. Other
approaches discussed in the survey such as supervised learning, transfer
learning, and semi-supervised learning are inapplicable to our scenario due to
the absence of bilingual or monolingual corpora.

Contrary to the prevailing assumption in the literature that rule-based
machine translation (RBMT) is a relic of the past, there remains active
research and development in RBMT systems tailored for the most under resourced
of languages Khanna et al. (2021); Pirinen (2019); Torregrosa et al. (2019).
Recent work has also explored the utilization of Large Language Models (LLMs)
for enhancing translation capabilities in low-resource languages through fine-
tuning techniques Lankford et al. (2023). Although this approach has shown
promise in improving LLM translation quality for low-resource languages like
(e.g., Irish), its reliance on bilingual corpora make it infeasible for no-
resource languages like OVP. Recently, semantic similarity has been used to
evaluate the quality of Machine Translation systems Cao et al. (2022); Song et
al. (2021). We observe that this technique is particularly useful for
evaluating the quality of the English to OVP translator presented in this
paper, due to the lack of parallel corpora. See Muennighoff et al. (2023) for
a comprehensive benchmark of different embeddings models used for computing
semantic similarities.

##  3 OVP to English Translation

In this section, we present an LLM-assisted selection-based OVP to English
translator. The first piece of this translation system is a selection-based
OVP sentence builder. The sentence builder allows users to select each of the
following parts of speech from a list of choices until they form a valid
sentence:

  * •

Subject: The subject of the sentence.

  * •

Subject Suffix: In OVP, noun subjects are always suffixed with either -ii (if
the subject is proximal to the speaker) or -uu (if the subject is distant to
the speaker).

  * •

Verb: the verb of the sentence.

  * •

Verb Suffix: In OVP, verb suffixes express the tense/aspect of the verb.

  * •

Object: The object of the sentence (disallowed if the selected verb is
intransitive, optional otherwise).

  * •

Object Suffix: In OVP, noun objects are always suffixed with either -(n)eika
(if the subject is proximal to the speaker) or -(n)oka (if the subject is
distant to the speaker).

  * •

Verb Object Pronoun Prefix: In OVP, object pronouns are prefixed to the verbs
they are the object of. Object pronouns are required for all transitive verbs.
Even when a noun object is specified, the pronoun is still required and should
match the object suffix (-(n)eika matches a- or ma- and -(n)oka matches u-).

Not all of these parts of speech are required to create a valid sentence in
OVP, though. In fact, some of them are incompatible. For example, if an
intransitive verb is chosen, then it cannot have an object or object pronoun.
In other words, the valid choices a user can make is a function of the choices
they have already made. In our python implementation for this translator, we
process each user selection and change the list of valid options for each
part-of-speech based on their current selections to ensure they always create
a valid OVP sentence. This is the rule part of LLM-Assisted Rule-Based Machine
Translation that requires expert knowledge in the target language to
implement. The entire vocabulary available for the users to select among fits
on a single page (and can be found in Appendix C).

After the user creates a valid OVP sentence, we translate it by first encoding
the following sentence information into an English-only (using vocabulary
definitions) structured simple sentence:

  * •

Subject: noun or pronoun subject in English

  * •

Subject Proximity: proximal if user selected subject suffix -ii or distant if
the user selected subject suffix -uu

  * •

Object: noun or pronoun object in English

  * •

Object Proximity: proximal if user selected object suffix -(n)eika or distant
if the user selected object suffix -(n)oka

  * •

Verb: verb in English

  * •

Verb Tense/Aspect: one of past, present, future, past-continuous, present-
continuous, present-perfect

Then, we use few-shot prompting to encourage an LLM to transform the
structured English data into a natural language sentence. Consider the example
in Figure 1 where few-shot training examples (colored black) tell the LLM how
to respond to the actual structured data for the randomly generated sentence
for “Wo’ada-ii pagwi-noka u-zawa-dü.” (colored blue). Observe, the LLM is
prompted to translate using only the English, structured version of the
selected sentence.

Wo’ada-ii pagwi-noka u-zawa-dü. system You are an assistant for translating
structured sentences into simple natural English sentences. user
[{‘part_of_speech’: ‘subject’, ‘positional’: ‘proximal’, ‘word’: ‘wood’},
{‘part_of_speech’: ‘object’, ‘positional’: ‘proximal’, ‘word’: ’dog’},
{‘part_of_speech’: ‘verb’, ‘tense’: ‘present continuous (-ing)’, ‘word’:
‘see’}] assistant This wood is seeing this dog. user [{‘part_of_speech’:
‘subject’, ‘positional’: ‘distal’, ‘word’: ‘pinenuts’}, {‘part_of_speech’:
‘object’, ‘positional’: ‘distal’, ‘word’: ‘horse’}, {‘part_of_speech’: ‘verb’,
‘tense’: ‘future (will)’, ‘word’: ‘see’}] assistant Those pinenuts will see
that horse. user [{‘part_of_speech’: ‘subject’, ‘word’: ‘mosquito’,
‘positional’: ‘proximal’}, {‘part_of_speech’: ‘object’, ‘word’: ‘fish’,
‘positional’: ‘distal’}, {‘part_of_speech’: ‘verb’, ‘word’: ‘cook’, ‘tense’:
‘present’}] assistant This mosquito is cooking that fish. Figure 1: Few-shot
examples for translating “Wo’ada-ii pagwi-noka u-zawa-dü.” using
gpt-3.5-turbo.

To evaluate the accuracy of the translator, we generated 100 random valid OVP
sentences by iteratively selecting a random choice among available choices for
each of the parts of speech until the sentence is valid. Of the 100 random
sentences generated, 98 were translated into English accurately using
gpt-3.5-turbo model from OpenAI’s GPT-series. Translations and their accuracy
labels can be found in Appendix A. While impressively accurate, this
translator has many disadvantages. It only works for simple subject-verb and
subject-verb-object sentences that use the nouns and verbs available in the
system. Also, since pronouns and suffixes in OVP encode temporal/spatial
information, translations don’t always capture full meaning of the sentence.
The English translations are correct, but may be missing useful information.
For example "kamü-uu wo’abi-neika a-düka-ti" translates to "the jackrabbit is
eating the worm", which is technically correct, but -uu also indicates that
the jackrabbit is not present and the -neika indicates the worm is present.
Then, since -ti can is used for both the present and past continuous tenses
(is [x]-ing or was [x]-ing), a better translation would be “the jackrabbit was
eating this worm”. More advanced rules and better prompt-engineering may help
mitigate this issue.

Despite some expected disadvantages, this translator has many advantages.
First, it is the first machine translator for OVP. It is also easy to extend
the tool with new nouns and verbs. Also, while implementing the rules required
expert knowledge of what makes an OVP sentence valid, no expert knowledge of
how the rules map to English was required (or needed to be programmed), thanks
to the LLM. Finally, we believe this kind of translation system might be a
useful educational tool that helps students learn how to build simple
sentences. It also may be useful as a data augmentation technique for training
neural machine translation models for low-resource languages.

##  4 English to OVP Translation

In this section, we present the first English to Owens Valley Paiute machine
translator. Unlike the OVP to English translator presented in the previous
section, the user can input any sentence in natural language. The translator
works by first using an LLM to break the input sentence into a set of simple
structured subject-verb and subject-verb-object sentences, discarding any
adjectives, adverbs, prepositions, objects (except for direct objects), etc.
We encourage (through few-shot prompt engineering) the LLM to preserve as much
semantic meaning as possible between the original sentence and the set of
simple sentences. Consider the example in Figure 2 where few-shot examples
(colored black) tell the LLM how to respond to a given input sentence “We are
playing and laughing.” (colored blue)222We also leverage OpenAI’s function
calling capability to ensure that responses are consistently formatted. We
refer interested readers to the open-source implementation’s documentation at
https://github.com/kubishi/kubishi_sentences for full details..

We are playing and laughing. system You are an assistant that splits user
input sentences into a set of simple SVO or SV sentences. The set of simple
sentences should be as semantically equivalent as possible to the user input
sentence. No adjectives, adverbs, prepositions, or conjunctions should be
added to the simple sentences. Indirect objects and objects of prepositions
should not be included in the simple sentences. user I am sitting in a chair.
assistant [{‘subject’: ‘I’, ‘verb’: ‘sit’, ‘verb_tense’: ‘present_continuous’,
‘object’: null}] user I saw two men walking their dogs yesterday at Starbucks
while drinking a cup of coffee assistant [{‘subject’: ‘I’, ‘verb’: ‘see’,
‘verb_tense’: ‘past’, ‘object’: ‘man’}, {‘subject’: ‘man’, ‘verb’: ‘walk’,
‘verb_tense’: ‘past_continuous’, ‘object’: ’dog’}, {‘subject’: ‘man’, ‘verb’:
‘drink’, ‘verb_tense’: ‘past_continuous’, ‘object’: ‘coffee’}] user We are
playing and laughing. assistant [{‘subject’: ‘we’, ‘verb’: ‘play’,
‘verb_tense’: ‘present_continuous’, ‘object’: null}, {‘subject’: ‘we’, ‘verb’:
‘laugh’, ‘verb_tense’: ‘present_continuous’, ‘object’: null}] Figure 2: Few-
shot training examples for the English to OVP using gpt-3.5-turbo.

Then, we use these structured sentences and available vocabulary to build
valid OVP sentences with the sentence-building tool described in Section 3.
Once the sentence is built, we use the translator described in Section 3 to
translate the OVP sentences back into English. The idea is that, while some
meaning may have been lost between the original input sentence and the final
output English sentences, the user can be fairly confident (given the accuracy
of the OVP to English translator) that the final translations are correct. The
entire English to OVP translation process is depicted in Figure 3.

Simple 1Simple 2$\ldots$Simple $n$ptInputOVP 1OVP 2$\ldots$OVP $n$English
1English 2$\ldots$English $n$Sentence SegmenterEnglish to OVP TranslatorOVP to
English Translator Figure 3: The entire English to OVP translation process.
The box with a red, dashed border indicates the set of sentences in Owens
Valley Paiute (the target language) and the box with a blue, dashed border
indicates the set of English sentences they translate to. Ideally, the input
sentence, simple sentences, and English output sentences will have equivalent
or very similar semantic meaning.

We evaluate the system by translating a set of 125 sentences. There are five
types of sentences in the dataset (25 of each):

  * •

subject-verb (e.g., “I read” or “she sings”)

  * •

subject-verb-object (e.g., “Mom made dinner” or “John read a book”)

  * •

two-verb (e.g., “She sings and dances.” or “I ate while watching TV.”)

  * •

two-clause (e.g., “My brother drove and I waited.” or “Harry wrote and Ron
read.”)

  * •

complex (e.g., “Rachel and Monica share an apartment.” or “Romeo and Juliet
loved deeply.”)

We translated all 125 sentences using both the gpt-3.5-turbo and gpt-4 models
from OpenAI’s GPT-series, resulting in a total of 250 translations.

To measure the quality of each translation, we compute the semantic similarity
between the input sentence and:

  * •

the set of simple sentences generated by the LLM-powered segmenter (denoted
simple).

  * •

the set of simple sentences with unknown vocabulary removed (denoted
comparator). Intuitively, this represents the best the translator can achieve
given the vocabulary it has access to. For example, suppose the verb “wash” is
not in our available vocabulary. Then the comparator sentence for the simple
sentence “The woman is washing” would be “The woman is [VERB]-ing”

  * •

the “round-trip” English translation (denoted backwards). This is the sentence
produced using by translating the translated sentence (in OVP) to English
using the method described in Section 3.

The usefulness of these measurements depends greatly on the function used to
compute semantic similarity. We compute semantic similarity by generating
embeddings for sentences (using some embeddings model) and computing the
normalized cosine similarity between these embeddings. For our application, we
want semantically similar sentences to have small normalized cosine
similarity, independent of other linguistic features like syntax. For example,
an ideal semantic similarity function would rank “an apple is eaten by a man”
more similar to “a man eats apples” than the sentence “a woman drinks coffee”,
despite the latter sentence being essentially grammatically equivalent to the
target sentence.

We evaluated seven different embeddings models for this purpose and measured
the semantic similarity between twelve target sentences and a ranked list of
10 sentences for each ranging from most to least semantically similar
(sentences can be found in Appendix B). For each target sentence, we compare
the ground-truth ranking of the 10 sentences to the ranking determined by the
semantic similarity scores yielded by a particular embeddings model. We
measure the similarity between the two rankings using two metrics: average
displacement (average distance between a sentence’s position in the computed
ranking and its position in the target ranking) and RBO (Rank-biased Overlap
Webber et al. (2010)). Table 1 tabulates the results of this evaluation.

|  | Average |
---|---|---|---
|  | Displacement | RBO
Embeddings Model |  | mean | std | mean | std
text-embedding-ada-002 | OpenAI (2024a) | 0.967 | 0.442 | 0.885 | 0.053
all-MiniLM-L6-v2 | Reimers and Gurevych (2020) | 0.933 | 0.323 | 0.884 | 0.050
text-embedding-3-small | OpenAI (2024b) | 1.000 | 0.362 | 0.882 | 0.051
text-embedding-3-large | OpenAI (2024b) | 0.917 | 0.463 | 0.882 | 0.054
paraphrase-MiniLM-L6-v2 | Reimers and Gurevych (2020) | 1.150 | 0.410 | 0.870 | 0.054
bert-base-uncased | Reimers and Gurevych (2019) | 1.600 | 0.703 | 0.777 | 0.100
spacy/en_core_web_md | Explosion (2024) | 1.833 | 0.466 | 0.760 | 0.090
Table 1: Quality of different embeddings models in measuring semantic
similarity between sentences. A lower average displacement and higher RBO
indicate a better embeddings model for this purpose.

Results indicate that the all-MiniLM-L6-v2 embeddings model perform well with
respect to both Average Displacement and RBO. For this reason, we run the rest
of our experiments using this embeddings model for computing the semantic
similarity between sentences.

We computed the semantic similarity between all pairs of sentences in the
dataset to establish a baseline for comparison. The mean semantic similarity
between a pair of unrelated sentences was $\mu\approx 0.574$ with a standard
deviation of $\sigma\approx 0.061$. Furthermore, the distribution appears to
be relatively Gaussian (a histogram can be found in Appendix E). Intuitively,
this suggests that semantically unrelated sentences are very unlikely to have
a semantic similarity score of above $\mu+3\sigma$ (i.e., greater than
$0.757$).

A good translation, then, should score high on all three semantic similarity
metrics. For example, the translation

I am swimming. Simple Semantic Similarity
I am swimming. 1 Comparator I am swimming. 1 Target Nüü pahabi-ti. Backwards I
am swimming. 1 (model: gpt-3.5-turbo)

is perfect. There are other interesting cases too, though. For example, when
the comparator score is low but simple and backwards scores are high, the
translator appears to do well with the vocabulary available, but is only able
to give a partial translation. For example, the translation

Birds will migrate and return. Simple Semantic Similarity
A bird will migrate. The bird will return. 0.955 Comparator A bird will
[VERB]. The bird will [VERB]. 0.778 Target [migrate]-wei tsiipa-uu.
[return]-wei tiipa-uu. Backwards That bird will migrate. That bird will
return. 0.944 (model: gpt-4)

has a high simple score, since the “Birds will migrate and return” is quite
similar (semantically) to “A bird will migrate. The bird will return.”. The
comparator score is much lower, however, since the words for “migrate” and
“return” are not available in the vocabulary. The backwards score is good
because the sentence “[migrate]-wei tsiipa-uu. [return]-wei tiipa-uu.”, when
translated using the OVP to English translator described in Section 3, becomes
“That bird will migrate. That bird will return”, which is almost equivalent to
the simple sentence. This example also highlights an advantage to our
approach. No-resource language learners can use this system to understand how
sentences should be structured even if it doesn’t have all of the vocabulary
necessary to generate the entire translation. Furthermore, users can be fairly
confident that the OVP target sentence is accurately translated into the
English backwards sentence, due to the accuracy of the OVP to English
translator.

The simple score being low indicates that a significant portion of the input
sentence’s meaning is lost during segmentation (due, for example, to the input
sentence containing adverbs, adjectives, prepositions, etc. that don’t fit
anywhere in a simple subject-verb or subject-verb-object sentence). For
example, the translation

My brother and I went hiking. Simple Semantic Similarity
Brother went. I went. 0.794 Comparator [SUBJECT] went. I went. 0.608 Target
mia-ku [brother]-uu. nüü mia-ku. Backwards The brother went. I went. 0.806
(model: gpt-3.5-turbo)

lost meaning in the first step segmenting the input sentence into simple
sentences because it chose to use the verb "to go" instead of "hike" which is
the main topic of the sentence. Perhaps a better way to have segmented this
sentence would be: “Brother hiked. I hiked”. It may be possible to encourage
the LLM to prefer “topic” verbs through prompt engineering.

Another interesting case is when the simple and comparator scores are high and
only the backwards score is lower. This is observed in cases where there is
ambiguity in OVP where there is not in English. For example, in the
translation

She is cooking. Simple Semantic Similarity
She is cooking. 1 Comparator She is cooking. 1 Target Uhu sawa-ti. Backwards
He is cooking. 0.836 (model: gpt-4)

“she” turns to “he” in the backwards translation because OVP does not have
gendered pronouns. Despite the lower backwards score, this translation is
accurate.

In general, both gpt-3.5-turbo and gpt-4 models do well with respect to the
simple and backwards semantic similarity scores. Table 2 summarizes the mean
semantic similarity scores for each model and type of sentence.

|  | Mean
---|---|---
Model | Type | Sim.
gpt-3.5-turbo | subject-verb | 0.941
| two-verb | 0.906
| subject-verb-object | 0.869
| two-clause | 0.879
| complex | 0.829
gpt-4 | subject-verb | 0.941
| subject-verb-object | 0.866
| two-verb | 0.905
| two-clause | 0.877
| complex | 0.830
Table 2: Translation qualities by model and sentence type: mean semantic
similarities between input sentence and the simple, comparator, and backwards
sentences produced during translation.

Figure 4 depicts results for subject-verb sentences. Plots for the rest of the
results can be found in Appendix D.

Figure 4: Results for subject-verb sentences. The dark, medium, and light gray
bands represent the baseline similarity (between unrelated sentences in the
dataset) +/- one, two, and three standard deviations, respectively.

That the simple and backwards scores are consistently higher than the
comparator scores suggests that the translator can be greatly improved simply
by expanding its vocabulary. It is also interesting to note that the cheaper,
weaker model performs quite well333The cost to run generate all translations
was $\$0.09$ using gpt-3.5-turbo and $\$5.48$ using gpt-4..

##  5 Conclusion

In this paper, we present the first translation tool for Owens Valley Paiute
(a critically endangered Indigenous American language) and, in doing so,
propose a new methodology for low/no-resource machine translation: LLM-RBMT
(LLM-Assisted Rule-Based Machine Translation). Due to a lack of bilingual or
monolingual corpora, we use semantic similarity to measure the quality of
translations, and observe interesting cases where the translator performs
well, where it performs well given the vocabulary (primarily verbs and nouns)
available to it, and where it performs poorly due to current implementation
limitations.

This research opens up many directions for future work. First, expanding the
translation tool (e.g., by adding more vocabulary, building versions for other
languages, etc.) will make it more useful for the communities it is meant to
serve. We are currently working to add more complex sentence structures to
test the limits of this methodology. Building this system required expert
knowledge of the target language. It may be interesting to explore how large
language models might be able to help with this task as well (e.g., in finding
patterns and implementing RBMT logic) to make it easier for non-experts (and
even experts in the language who are less comfortable writing code) to build
these kinds of translators. Finally, other methodologies for leveraging LLMs
for low/no-resource machine translation should be explored. For example, a RAG
(retrieval augmented generation) Lewis et al. (2020) approach that searches a
knowledge-base for known sentence structures, vocabulary, grammar rules, etc.
and uses them to perform zero-shot translation might work well. The remarkable
general-purpose language skills that LLMs exhibit make them a promising tool
in helping revitalize critically endangered languages.

## References

  * Baird (2016) Jessie Little Doe Baird. 2016.  Wopanaak language reclamation program: bringing the language home.  Journal of Global Indigeneity, 2(2).
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.  Sparks of Artificial General Intelligence: Early experiments with GPT-4.
  * Cao et al. (2022) Jialun Cao, Meiziniu Li, Yeting Li, Ming Wen, Shing-Chi Cheung, and Haiming Chen. 2022.  Semmt: A semantic-based testing approach for machine translation systems.  _ACM Trans. Softw. Eng. Methodol._ , 31(2).
  * Chowdhery et al. (2022) Aakanksha Chowdhery et al. 2022.  PaLM: Scaling Language Modeling with Pathways.
  * Coronel-Molina and McCarty (2016) Serafín M. Coronel-Molina and Teresa L. McCarty. 2016.  Indigenous language revitalization in the americas.
  * Explosion (2024) Explosion. 2024.  Industrial-Strength Natural Language Processing.  https://spacy.io/.  Accessed: 3 Mar 2024.
  * Hendy et al. (2023) Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023.  How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation.
  * Khanna et al. (2021) Tanmai Khanna, Jonathan N. Washington, Francis M. Tyers, Sevilay Bayatlı, Daniel G. Swanson, Tommi A. Pirinen, Irene Tang, and Hèctor Alòs i Font. 2021.  Recent advances in Apertium, a free/open-source rule-based machine translation platform for low-resource languages.  _Machine Translation_ , 35(4):475–502.
  * Lankford et al. (2023) Séamus Lankford, Haithem Afli, and Andy Way. 2023.  adaptmllm: Fine-tuning multilingual language models on low-resource languages with integrated LLM playgrounds.  _Inf._ , 14(12):638.
  * Lewis et al. (2020) Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Moseley (2010) Christopher Moseley. 2010.  _Atlas of the World’s Languages in Danger_.  Unesco.
  * Muennighoff et al. (2023) Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023.  MTEB: massive text embedding benchmark.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023_ , pages 2006–2029. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.
  * OpenAI (2024a) OpenAI. 2024a.  New and Improved Embedding Model.  https://openai.com/blog/new-and-improved-embedding-model.  Accessed: 3 Mar 2024.
  * OpenAI (2024b) OpenAI. 2024b.  New embedding models and API updates.  https://openai.com/blog/new-embedding-models-and-api-updates.  Accessed: 3 Mar 2024.
  * Pirinen (2019) Tommi A Pirinen. 2019.  Workflows for kickstarting RBMT in virtually no-resource situation.  In _Proceedings of the 2nd Workshop on Technologies for MT of Low Resource Languages_ , pages 11–16, Dublin, Ireland. European Association for Machine Translation.
  * Ranathunga et al. (2023) Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. 2023.  Neural machine translation for low-resource languages: A survey.  _ACM Comput. Surv._ , 55(11):229:1–229:37.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Robinson et al. (2023) Nathaniel R. Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023.  ChatGPT MT: Competitive for High- (but not Low-) Resource Languages.
  * SIL International (2024) SIL International. 2024.  639 Identifier Documentation: mnr.  Accessed: 11 Mar 2024.
  * Song et al. (2021) Yurun Song, Junchen Zhao, and Lucia Specia. 2021.  Sentsim: Crosslingual semantic evaluation of machine translation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 3143–3156. Association for Computational Linguistics.
  * Taylor and Kochem (2022) Joshua Taylor and Timothy Kochem. 2022.  Access and empowerment in digital language learning, maintenance, and revival: a critical literature review.  _Diaspora, Indigenous, and Minority Education_ , 16(4):234–245.
  * Torregrosa et al. (2019) Daniel Torregrosa, Nivranshu Pasricha, Maraim Masoud, Bharathi Raja Chakravarthi, Juan A. Alonso, Noe Casas, and Mihael Arcan. 2019.  Leveraging rule-based machine translation knowledge for under-resourced neural machine translation models.  In _Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, MTSummit 2019, Dublin, Ireland, August 19-23, 2019_ , pages 125–133. European Association for Machine Translation.
  * Webber et al. (2010) William Webber, Alistair Moffat, and Justin Zobel. 2010.  A similarity measure for indefinite rankings.  _ACM Trans. Inf. Syst._ , 28(4):20:1–20:38.

##  Appendix A OVP to English Translator Results

Table 3: One hundred randomly generated sentences (using the OVP sentence building tool described in Section 3) and translated (using the OVP to English translator described in Section 3) labeled 1 if the translation is accurate and 0 otherwise. sentence | translation | label
---|---|---
tei-zawa-ku ihi | This cooked us. | 1
isha’-uu tüba-neika ai-hibi-pü | That coyote has drunk these pinenuts. | 1
tüba-uu tsibui-dü | Those pinenuts climb. | 1
ta-w̃ui-gaa-wei mahuw̃a | "They are going to write to us, you and I." | 1
wo’ada-uu aaponu’-oka u-zawa-ti | The mosquito is cooking the apple. | 1
tübbi-uu tüwoobü-neika a-yadohi-pü | The rock has talked to the earth. | 1
payahuupü-uu katü-ti | That river is sitting. | 1
kwisha’i-wei üü | You will sneeze. | 1
isha’-ii tübbi-neika mai-w̃ui-gaa-wei | This coyote is going to write these rocks. | 1
toni-uu wünü-ti | The wickiup is standing. | 1
isha’pugu-neika ihi mai-dama’i-ku | This found these dogs. | 1
wo’ada-neika ihi mai-dama’i-gaa-wei | This will find these mosquitoes. | 1
tabuutsi’-uu tüba-noka u-buni-ku | The cottontail saw those pinenuts. | 1
maishibü-neika uhu ai-nia-ti | He/she/it is reading these corn. | 1
koopi’-ii wükada-noka ui-nia-ku | This coffee read those bird snakes. | 1
tuunapi-uu waakü-pü | That food has worked. | 1
katünu-ii koopi’-oka ui-nobini-wei | This chair will visit those coffees. | 1
aingwü-neika mahu ma-nia-dü | He/she/it reads the squirrel. | 1
maishibü-uu wükihaa-gaa-wei | That corn is going to smile. | 1
isha’-uu isha’-oka ui-zawa-wei | That coyote will cook those coyotes. | 1
aingwü-ii tsibui-dü | This squirrel climbs. | 1
katünu-ii tübbi-neika ma-buni-wei | This chair will see this rock. | 1
wükada-uu tsibui-ku | The bird snake climbed. | 1
wo’ada-uu paya-neika ma-hibi-ti | The mosquito is drinking the water. | 1
pagwi-neika mahu ma-w̃ui-pü | He/she/it has written/is writing this fish. | 1
tabuutsi’-uu isha’pugu-neika mai-nobini-gaa-wei | That cottontail is going to visit those dogs. | 1
paya-neika mahu mai-hibi-gaa-wei | He/she/it is going to drink this water. | 1
nishua’i-pü nüü | I am laughing. | 1
aaponu’-ii küna-neika a-düka-pü | This apple has eaten this wood. | 1
katü-dü uhu | He/she/it sits. | 1
mukita-uu isha’pugu-noka u-naki-ti | The lizard is chasing the dog. | 1
isha’-oka üü ui-dama’i-ku | You found those coyotes. | 1
pahabichi-uu wükihaa-dü | That bear smiles. | 1
pahabichi-ii wo’abi-noka ui-naka-dü | This bear hears those worms. | 1
habi-ku ihi | This lay down. | 1
tümui-ku taa | You and I wrote. | 1
üwi-ku ihiw̃a | These slept. | 1
tübbi-uu ta-naka-ku | "That rock heard us, you and I." | 1
wo’ada-uu tei-gwana-dü | That mosquito smells us. | 1
tümui-dü mahuw̃a | They write. | 1
üwi-dü nüügwa | We are sleeping. | 1
tsibui-ti mahu | He/she/it is climbing. | 1
mukita-uu tsibui-pü | That lizard has climbed. | 1
payahuupü-uu toyabi-neika ma-zawa-gaa-wei | The river is going to cook the mountain. | 1
nobi-uu kwisha’i-ku | That house sneezed. | 1
kamü-uu wükihaa-dü | That jackrabbit smiles. | 1
toni-uu katü-wei | That wickiup will sit. | 1
aingwü-uu katünu-noka u-zawa-gaa-wei | The squirrel is going to cook that chair. | 1
paya-uu pasohobü-neika ai-buni-wei | That water will see those trees. | 1
toyabi-ii tsibui-ku | The mountain climbed. | 1
tsibui-wei taa | You and I will climb. | 1
pugu-uu wo’abi-neika ai-naki-ku | That horse chased those worms. | 1
mukita-uu wai-noka u-nobini-gaa-wei | The lizard is going to visit the rice. | 1
wükihaa-ti mahu | He/she/it is smiling. | 1
tüsüga-ii tüwoobü-neika ma-naki-gaa-wei | This weasel is going to chase this earth. | 1
yadoha-ku uhuw̃a | They talked. | 1
pahabichi-ii pugu-noka ui-nia-ku | The bear read those horses. | 1
paya-uu katünu-noka ui-yadohi-gaa-wei | Water is going to talk to those chairs. | 1
pagwi-ii wo’abi-noka ui-düka-ti | This fish is eating those worms. | 1
tabuutsi’-uu tübinohi-ku | That cottontail played. | 1
tünia-ku nüü | I read. | 1
poyoha-pü ihiw̃a | These are running. | 0
mukita-uu yotsi-wei | That lizard will fly. | 1
tabuutsi’-uu tübbi-neika ma-buni-pü | That cottontail has seen this rock. | 1
isha’pugu-ii tüba-neika ai-nobini-ku | This dog visited these pinenuts. | 1
isha’-uu katü-dü | That coyote sits. | 1
pasohobü-ii tsiipa-noka ui-naka-ku | The tree heard those birds. | 1
kamü-uu mukita-neika ma-zawa-wei | That jackrabbit will cook this lizard. | 1
tuunapi-uu tümui-gaa-wei | That food is going to write. | 1
wai-uu aingwü-neika ai-naka-pü | That rice has heard those squirrels. | 1
tsiipa-uu pugu-noka ui-naka-pü | The bird has heard those horses. | 1
pagwi-ii wükihaa-gaa-wei | These fish are going to smile. | 1
tei-nobini-dü uhuw̃a | They visit us. | 1
paya-neika ihi ma-dama’i-dü | This finds this water. | 1
isha’pugu-neika nüügwa ma-düka-ku | We ate this dog. | 1
tabuutsi’-uu hubiadu-dü | The cottontail sings. | 1
kwadzi-ii yadoha-wei | This tail will talk. | 1
isha’pugu-uu mukita-neika ai-naka-ti | That dog is hearing these lizards. | 1
pasohobü-noka üü u-zawa-gaa-wei | You are going to cook that tree. | 1
tsiipa-noka uhu u-buni-pü | He/she/it has seen or is seeing that bird. | 1
tuunapi-neika mahu mai-gwati-ku | He/she/it hit the foods here. | 0
isha’-eika nüügwa ai-w̃ui-gaa-wei | We are going to write coyotes. | 1
pasohobü-uu toyabi-neika ma-hibi-pü | The tree has drunk the mountain. | 1
aingwü-uu aaponu’-eika ai-naka-ti | The squirrel is hearing these apples. | 1
tabuutsi’-uu wai-noka u-yadohi-pü | The cottontail has talked to the rice. | 1
katünu-noka mahuw̃a ui-gwati-dü | They are hitting those chairs. | 1
tüwoobü-neika uhuw̃a mai-buni-ti | They are seeing these earths. | 1
koopi’-uu hukaw̃ia-ti | Coffee is walking. | 1
küna-ii ni-naka-ti | This wood is hearing us. | 1
toyabi-neika taagwa ma-buni-gaa-wei | We are going to see this mountain. | 1
isha’pugu-ii üwi-dü | This dog sleeps. | 1
pagwi-neika ihi mai-naka-gaa-wei | This will hear these fish. | 1
kwadzi-ii toni-neika ai-gwati-gaa-wei | This tail is going to hit those wickiups. | 1
paya-ii tuunapi-noka u-düka-ti | This water is eating that food. | 1
mukita-uu tümui-gaa-wei | That lizard is going to write. | 1
pahabichi-uu küna-neika ma-zawa-ku | The bear cooked the wood. | 1
isha’pugu-uu tabuutsi’-eika a-zawa-dü | That dog is cooking this cottontail. | 1
katünu-uu pahabichi-noka u-naki-ku | The chair chased the bear. | 1
küna-uu waakü-gaa-wei | That wood is going to work. | 1
pugu-neika mahu ai-naka-ku | He/she/it heard these horses. | 1

##  Appendix B Semantic Similarity Sentences

Table 4: Base sentences and other sentences ordered by their semantic similarity to the base sentence (as determined by authors). Base Sentence |  Other Sentences
(in order of most to least semantically similar to Base Sentence)
---|---
She sings. |  He sings.
|  He/she/it sings.
|  She performs a song.
|  A song is being sung by her.
|  She hums a tune.
|  She listens to music.
|  She dances.
|  She eats.
|  The cat sleeps.
|  Mountains echo silently.
The dog fell. |  The dog fell yesterday.
|  A dog stumbled.
|  The puppy tripped over.
|  The cat is running.
|  An animal is in motion.
|  The bird flies.
|  Leaves fall in autumn.
|  He reads a book.
|  Clouds cover the sky.
|  Apples on the moon are hungry.
The man ate an apple. |  The apple was eaten by the man.
|  A man consumes a fruit.
|  The boy nibbles on an apple.
|  Someone is eating.
|  He drinks water.
|  The woman ate a pie.
|  A cat chases a mouse.
|  Trees grow in the forest.
|  The car is red.
|  Stars twinkle at night.
The sun rises in the east. |  The east welcomes the sunrise.
|  Sunrise occurs in the east.
|  Day breaks in the east.
|  The moon sets in the west.
|  The stars shine at night.
|  Clouds gather before rain.
|  The wind changes direction.
|  Leaves fall in autumn.
|  Snow covers the mountains.
|  A book rests on the table.
Birds fly south for the winter. |  For winter, birds head south.
|  Migratory birds travel south when it gets cold.
|  Birds migrate to warmer climates during winter.
|  Fish swim upstream.
|  Bears hibernate in winter.
|  Flowers bloom in spring.
|  The earth orbits the sun.
|  Trees lose their leaves in fall.
|  The sky is blue.
|  A cat sleeps on the couch.
I read a book yesterday. |  Yesterday, I finished reading a book.
|  A book was read by me yesterday.
|  I watched a movie last night.
|  I’ll visit the library tomorrow.
|  She writes a letter.
|  He cooks dinner.
|  They are painting a house.
|  The sun sets in the evening.
|  A dog barks at night.
|  The car needs fuel.
The cake was delicious. |  Delicious was the cake.
|  The dessert tasted great.
|  We enjoyed the tasty cake.
|  The pie is sour.
|  Coffee complements breakfast.
|  Leaves rustle in the wind.
|  A bird sings outside.
|  Children play in the park.
|  Traffic is heavy today.
|  The phone is ringing.
Lightning precedes thunder. |  Thunder follows lightning.
|  First comes lightning, then comes thunder.
|  The storm brings lightning and thunder.
|  Rain refreshes the earth.
|  The sun warms the ground.
|  A river flows to the sea.
|  Mountains reach towards the sky.
|  A cat chases a mouse.
|  Books fill the shelf.
|  The clock ticks steadily.
She painted a beautiful picture. |  A beautiful picture was painted by her.
|  The painting she created is beautiful.
|  She sketches a portrait.
|  He writes a poem.
|  They are filming a movie.
|  Birds nest in spring.
|  Flowers wilt in the heat.
|  Kids play video games.
|  Cars fill the parking lot.
|  The sun sets late in summer.
The computer is broken. |  A broken state afflicts the computer.
|  The machine isn’t working.
|  We need to repair the computer.
|  The phone’s battery is dead.
|  Lights flicker during a power outage.
|  A book lies open on the desk.
|  Water boils at 100 degrees Celsius.
|  A cat purrs contentedly.
|  The door creaks when opened.
|  Birds migrate in autumn.
He solved the puzzle quickly. |  The puzzle was quickly solved by him.
|  Quickly, he found the solution to the puzzle.
|  She completes the crossword.
|  The mystery remains unsolved.
|  A race against time.
|  Flowers are sold at the market.
|  The river cuts through the valley.
|  A key unlocks the door.
|  Leaves turn red in autumn.
|  The train arrives at noon.
The stars twinkle at night. |  At night, the stars shimmer.
|  Twinkling stars fill the night sky.
|  Night unveils a sky full of stars.
|  The moon glows brightly.
|  Clouds mask the moon.
|  The sun sets, stars appear.
|  A comet streaks through the sky.
|  Fireflies glow in the dark.
|  Crickets chirp in the evening.
|  A candle flickers in the window.

##  Appendix C Vocabulary

tüka | eat
---|---
puni | see
hibi | drink
naka | hear
kwana | smell
kwati | hit
yadohi | talk to
naki | chase
tsibui | climb
sawa | cook
tama’i | find
nia | read
mui | write
nobini | visit

(a) Transitive Verbs

katü | sit
---|---
üwi | sleep
kwisha’i | sneeze
poyoha | run
mia | go
hukaw̃a | walk
wünü | stand
habi | lie down
yadoha | talk
kwatsa’i | fall
waakü | work
wükihaa | smile
hubiadu | sing
nishua’i | laugh
tsibui | climb
tübinohi | play
yotsi | fly
nüga | dance
pahabi | swim
tünia | read
tümui | write
tsiipe’i | chirp

(b) Intransitive Verbs

isha’ | coyote
---|---
isha’pugu | dog
kidi’ | cat
pugu | horse
wai | rice
tüba | pinenuts
maishibü | corn
paya | water
payahuupü | river
katünu | chair
toyabi | mountain
tuunapi | food
pasohobü | tree
nobi | house
toni | wickiup
apo | cup
küna | wood
tübbi | rock
tabuutsi’ | cottontail
kamü | jackrabbit
aaponu’ | apple
tüsüga | weasle
mukita | lizard
wo’ada | mosquito
wükada | bird snake
wo’abi | worm
aingwü | squirrel
tsiipa | bird
tüwoobü | earth
koopi’ | coffee
pahabichi | bear
pagwi | fish
kwadzi | tail

(c) Nouns

ku | completive (past)
---|---
ti | present ongoing (-ing)
dü | present
wei | future (will)
gaa-wei | future (going to)
pü | have x-ed, am x-ed

(d) Object Suffixes

nüü | I
---|---
uhu | he/she/it
uhuw̃a | they
mahu | he/she/it
mahuw̃a | they
ihi | this
ihiw̃a | these
taa | you and I
nüügwa | we (exclusive)
taagwa | we (inclusive)
üü | you
üügwa | you (plural)

(e) Subject Pronouns

ii | (proximal)
---|---
uu | (distal)

(f) Subject Suffixes

i | me
---|---
u | him/her/it (distal)
ui | them (distal)
ma | him/her/it (proximal)
mai | them (proximal)
a | him/her/it (proximal)
ai | them (proximal)
ni | us (plural, exclusive)
tei | us (plural, inclusive)
ta | us (dual), you and I
ü | you (singular)
üi | you (plural), you all

(g) Object Pronouns

eika | (proximal)
---|---
oka | (distal)

(h) Object Suffixes

Table 5: Vocabulary available in sentence building system.

##  Appendix D English to OVP Translation Results

Figure 5: Results for subject-verb sentences. The dark, medium, and light gray
bands represent the baseline similarity (between unrelated sentences in the
dataset) +/- one, two, and three standard deviations, respectively. Figure 6:
Results for subject-verb-object sentences. The dark, medium, and light gray
bands represent the baseline similarity (between unrelated sentences in the
dataset) +/- one, two, and three standard deviations, respectively. Figure 7:
Results for two-verb sentences. The dark, medium, and light gray bands
represent the baseline similarity (between unrelated sentences in the dataset)
+/- one, two, and three standard deviations, respectively. Figure 8: Results
for two-clause sentences. The dark, medium, and light gray bands represent the
baseline similarity (between unrelated sentences in the dataset) +/- one, two,
and three standard deviations, respectively. Figure 9: Results for complex
sentences. The dark, medium, and light gray bands represent the baseline
similarity (between unrelated sentences in the dataset) +/- one, two, and
three standard deviations, respectively.

##  Appendix E Semantic Similarity Baseline Score

Figure 10: Distribution of semantic similarity scores between all pairs of
sentences in the dataset.


11institutetext: Simula Research Laboratory, Oslo, Norway
11email: {hassan, shaukat<EMAIL_ADDRESS>
22institutetext: Section of Welfare Technologies, Oslo Kommune Helseetaten,
Norway
22email<EMAIL_ADDRESS>
33institutetext: Oslo Metropolitan University, Oslo, Norway

# HITA: An Architecture for System-level Testing of Healthcare IoT
Applications

Hassan Sartaj 11 Shaukat Ali 11 3 3 Tao Yue 11 Kjetil Moberg 22

###### Abstract

System-level testing of healthcare Internet of Things (IoT) applications
requires creating a test infrastructure with integrated medical devices and
third-party applications. A significant challenge in creating such test
infrastructure is that healthcare IoT applications evolve continuously with
the addition of new medical devices from different vendors and new services
offered by different third-party organizations following different
architectures. Moreover, creating test infrastructure with a large number of
different types of medical devices is time-consuming, financially expensive,
and practically infeasible. Oslo City’s healthcare department faced these
challenges while working with various healthcare IoT applications. This paper
presents a real-world software architecture (HITA) to create a test
infrastructure for healthcare IoT applications. We discuss the quality
requirements achieved by HITA and the status of work products developing as a
part of HITA. We also present our experiences and lessons learned from the
architectural work related to HITA.

###### Keywords:

Healthcare Internet of Things (IoT) Software Architecture System Testing.

##  1 Introduction

Healthcare Internet of Things (IoT) applications follow a cloud-based
architecture to create an interconnected network with various medical devices
and third-party applications [11]. The ultimate goal is to create a central
access point for medical professionals, patients, hospitals, pharmacies, and
caretakers for delivering efficient healthcare services. Failure to provide
timely healthcare services may lead to financial and human life loss.
Automated and rigorous system-level testing of healthcare IoT applications is
essential to ensure their dependability.

This work is conducted with Oslo City’s healthcare department [2] under the
national welfare technology program [1]. Oslo City’s healthcare department is
working with various industries for developing healthcare IoT applications to
deliver patients with high-quality services. One of the primary objectives is
to create a test infrastructure for the system-level testing of healthcare IoT
applications. Such test infrastructure requires integrating physical medical
devices (e.g., medicine dispensers) and third-party applications (e.g.,
pharmacies) with a healthcare IoT application. A major testing challenge is
that healthcare IoT applications evolve continuously with the addition of new
medical devices, new/updated medical services, and new third-party
applications. Integrating several different types of medical devices from
various vendors is time-consuming, costly, and not a practical solution. Each
third-party application has a limit on the maximum number of allowed requests
for a particular time interval. Testing healthcare IoT applications within the
limitations of third-party applications is challenging.

Several architectures have been proposed in the literature for developing
healthcare IoT applications [21]. A few works also utilize architectures for
various software testing activities, e.g., integration testing [20]. Our work
focuses on designing an architecture to create a test infrastructure that
enables automated system-level testing of healthcare IoT applications.

This paper presents a real-world software architecture (HITA) to create test
infrastructure for healthcare IoT applications. HITA is designed considering
the quality requirements of Oslo City such as evolvability, extensibility,
heterogeneity, scalability, maintainability, availability, security, privacy,
robustness, portability, and cost-effective solution. We discuss the status of
work products developed as a part of HITA. We also describe our experiences
and lessons learned from applying HITA in a real-world industrial context.

The remaining paper is organized as follows. Related works are discussed in
Section 2, HITA is presented in Section 3, lessons learned are described in
Section 4, and the paper’s conclusion is given in Section 5.

##  2 Related Works

Several works are available related to architectures for healthcare IoT
targeting various aspects such as; analysis of IoT architectures for
healthcare applications [21], design patterns for healthcare IoT [18],
architecture for intelligent IoT-based healthcare systems [10], architectural
design decisions for developing digital twins of IoT systems [16], a tool for
modeling IoT architectures [25], health monitoring architecture for IoT
systems [6], an architecture for IoT-based remote patient monitoring [3], a
requirements-based healthcare IoT architecture [15], distributed IoT
architecture [4], health data sharing architecture [22], and an architecture
for blockchain-driven healthcare IoT systems [26]. Our work focuses on an
architecture to develop a test infrastructure for testing healthcare IoT
applications.

Some works are also available concerning architecture-based testing including:
analysis of architecture’s function in software testing [7], architecture-
based test criteria [14], regression testing with software architectures [19],
architecture-driven integration testing [20], an architecture for analyzing
fault tolerance [8], and reliability assessment using architecture models [9].
Compared with the works mentioned above, our work presents an architecture for
creating a test infrastructure to enable the system-level testing of
healthcare IoT applications.

Figure 1: An overview of HITA. The arrow ( ) shows one-way information flow, (
) indicates two-way information flow, ( ) exhibits behavior simulation, and (
) depicts optional information flow.

##  3 HITA: An Architecture for Test Infrastructure

Figure 1 shows a real-world software architecture (HITA) to create test
infrastructure for healthcare IoT applications. HITA is designed based on two
commonly used architectural patterns, i.e., collaborative and centralized for
healthcare IoT [21]. HITA follows IoT reference architecture [13] which is
composed of an Application Layer including healthcare IoT core and testing
process, IoT Integration Middleware with the digital twins (DTs) and test
stubs (TS) components, Gateways, and Device comprising physical medical
devices.

Healthcare IoT Core. The system under test (SUT) is a healthcare IoT
application core that consists of several web and mobile clients for different
users such as patients, medical professionals, caregivers, and health
authorities. The primary communication channel for mobile clients (including
iPad/Tablets) is the 4G/5G network due to its availability and access in
remote areas. WiFi is used as an alternative communication channel in rare
cases. An important component of healthcare IoT applications is Application
Programming Interfaces (APIs) developed according to Representational State
Transfer (REST) architecture [12]. These REST APIs allow communication among
various clients, third-party applications, and medical devices. The data
interchange format used for this purpose is JavaScript Object Notation (JSON).
To execute the tests on SUT, several different types of medical devices and
third-party applications need to be integrated. HITA utilizes the DTs
component for medical devices and TS components for third-party applications
to handle integration challenges. API Gateways are used for secure
communication with DTs and TS components using Hypertext Transfer Protocol
(HTTP) and secure API keys.

Medical Devices - DTs Component. For integration with medical devices, HITA
utilizes the concept of digital twins to create a virtual representation of
physical devices. Each medical device from a different vendor is connected to
a server with several APIs for integration (as shown in the right-bottom of
Figure 1). The architecture for creating physical medical devices DTs consists
of one DT Server with APIs (e.g., APIs DT-D1) specific to a certain type of
DTs representing a particular device (e.g., DTs-D1). APIs need to be developed
following REST architecture [12] to allow easy integration with SUT. DTs of a
medical device are generated using a model-driven engineering (MDE) approach
in which structural aspects are modeled as a metamodel and behavioral aspects
are specified using executable state machines. In the case of multiple
versions of a medical device, a separate DT is required to be generated which
is easy with the MDE approach. JSON data interchange format is used between
the DT Server and DTs. Schema Registry can be used for compatibility with
medical devices supporting different JSON schema. The DT component also
consists of Data Persistence to preserve the state of DTs among various
requests. The APIs of DTs are used to integrate DTs with SUT and physical
medical devices. During testing, the DTs act as middle-ware between SUT and
physical medical devices. DTs handle all communication traffic from SUT and
communicate (via HTTP) with their physical twins when necessary.

Third-party Applications - TS Component. To handle the challenges of
integrating third-party applications for testing purposes, HITA’s TS component
plays an important role. Each third-party application has dedicated servers
with APIs for integration. The architecture for test stubs creation consists
of one TS Server with APIs (e.g., APIs TS1) simulating the behavior of various
applications (e.g., App 1). The APIs for each test stub are required to be
developed according to REST architecture [12] for easy integration with SUT.
Test stubs play a key role in replicating the functionality of third-party
applications. For APIs requiring data (e.g., health data), the architecture
includes an artificial data store with multiple databases corresponding to
APIs representing different applications. The data manipulation is performed
using query language compliant with the database type.

Testing Process. The testing process starts with the test generation step
using techniques for generating test data, test sequence, and test oracle.
Before testing SUT, it is important to ensure that DTs and TS components are
accurately representing all behaviors. This can be done through pilot
experiments evaluating the similarity in behaviors. The similarity in outputs
should ideally be close to 100% because it can affect testing results. The
generated tests in the form of test scripts are executed on the SUT. Test
execution requires API keys for communicating with SUT according to test
scripts. The results of test execution are evaluated to analyze errors,
faults, and failures. Moreover, test optimization during test generation is
required in the case of testing in a rapid-release environment and within a
short time frame.

HITA Operational Context. A tester initiates the testing process for testing a
particular aspect of SUT such as REST APIs testing or graphical user interface
(GUI) testing. This requires SUT integrated and operated with DTs and TS
components. Tests are executed on SUT through HTTP using JSON format. The SUT
processes the request and communicates with medical devices DTs or third-party
applications TS depending on the test case. Finally, SUT generates a JSON
response containing test execution results and sends it to the test execution
module. This process continues for a specified testing budget.

###  3.1 Quality Attributes

Scalability. An important concern when testing a healthcare IoT application
with a growing number of medical devices is scalability of development
efforts. HITA provides a component for digital twins that are used in place of
physical medical devices during the testing process. Any number of digital
twins corresponding to a particular medical device can be easily created and
operated by utilizing model-driven practices. Digital twins eliminate the need
for integrating several medical devices physically and the risk of damaging
physical devices. The use of digital twins is also cost-effective which is
another key consideration for creating test infrastructure.

Maintainability. Model-driven generation of digital twins allows for achieving
maintainability quality. Furthermore, HITA utilizes one server for digital
twin and one server for TS components that can operate locally or on the cloud
depending upon industrial preferences. Using one server each for both
components requires less maintainability effort as compared to using
individual servers for different applications.

Extensibility. The modular structure of HITA components allows for achieving
extensibility. For the addition of a new medical device, digital twins and
their APIs need to be created using the model-driven approach. The APIs of
digital twins are used for communication with SUT and the physical device. In
the case of adding new healthcare services or features from a third-party
application, a test stub is required to be created consisting of APIs for
communication with SUT. The artificial dataset is created for testing if the
new application is data-intensive.

Evolvability. HITA implicitly achieves evolvability quality attribute due to
its overall modular structure and components with scalable, maintainable, and
extensible characteristics. This leads to achieving overall evolvability
requirements of SUT and integrated devices/applications during testing.

Heterogeneity. Creating test infrastructure for healthcare IoT applications
involves integration with heterogeneous systems such as different medical
devices and various types of third-party applications. In HITA, heterogeneity
of different types of medical devices is handled using model-driven creation
of digital twins and other APIs. For the heterogeneity of third-party
applications, HITA creates test stubs in the form of APIs. The APIs of digital
twins and test stubs allow easy integration of diverse systems with SUT.

Security & Privacy. The use of real health records (e.g., patients’ health
data) during the testing process may lead to security breaches and data
privacy issues. To handle security concern, HITA impose authentication and
authorization mechanisms on all components. For data privacy, the TS and DTs
components consist of Artificial Data Store and Data Persistence respectively
which contain synthetic data instead of real patients’ health data.

Availability. A critical problem solved by HITA is the availability of medical
devices and third-party applications during testing. Running extensive tests
with the goal of rigorous testing may lead to unavailable services. HITA uses
third-party applications’ test stubs in the form of APIs running on a server
(TS Server). Similarly, digital twins have DT Server to handle requests during
test execution. These servers are dedicated for testing purposes and hosted
locally or on the cloud according to the up-time required for the testing
process.

Robustness & Portability. The goal of testing a healthcare IoT application is
to identify errors, faults, and failures with the assumption that integrated
applications are robust. HITA instructs the development of APIs for DTs and TS
components following REST architecture, which provides a reliable mechanism
for integration and communication among various applications [12]. Moreover,
as a result of test execution, DTs and TS components generate responses with
failure and success information that enables identifying errors/faults in SUT.
Portability is an additional feature of HITA. The architecture followed by TS
and DTs components can work on local machines when testing in offline mode and
remotely in different cloud environments.

###  3.2 Work Products

As a part of the test infrastructure, the DTs component is currently being
developed according to HITA. Specifically, digital twins for medicine
dispensers are developed using a model-driven approach (available online
[23]). The approach utilizes metamodeling and executable state machines to
create and operate digital twins of medicine dispensers. The DT Server is
developed using the Flask framework that can work locally and can be easily
deployed on any cloud. The APIs following the REST architecture are generated
using the Flask-compatible Flask-RESTful framework. The REST APIs use JSON
format for data interchanges with SUT and physical medicine dispensers.
Moreover, the technologies used for developing the DTs component make them
easily portable from one platform to another without requiring major changes.

For the system-level testing of SUT, we utilize two methods, (i) testing of
the application’s backend REST APIs, and (ii) testing based on the GUI of web
and mobile clients. For REST APIs testing of SUT, initially, we explored
automated testing tools EvoMaster [5] and RESTest [17] and identified several
API failures. For GUI-based testing, we use Selenium
Webdriver111https://www.selenium.dev/documentation/webdriver/ with the web
client of SUT.

The work on creating the TS component of HITA is a future step. Nevertheless,
several tools available for creating test stubs are under consideration such
as JSON Server222https://www.npmjs.com/package/json-server, and
Mocki.io333https://mocki.io/. An initial consideration is using JSON Server
which is an open-source tool for creating test stubs in the form of REST APIs.

##  4 Experiences and Lessons Learned

Following we outline our experience and lessons learned while developing HITA
work products and analyzing them through experiments.

DTs Role in Test Infrastructure. System-level testing of healthcare IoT
applications requires different medical devices in the loop. Each type of
medical device from a different vendor is linked to a web server that has
certain constraints such as maximum allowed requests. The test generation and
execution process involves sending several requests to medical devices through
a healthcare IoT application. This leads to the blocking of service or the
damaging of a medical device. Further, testing with hundreds of such devices
is costly and not a practical option. Based on such experiences from Oslo
City, we propose the idea of using DTs in place of physical medical devices to
enable testing with multiple digital representations of physical devices.
Thus, DTs have an important role in this regard. DTs with dedicated DT Server
and APIs eliminate the risk of service blockage or device damage. Virtually
representing physical devices, DTs are a scalable and cost-effective solution.
Our experiments with 100 DTs in different batches (i.e., 10, 20, 30, …, 100)
indicated scalability, heterogeneity, and cost-effectiveness of the DTs
component.

Modeling for DTs. The model-driven approach for automated generation of DTs
requires creating a metamodel as an abstract structural representation of
similar types of medical devices, e.g., medicine dispensers, and modeling
behavioral aspects of medical devices using executable state machines. Several
modeling tools (e.g., IBM RSA and Papyrus) are available for this purpose.
Test engineers need to have a fundamental level of familiarity with any of the
modeling tools. Models developed in this way involve a one-time effort and can
be reused for testing multiple evolution phases of SUT. In the case of adding
new medical devices, only metamodel and executable state machines need to be
fine-tuned.

Fidelity Evaluation of DTs. While utilizing DTs of physical devices, an
important consideration is the fidelity of DTs corresponding to physical
twins. For this purpose, we empirically evaluated the fidelity of medicine
dispenser DTs (up to 100) in terms of their functional similarities with a
physical medicine dispenser device. The results highlighted the functionality
of DTs was almost similar to medicine dispensers. Moreover, fidelity
evaluation in terms of internal behaviors is challenging due to limited access
to internal operations of physical medicine dispensers. A future direction is
the evaluation of DTs’ fidelity considering the internal behavior of medicine
dispensers.

Testing with Third-party Applications. An initial version of HITA was
experimented with third-party applications and without a TS component using
testing tools like EvoMaster and RESTest. With third-party applications in a
loop, we observed that it is difficult to analyze the primary sources of
failures or faults, i.e., a failure occurred due to a fault in SUT or the
integrated application [24]. Moreover, we observed that services provided by
third-party applications often become unavailable during the testing process.
This led to a significant bottleneck in the rigorous testing of healthcare IoT
applications. Thus, creating test stubs for third-party applications is a
viable solution.

Domain-specific Testing Strategies. Our experiments with EvoMaster and RESTest
highlighted the need for domain-specific testing strategies for healthcare IoT
applications [24]. We analyzed that automated realistic test data generation
is a challenging and open research problem. For example, automatically
generating a valid medication plan for a patient is not a simple task.
Generating a valid medication plan requires information regarding the start
date, dose intake, number of days to take medicines, number of doses, and the
total number of medicines allowed in a roll of a medicine dispenser. This
involves understanding domain properties related to medications and the
context of a medicine dispenser. There is still a need for domain-specific
testing strategies.

Intelligent Test Generation Technique. Healthcare IoT applications commonly
have a two-way communication mechanism with different medical devices and
third-party applications. Several scenarios require an integrated medical
device or third-party application to initiate the first step of the process.
Automatically generating test cases for such scenarios is challenging. For
instance, the steps to assign an alert (received from a patient) to concerned
personnel include: (i) the patient’s medical device generates an alert, (ii)
the alert is received as an unassigned alert, (iii) identify an appropriate
person (doctor, nurse, caretaker, etc.) to assign the alert, and (iv) assign
the alert with notification to health authorities. An alert should be
generated beforehand to test the alert-assigning scenario. This requires an
intelligent technique for automated test case generation since HITA is
designed for creating test infrastructure.

Test Optimization. Testing an industrial healthcare IoT application in a
production and a rapid-release environment requires certain conditions such as
a designated time budget for test generation and execution. Executing a
maximum number of test cases with the aim of rigorous testing for each release
is desirable but not feasible, even using test stubs and digital twins. An
approach for generating and executing optimized test cases is necessary to
ensure the dependability of healthcare IoT applications within a given time
frame.

##  5 Conclusion

In this paper, we presented real-world architectural work in collaboration
with Oslo City’s healthcare department. We introduced HITA – a software
architecture for creating test infrastructure to facilitate automated system-
level testing of healthcare IoT applications. HITA is designed considering
quality attributes of paramount importance. We also described the status of
work products developed or currently being developed as a part of HITA-based
test infrastructure. Finally, we presented experience and lessons learned
based on experiments conducted with work products of HITA that are valuable
for industry practitioners working in a similar domain. Our lessons learned
are generalizable to various IoT-based systems such as activity/fitness
trackers, smart homes, and smart security systems.

#### Acknowledgements.

This work is a part of the WTT4Oslo project (No. 309175) funded by the
Research Council of Norway. All the experiments reported in this paper are
conducted in a laboratory setting of Simula Research Laboratory; therefore,
they do not by any means reflect the quality of services Oslo City provides to
its citizens.

## References

  * [1] National welfare technology program. https://www.helsedirektoratet.no/tema/velferdsteknologi, [Online; accessed 18-May-2023]
  * [2] Norwegian health authority. https://www.oslo.kommune.no/etater-foretak-og-ombud/helseetaten/, [Online; accessed 18-May-2023]
  * [3] Al-Joboury, I.M., Hemiary, E.H.: Internet of things architecture based cloud for healthcare. Iraqi Journal of Information and Communication Technology 1(1), 18–26 (2018)
  * [4] Alnefaie, S., Cherif, A., Alshehri, S.: Towards a distributed access control model for iot in healthcare. In: 2019 2nd International Conference on Computer Applications & Information Security (ICCAIS). pp. 1–6. IEEE (2019)
  * [5] Arcuri, A.: Restful api automated test case generation with evomaster. ACM Transactions on Software Engineering and Methodology (TOSEM) 28(1), 1–37 (2019)
  * [6] Azimi, I., Anzanpour, A., Rahmani, A.M., Pahikkala, T., Levorato, M., Liljeberg, P., Dutt, N.: Hich: Hierarchical fog-assisted computing architecture for healthcare iot. ACM Transactions on Embedded Computing Systems 16(5s), 1–20 (2017)
  * [7] Bertolino, A., Inverardi, P., Muccini, H.: Software architecture-based analysis and testing: a look into achievements and future challenges. Computing 95, 633–648 (2013)
  * [8] Brito, P.H., de Lemos, R., Rubira, C.M.: Development of fault-tolerant software systems based on architectural abstractions. In: Software Architecture: Second European Conference, ECSA 2008 Paphos, Cyprus, September 29-October 1, 2008 Proceedings 2. pp. 131–147. Springer (2008)
  * [9] Brosch, F., Koziolek, H., Buhnova, B., Reussner, R.: Architecture-based reliability prediction with the palladio component model. IEEE Transactions on Software Engineering 38(6), 1319–1339 (2011)
  * [10] Catarinucci, L., De Donno, D., Mainetti, L., Palano, L., Patrono, L., Stefanizzi, M.L., Tarricone, L.: An iot-aware architecture for smart healthcare systems. IEEE internet of things journal 2(6), 515–526 (2015)
  * [11] Fiedler, M., Meissner, S.: Iot in practice: examples: Iot in logistics and health. Enabling Things to Talk: Designing IoT solutions with the IoT Architectural Reference Model pp. 27–36 (2013)
  * [12] Fielding, R.T.: Architectural styles and the design of network-based software architectures. University of California, Irvine (2000)
  * [13] Guth, J., Breitenbücher, U., Falkenthal, M., Leymann, F., Reinfurt, L.: Comparison of iot platform architectures: A field study based on a reference architecture. In: 2016 Cloudification of the Internet of Things (CIoT). pp. 1–6. IEEE (2016)
  * [14] Jin, Z., Offutt, J.: Deriving tests from software architectures. In: Proceedings 12th International Symposium on Software Reliability Engineering. pp. 308–313. IEEE (2001)
  * [15] Lindquist, W., Helal, S., Khaled, A., Hutchinson, W.: Iotility: architectural requirements for enabling health iot ecosystems. IEEE Transactions on Emerging Topics in Computing 9(3), 1206–1218 (2019)
  * [16] Malakuti, S., Grüner, S.: Architectural aspects of digital twins in iiot systems. In: Proceedings of the 12th European conference on software architecture: companion proceedings. pp. 1–2 (2018)
  * [17] Martin-Lopez, A., Segura, S., Ruiz-Cortés, A.: Restest: automated black-box testing of restful web apis. In: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. pp. 682–685 (2021)
  * [18] Mezghani, E., Exposito, E., Drira, K.: A model-driven methodology for the design of autonomic and cognitive iot-based systems: Application to healthcare. IEEE Transactions on Emerging Topics in Computational Intelligence 1(3), 224–234 (2017)
  * [19] Muccini, H., Dias, M., Richardson, D.J.: Software architecture-based regression testing. Journal of Systems and Software 79(10), 1379–1396 (2006)
  * [20] Muccini, H., Inverardi, P., Bertolino, A.: Using software architecture for code testing. IEEE Transactions on Software Engineering 30(3), 160–171 (2004)
  * [21] Muccini, H., Spalazzese, R., Moghaddam, M.T., Sharaf, M.: Self-adaptive iot architectures: An emergency handling case study. In: Proceedings of the 12th European Conference on Software Architecture: Companion Proceedings. pp. 1–6 (2018)
  * [22] Nguyen, D.C., Pathirana, P.N., Ding, M., Seneviratne, A.: Bedgehealth: A decentralized architecture for edge-based iomt networks using blockchain. IEEE Internet of Things Journal 8(14), 11743–11757 (2021)
  * [23] Sartaj, H.: APD-DT: Automated tool for digital twins of medicine dispensers (Apr 2023), https://github.com/Simula-COMPLEX/WTSPublic
  * [24] Sartaj, H., Ali, S., Yue, T., Moberg, K.: Testing real-world healthcare iot application: Experiences and lessons learned. In: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’23) (2023). https://doi.org/10.1145/3611643.3613888, To Appear
  * [25] Sharaf, M., Abughazala, M., Muccini, H.: Arduino realization of caps iot architecture descriptions. In: Proceedings of the 12th European Conference on Software Architecture: Companion Proceedings. pp. 1–4 (2018)
  * [26] Sharma, P., Namasudra, S., Crespo, R.G., Parra-Fuente, J., Trivedi, M.C.: Ehdhe: Enhancing security of healthcare documents in iot-enabled digital healthcare ecosystems using blockchain. Information Sciences 629, 703–718 (2023)