# A Lost Croatian Cybernetic Machine Translation Program

Sandro Skansi  Leo Mršić  Ines Skelac  Sandro Skansi*
University of Zagreb
University Department of Croatian Studies
<EMAIL_ADDRESS>Leo Mršić
Algebra University College
<EMAIL_ADDRESS>Ines Skelac
Algebra University College
<EMAIL_ADDRESS>
###### Abstract

We are exploring the historical significance of research in the field of
machine translation conducted by Bulcsu Laszlo, Croatian linguist, who was a
pioneer in machine translation in Yugoslavia during the 1950s. We are focused
on two important seminal papers written by members of his research group from
1959 and 1962, as well as their legacy in establishing a Croatian machine
translation program based around the Faculty of Humanities and Social Sciences
of the University of Zagreb in the late 1950s and early 1960s. We are
exploring their work in connection with the beginnings of machine translation
in the USA and USSR, motivated by the Cold War and the intelligence needs of
the period. We also present the approach to machine translation advocated by
the Croatian group in Yugoslavia, which is different from the usual logical
approaches of the period, and his advocacy of cybernetic methods, which would
be adopted as a canon by the mainstream AI community only decades later.

Keywords: Bulcsu Laszlo, Machine Translation, History of Technology in
Croatia, Language Technologies, Natural Language Processing

##  1 Beginnings of Machine Translation and Artificial Intelligence in the USA
and USSR

In this paper, we are exploring the historical significance of Croatian
machine translation research group. The group was active in 1950s, and it was
conducted by Bulcsu Laszlo, Croatian linguist, who was a pioneer in machine
translation during the 1950s in Yugoslavia.

To put the research of the Croatian group in the right context, we have to
explore the origin of the idea of machine translation. The idea of machine
translation is an old one, and its origin is commonly connected with the work
of Rene Descartes, i.e. to his idea of universal language, as described in his
letter to Mersenne from 20.xi.1629 [5]. Descartes describes universal language
as a simplified version of the language which will serve as an “interlanguage”
for translation. That is, if we want to translate from English to Croatian, we
will firstly translate from English to an “interlanguage”, and then from the
“interlanguage” to Croatian. As described later in this paper, this idea had
been implemented in the machine translation process, firstly in the
Indonesian-to-Russian machine translation system created by Andreev, Kulagina
and Melchuk from the early 1960s.

In modern times, the idea of machine translation was put forth by the
philosopher and logician Yehoshua Bar-Hillel (most notably in [3] and [4]),
whose papers were studied by the Croatian group. Perhaps the most important
unrealized point of contact between machine translation and cybernetics
happened in the winter of 1950/51. In that period, Bar-Hillel met Rudolf
Carnap in Chicago, who introduced to him the (new) idea of cybernetics. Also,
Carnap gave him the contact details of his former teaching assistant, Walter
Pitts, who was at that moment with Norbert Wiener at MIT and who was supposed
to introduce him to Wiener, but the meeting never took place [8].
Nevertheless, Bar-Hillel was to stay at MIT where he, inspired by cybernetics,
would go to organize the first machine translation conference in the world in
1952 [8].

The idea of machine translation was a tempting idea in the 1950s. The main
military interest in machine translation as an intelligence gathering tool
(translation of scientific papers, daily press, technical reports, and
everything the intelligence services could get their hands on) was sparked by
the Soviet advance in nuclear technology, and would later be compounded by the
success of Vostok 1 (termed by the USA as a “strategic surprise”). In the
nuclear age, being able to read and understand what the other side was working
on was of crucial importance [24]. Machine translation was quickly absorbed in
the program of the Dartmouth Summer Research Project on Artificial
Intelligence in 1956 (where Artificial Intelligence as a field was born), as
one of the five core fields of artificial intelligence (later to be known as
natural language processing). One other field was included here, the “nerve
nets” as they were known back then, today commonly known as artificial neural
networks. What is also essential for our discussion is that the earliest
programming language for artificial intelligence, Lisp, was invented in 1958
by John McCarthy [17]. But let us take a closer look at the history of machine
translation. In the USA, the first major wave of government and military
funding for machine translation came in 1954, and the period of abundancy
lasted until 1964, when the National Research Council established the
Automatic Language Processing Advisory Committee (ALPAC), which was to assess
the results of the ten years of intense funding. The findings were very
negative, and funding was almost gone [24], hence the ALPAC report became the
catalyst for the first “AI Winter”.

One of the first recorded attempts of producing a machine translation system
in the USSR was in 1954 [20], and the attempt was applauded by the Communist
party of the Soviet Union, by the USSR Committee for Science and Technology
and the USSR Academy of Sciences. The source does not specify how this first
system worked, but it does delineate that the major figures of machine
translation of the time were N. Andreev of the Leningrad State University, O.
Kulagina and I. Melchuk of the Steklov Mathematical Institute. There is
information on an Indonesian-to-Russian machine translation system by Andreev,
Kulagina and Melchuk from the early 1960s, but it is reported that the system
was ultimately a failure, in the same way early USA systems were. The system
had statistical elements set forth by Andreev, but the bulk was logical and
knowledge-heavy processing put forth by Kulagina and Melchuk. The idea was to
have a logical intermediate language, under the working name “Interlingua”,
which was the connector of both natural languages, and was used to model
common-sense human knowledge. For more details, see [20].

In the USSR, there were four major approaches to machine translation in the
late 1950s [16]. The first one was the research at the Institute for Precise
Mechanics and Computational Technology of the USSR Academy of Sciences. Their
approach was mostly experimental and not much different from today’s empirical
methods. They evaluated the majority of algorithms known at the time
algorithms over meticulously prepared datasets, whose main strength was data
cleaning, and by 1959 they have built a German-Russian machine translation
prototype. The second approach, as noted by Mulić [16], was championed by the
team at the Steklov Mathematical Institute of the USSR Academy of Sciences led
by A. A. Reformatsky. Their approach was mainly logical, and they extended the
theoretical ideas of Bar-Hillel [4] to build three algorithms: French-Russian,
English-Russian and Hungarian-Russian. The third and perhaps the most
successful approach was the one by A. A. Lyapunov, O. S. Kulagina and R. L.
Dobrushin. Their efforts resulted in the formation of the Mathematical
Linguistics Seminar at the Faculty of Philology in Moscow in 1956 and in
Leningrad in 1957. Their approach was mainly information-theoretic (but they
also tried logic-based approaches [16]), which was considered cybernetic at
that time. This was the main role model for the Croatian efforts from 1957
onwards. The fourth, and perhaps most influential, was the approach at the
Experimental Laboratory of the Leningrad University championed by N. D.
Andreev [16]. Here, the algorithms for Indonesian-Russian, Arabic-Russian,
Hindu-Russian, Japanese-Russian, Burmese-Russian, Norwegian-Russian, English-
Russian, Spanish-Russian and Turkish-Russian were being built. The main
approach of Andreev’s group was to use an intermediary language, which would
capture the meanings [16]. It was an approach similar to KL-ONE, which would
be introduced in the West much later (in 1985) by Brachman and Schmolze [25].
It is also interesting to note that the Andreev group had a profound influence
on the Czechoslovakian machine translation program [21], which unfortunately
suffered a similar fate as the Yugoslav one due to the lack of funding.

Andreev’s approach was in a sense ”external”. The modelling would be
statistical, but its purpose would not be to mimic the stochasticity of the
human thought process, but rather to produce a working machine translation
system. Kulagina and Melchuk disagreed with this approach as they thought that
more of what is presently called ”philosophical logic” was needed to model the
human thought process at the symbolic level, and according to them, the
formalization of the human thought process was a prerequisite for developing a
machine translation system (cf. [20]). We could speculate that sub-symbolic
processing would have been acceptable too, since that approach is also rooted
in philosophical logic as a way of formalizing human cognitive functions and
is also ”internal” in the same sense symbolic approaches are.

There were many other centers for research in machine translation: Gorkovsky
University (Omsk), 1st Moscow Institute for Foreign Languages, Computing
Centre of the Armenian SSR and at the Institute for Automatics and
Telemechanics of the Georgian SSR [16]. It is worthwhile to note that both the
USA and the USSR had access to state-of-the-art computers, and the political
support for the production of such systems meant that computers were made
available to researchers in machine translation. However, the results were
poor in the late 1950s, and a working system was yet to be shown. All work was
therefore theoretical work implemented on a computer, which proved to be sub-
optimal.

##  2 The formation of the Croatian group in Zagreb

In Yugoslavia, organized effort in machine translation started in 1959, but
the first individual effort was made by Vladimir Matković from the Institute
for Telecommunications in Zagreb in 1957 in his PhD thesis on entropy in the
Croatian language [7]. The main research group in machine translation was
formed in 1958, at the Circle for Young Linguists in Zagreb, initiated by a
young linguist Bulcsu Laszlo, who graduated in Russian language, Southern
Slavic languages and English language and literature at the University of
Zagreb in 1952. The majority of the group members came from different
departments of the Faculty of Humanities and Social Sciences of the University
of Zagreb, with several individuals from other institutions. The members from
the Faculty of Humanities and Social Sciences were: Svetozar Petrović
(Department of Comparative Literature), Stjepan Babić (Department of Serbo-
Croatian Language and Literature), Krunoslav Pranjić (Department of Serbo-
Croatian Language and Literature), Željko Bujas (Department of English
Language and Literature), Malik Mulić (Department of Russian Language and
Literature) and Bulcsu Laszlo (Department of Comparative Slavistics). The
members of the research group from outside the Faculty of Humanities and
Social Sciences were: Božidar Finka (Institute for Language of the Yugoslav
Academy of Sciences and Arts), Vladimir Vranić (Center for Numerical Research
of the Yugoslav Academy of Sciences and Arts), Vladimir Matković (Institute
for Telecommunications), Vladimir Muljević (Institute for Regulatory and
Signal Devices) [7].

Laszlo and Petrović [11] also commented on the state of the art of the time,
noting the USA prototype efforts from 1954 and the publication of a collection
of research papers in 1955 as well as the USSR efforts starting from 1955 and
the UK prototype from 1956. They do not detail or cite the articles they
mention. However, the fact that they referred to them in a text published in
1959 (probably prepared for publishing in 1958, based on [11], where Laszlo
and Petrović described that the group had started its work in 1958) leads us
to the conclusion that the poorly funded Croatian research was lagging only a
couple of years behind the research of the superpowers (which invested heavily
in this effort). Another interesting moment, which they delineated in [11], is
that the group soon discovered that some experimental work had already been
done in 1957 at the Institute of Telecommunications (today a part of the
Faculty of Electrical Engineering and Computing at the University of Zagreb)
by Vladimir Matković. Because of this, they decided to include him in the
research group of the Faculty of Humanities and Social Sciences at the
University of Zagreb. The work done by Matković was documented in his doctoral
dissertation but remained unpublished until 1959.

The Russian machine translation pioneer Andreev expressed hope that the
Yugoslav (Croatian) research group could create a prototype, but sadly, due to
the lack of federal funding, this never happened [7]. Unlike their colleagues
in the USA and the USSR, Laszlo’s group had to manage without an actual
computer (which is painfully obvious in [27]), and the results remained mainly
theoretical. Appealing probably to the political circles of the time, Laszlo
and Petrović note that, although it sounds strange, research in computational
linguistics is mainly a top-priority military effort in other countries [11].
There is a quote from [7] which perhaps best delineates the optimism and
energy that the researchers in Zagreb had:

> ”[…] The process of translation has to mechanicalized as soon as possible,
> and this is only possible if a competent, fast and inexhaustible machine
> which could inherit the translation task is created, even if just schematic.
> The machine needs to think for us. If machines help humans in physical
> tasks, why would they not help them in mental tasks with their mechanical
> memory and automated logic” (p. 118).

##  3 Contributions of the Croatian group

Laszlo and Petrović [11] considered cybernetics (as described in [29] by
Wiener, who invented the term “cybernetics”) to be the best approach for
machine translation in the long run. The question is whether Laszlo’s idea of
cybernetics would drive the research of the group towards artificial neural
networks. Laszlo and his group do not go into neural network details (bear in
mind that this is 1959, the time of Rosenblatt), but the following passage
offers a strong suggestion about the idea they had (bearing in mind that
Wiener relates McCulloch and Pitts’ ideas in his book): ”Cybernetics is the
scientific discipline which studies analogies between machines and living
organisms” ([11], p. 107). They fully commit to the idea two pages later
([11], p. 109): ”An important analogy is the one between the functioning of
the machine and that of the human nervous system”. This could be taken to mean
a simple computer brain analogy in the spirit of [13] and later [23], but
Laszlo and Petrović specifically said that thinking of cybernetics as the
”theory of electronic computers” (as they are made) is wrong [11], since the
emphasis should be on modelling analogical processes. There is a very
interesting quote from [11], where Laszlo and Petrović note that ”today, there
is a significant effort in the world to make fully automated machine
translation possible; to achieve this, logicians and linguists are making
efforts on ever more sophisticated problems”. This seems to suggest that they
were aware of the efforts of logicians (such as Bar Hillel, and to some degree
Pitts, since Wiener specifically mentions logicians-turned-cyberneticists in
his book [29]), but still concluded that a cybernetic approach would probably
be a better choice.

Laszlo and Petrović [11] argued that, in order to trim the search space, the
words would have to be coded so as to retain their information value but to
rid the representations of needless redundancies. This was based on previous
calculations of language entropy by Matković, and Matković’s idea was simple:
conduct a statistical analysis to determine the most frequent letters and
assign them the shortest binary code. So A would get 101, while F would get
11010011 [11]. Building on that, Laszlo suggested that, when making an
efficient machine translation system, one has to take into account not just
the letter frequencies but also the redundancies of some of the letters in a
word [10]. This suggests that the strategy would be as follows: first make a
thesaurus, and pick a representative for each meaning, then stem or lemmatize
the words, then remove the needless letters from words (i.e. letters that
carry little information, such as vowels, but being careful not to equate two
different words), and then encode the words in binary strings, using the
letter frequencies. After that, the texts are ready for translation, but
unfortunately, the translation method is never explicated. Nevertheless, it is
hinted that it should be ”cybernetic”, which, along with what we have
presented earlier, would most probably mean artificial neural networks. This
is highlighted by the following passage ([11], p. 117):

> ”A man who spends 50 years in a lively and multifaceted mental activity
> hears a billion and a half words. For a machine to have an ability
> comparable to such an intellectual, not just in terms of speed but also in
> terms of quality, it has to have a memory and a language sense of the same
> capacity, and for that - which is paramount - it has to have in-built
> conduits for concept association and the ability to logically reason and
> verify, in a word, the ability to learn fast.”

Unfortunately, this idea of using machine learning was never fully developed,
and the Croatian group followed the Soviet approach(es) closely. Pranjić [22]
analyses and extrapolates five basic ideas in the Soviet Machine Translation
program, which were the basis for the Croatian approach:

  1. 1.

Separation of the dictionary from the MT algorithm

  2. 2.

Separation of the understanding and generation modules of the MT algorithms

  3. 3.

All words need to be lemmatized

  4. 4.

The word lemma should be the key of the dictionary, but other forms of the
word must be placed as a list in the value next to the key

  5. 5.

Use context to determine the meaning of polysemous words.

The dictionary that was mentioned before is, in fact, the intermediary
language, and all the necessary knowledge should be placed in this dictionary,
the keys should ideally be just abstract codes, and everything else would
reside and be accessible as values next to the keys [27]. Petrović, when
discussing the translation of poetry [19], noted that ideally, machine
translation should be from one language to another, without the use of an
intermediate language of meanings.

Finka and Laszlo envisioned three main data preparation tasks that are needed
before prototype development could commence [7]. The first task is to compile
a dictionary of words sorted from the end of the word to the beginning. This
would enable the development of what is now called stemming and lemmatization
modules: a knowledge base with suffixes so they can be trimmed, but also a
systematic way to find the base of the word (lemmatization) (p. 121). The
second task would be to make a word frequency table. This would enable
focusing on a few thousand most frequent words and dropping the rest. This is
currently a good industrial practice for building efficient natural language
processing systems, and in 1962, it was a computational necessity. The last
task was to create a good thesaurus, but such a thesaurus where every data
point has a ”meaning” as the key, and words (synonyms) as values. The
prototype would then operate on these meanings when they become substituted
for words.

But what are those meanings? The algorithm to be used was a simple statistical
alignment algorithm (in hopes of capturing semantics) described in [27] on a
short Croatian sentence ”čovjek [noun-subject] puši [verb-predicate] lulu
[noun-objective]” (A man is smoking a pipe). The first step would be to parse
and lemmatize. Nouns in Croatian have seven cases just in the singular, with
different suffixes, for example:
ČOVJEK - Nominative singular
ČOVJEKA - Genitive singular
ČOVJEKU - Dative singular
ČOVJEKA - Accusative singular
ČOVJEČE - Vocative singular
ČOVJEKU - Locative singular
ČOVJEKOM - Instrumental singular
Although morphologically transparent, the lemma in the mentioned case would be
“ČOVJEK-”; there is a voice change in the Vocative case, so for the purpose of
translation, “ČOVJE-” would be the “lemma”. The other two lemmas are PUš- and
LUL-.
The thesaurus would have multiple entries for each lemma, and they would be
ordered by descending frequency (if the group actually made a prototype, they
would have realized that this simple frequency count was not enough to avoid
only the first meaning to be used). The dictionary entry for ČOVJE- (using
modern JSON notation) is:
”ČOVJE-”: ”mankind”: 193.5: ”LITTLENESS”, 690.2: ”AGENT”, ”man”: 554.4:
”REPRESENTATION”, 372.1: ”MANKIND”, 372.3: ”MANKIND” …, …
The meaning of the numbers used is never explained, but they would probably be
used for cross-referencing word categories.
After all the lemmas comprising the sentence have been looked up in this
dictionary, the next step is to keep only the inner values and discard the
inner keys, thus collapsing the list, so that the example above would become:
”COVJE-”: 193.5: ”LITTLENESS”, 690.2: ”AGENT”, 554.4: ”REPRESENTATION”, 372.1:
”MANKIND”, 372.3: ”MANKIND” …
Next, the most frequently occurring meaning would be kept, but only if it
grammatically fits the final sentence. One can extrapolate that it is tacitly
assumed that the grammatical structure of the source language matches the
target language, and to do this, a kind of categorical grammar similar to
Lambek calculus [9] would have to be used. It seems that the Croatian group
was not aware of the paper by Lambek (but only of Bar-Hillel’s papers), so
they did not elaborate this part.

Finka [6] notes that Matković, in his dissertation from 1957, considered the
use of bigrams and trigrams to “help model the word context”. It is not clear
whether Finka means character bigrams, which was computationally feasible at
the time, or word bigrams, which was not feasible, but the suggestion of
modelling the word context does point in this direction. Even though the
beginnings of using character bigrams can be traced back to Claude Shannon
[26], using character-level bigrams in natural language processing was studied
extensively only by Gilbert and Moore [15]. It can be argued, that in a sense,
Matković predated these results, but his research and ideas were not known in
the west, and he was not cited. The successful use of word bigrams in text
classification had to wait until [12]. The long time it took to get from
character to words was mainly due to computational limitations, but Matković’s
ideas are not to be dismissed lightly on account of computational complexity,
since the idea of using word bigrams was being explored by the Croatian
group–perhaps the reason for considering such an idea was the lack of a
computer and the underestimation of the memory requirements. The whole process
described above is illustrated in Fig. 1.

Several remarks are in order. First, the group seemed to think that encodings
would be needed, but it seems that entropy-based encodings and calculations
added no real benefits (i.e. added no benefit that would not be offset by the
cost of calculating the codes). In addition, Finka and Laszlo [7] seem to
place great emphasis on lemmatization instead of stemming, which, if they had
constructed a prototype, they would have noticed it to be very hard to tackle
with the technology of the age. Nevertheless, the idea of proper lemmatization
would probably be replaced with moderately precise hard-coded stemming, made
with the help of the ”inverse dictionary”, which Finka and Laszlo proposed as
one of the key tasks in their 1962 paper. This paper also highlights the need
for a frequency count and taking only the most frequent words, which is an
approach that later became widely used in the natural language processing
community. Sentential alignment coupled with part-of-speech tagging was
correctly identified as one of the key aspects of machine translation, but its
complexity was severely underestimated by the group. One might argue that
these two modules are actually everything that is needed for a successful
machine translation system, which shows the complexity of the task.

As noted earlier, the group had no computer available to build a prototype,
and subsequently, they have underestimated the complexity of determining
sentential alignment. Sentential alignment seems rather trivial from a
theoretical standpoint, but it could be argued that machine translation can be
reduced to sentential alignment. This reduction vividly suggests the full
complexity of sentential alignment. But the complexity of alignment was not
evident at the time, and only several decades after the Croatian group’s
dissolution, in the late 1990s, did the group centered around Tillmann and Ney
start to experiment with statistical models using (non-trivial) alignment
modules, and producing state-of-the-art results (cf. [28]) and [18]. However,
this was statistical learning, and it would take another two decades for
sentential alignment to be implemented in cybernetic models, by then known
under a new name, deep learning. Alignment was implemented in deep neural
networks by [2] and [1], but a better approach, called attention, which is a
trainable alignment module, was being developed in parallel, starting with the
seminal paper on attention in computer vision by [14].

##  4 Conclusion

At this point, we are leaving the historical analysis behind to speculate on
what the group might have discovered if they had had access to a computer.
First of all, did the Croatian group have a concrete idea for tackling
alignment? Not really. However, an approach can be read between the lines of
primarily [10] and [22]. In [22], Pranić addresses the Soviet model by
Andreev, looking at it as if it was composed of two modules – an understanding
module and a generation module. Following the footsteps of Andreev, their
interaction should be over an idealized language. Laszlo [10] notes that such
an idealized language should be encoded by keeping the entropy in mind. He
literally calls for using entropy to eliminate redundancy while translating to
an artificial language, and as Mulić notes [16], Andreev’s idea (which should
be followed) was to use an artificial language as an intermediary language,
which has all the essential structures of all the languages one wishes to
translate.

The step which was needed here was to eliminate the notion of structure
alignment and just seek sentential alignment. This, in theory, can be done by
using only entropy. A simple alignment could be made by using word entropies
in both languages and aligning the words by decreasing entropy. This would
work better for translating into a language with no articles. A better
approach, which was not beyond the thinking of the group since it was already
proposed by Matković in his dissertation from 1957 [6], would be to use word
bigrams and align them. It is worth mentioning that, although the idea of
machine translation in the 1950s in Croatia did not have a significant
influence on development of the field, it shows that Croatian linguists had
contemporary views and necessary competencies for its development. But,
unfortunately, the development of machine translation in Croatia had been
stopped because of the previously discussed circumstances. In 1964, Laszlo
went to the USA, where he spent the next seven years, and after returning to
Croatia, he was active as a university professor, but because of disagreement
with the ruling political option regarding Croatian language issues, he
published very rarely and was mainly focused on other linguistic issues in
that period, but his work was a major influence on the later development of
computational linguistics in Croatia.

## References

  * [1] T. Alkhouli, G. Bretschner, J.-T. Peter, M. Hethnawi, A. Guta, and H. Ney.  Alignment-based neural machine translation.  In O. Bojar, Ch. Buc, and et al, editors, Proceedings of the First Conference on a Machine Translation, Volume 1: Research Papers, WMT 2016, Berlin, Germany, August 7-12, 2016, pages 54–65. Berlin: Association for Computational Linguistics, 2016.
  * [2] D. Bahbanau, K. H. Cho, and Y. Bengio.  Neural machine translation by jointly learning to align and translate.  In International Conference on Learning Representations, ICLR 2015, San Diego, CA, 2015.
  * [3] Y. Bar-Hillel.  The present state of research on mechanical translation.  American Documentation, 2:229–236, 1951.
  * [4] Y. Bar-Hillel.  Machine translation.  Computers and Automation, 2:1–6, 1953.
  * [5] R. Descartes.  Selected Correspondence of Descartes.  Cambridge:: Early Modern Philosophy, 2017.
  * [6] B. Finka.  Odjeci sp u jugoslaviji.  In B. Laszlo and S. Petrovic, editors, Strojno prevodenje i statistika u jeziku, pages 249–260. Zagreb: Naše teme, 1959.
  * [7] B. Finka and B. Laszlo.  Strojno prevodenje i naši neposredni zadaci [machine translation and our immediate tasks].  Jezik, 10:117–121, 1962.
  * [8] J. Hutchins.  Yehoshua bar-hillel. a philosopher’s contribution to machine translation.  In J. Hutchins, editor, Early years in machine translation, pages 299–312. Amsterdam: John Benjamins, 2000.
  * [9] J. Lambek.  The mathematics of sentence structure.  Amer. Math. Monthly, 65:154–170, 1958.
  * [10] B. Laszlo.  Broj u jeziku.  In B. Laszlo and S. Petrovic, editors, Strojno prevodenje i statistika u jeziku, pages 224–239. Zagreb: Naše teme, 1959.
  * [11] B. Laszlo and S. Petrovic.  Uvod.  In B. Laszlo and S. Petrovic, editors, Strojno prevodenje i statistika u jeziku, page 105–298. Zagreb: Naše teme, 1959.
  * [12] D. D. Lewis.  An evaluation of phrasal and clustered representations on a text categorization task.  In N. J. Belkin, P. Ingwersen, , and A. M. Pejtersen, editors, Proceedings of SIGIR ’92, 15th ACM International Conference on Research and Development in Information Retrieval, SIGIR ’92, Copenhagen, Denmark, June 21 \- 24, 1992, pages 37–50. New York: ACM Press, 1992.
  * [13] W. McCulloch and W. Pitts.  A logical calculus of ideas immanent in nervous activity.  Bulletin of Mathematical Biophysic, 5:115–133, 1943.
  * [14] V. Mnih, N. Heess, A. Graves, and K. Kavukcuoglu.  Recurrent models of visual attention.  In Z. Ghahramani and et al, editors, NIPS’14 Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS 2014, Montreal, Canada, December 08 - 13, 2014, pages 2204–2212. Cambridge, MA: MIT Press, 2014.
  * [15] E. N. Gilbert; E. F. Moore.  Variable-length binary encodings.  The Bell System Technical Journal, 38:933–967, 1959.
  * [16] M. Mulic.  Sp u sssr.  In B. Laszlo and S. Petrovic, editors, Strojno prevodenje i statistika u jeziku [Machine Translation and Statistics in Language], pages 213–221. Zagreb: Naše teme, 1959.
  * [17] P. Norvig.  Paradigms of Artificial Intelligence Programming.  San Francisco: Morgan Kaufmann, 1991.
  * [18] F. J. Och, Ch. Tillmann, and H. Ney.  Improved alignment models for statistical machine translation.  In H. Schiitze and K-Y. Su, editors, Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, SIGDAT 1999, College Park, MD, USA, June 21-22 1999, pages 20–28. Stroudsburg, PA: Association for Computational Linguistics, 1999.
  * [19] S. Petrovic.  Može li stroj prevoditi poeziju.  In B. Laszlo and S. Petrovic, editors, Strojno prevodenje i statistika u jeziku, pages 177–197. Zagreb: Naše teme, 1959.
  * [20] R. Piotrowski and Y. Romanov.  Machine translation in the former soviet union and in the newly independent states.  Histoire Epistemologie Langage, 21:105–117, 1999.
  * [21] B. Pogbrelec.  Počeci rada na sp.  In B. Laszlo and S. Petrovic, editors, Strojno prevodenje i statistika u jeziku, pages 222–223. Zagreb: Naše teme, 1959.
  * [22] K. Pranjic.  Suvremeno stanje sp.  In B. Laszlo and S. Petrovic, editors, Strojno prevodenje i statistika u jeziku, pages 224–239. Zagreb: Naše teme, 1959.
  * [23] H. Putnam.  Mathematics, Matter, and Method: Philosophical Papers, Vol. 1. Cambridge, MA: The MIT Press, 1979.
  * [24] S. Russell and P. Norvig.  Artificial Intelligence, a Modern Approach.  New York: Pearsons, 2009.
  * [25] F. Baader; I. Horrocks; U. Sattler.  Description logics.  In F. van Harmelen, V. Lifschitz, and B. Porter, editors, Handbook of Knowledge Representation, pages 135–180. Oxford: Elsevier, 2007\.
  * [26] C. E. Shannon.  A mathematical theory of communication.  The Bell System Technical Journal, 27:379–423, 1948.
  * [27] L. Spalatin.  Rječnik sinonima kao jezik posrednik.  In B. Laszlo and S. Petrovic, editors, Strojno prevodenje i statistika u jeziku, pages 240–248. Zagreb: Naše teme, 1959.
  * [28] S. Vogel, S. H. Ney, and H. C. Tillmann.  Hmm-based word alignment in statistical translation.  In J. Tsujii, editor, 16th International Conference on Computational Linguistics, Proceedings of the Conference, COLING ’96, Copenhagen, Denmark, August 05-09, 1996, pages 836–841. Copenhagen: Center for Sprogteknologi, 1996.
  * [29] N. Wiener.  Cybernetics: On Control and Communication in the Animal and the Machine.  Cambridge, MA: The MIT Press, 1948.


# AlloST: Low-resource Speech Translation without Source Transcription

###### Abstract

The end-to-end architecture has made promising progress in speech translation
(ST). However, the ST task is still challenging under low-resource conditions.
Most ST models have shown unsatisfactory results, especially in the absence of
word information from the source speech utterance. In this study, we survey
methods to improve ST performance without using source transcription, and
propose a learning framework that utilizes a language-independent universal
phone recognizer. The framework is based on an attention-based sequence-to-
sequence model, where the encoder generates the phonetic embeddings and phone-
aware acoustic representations, and the decoder controls the fusion of the two
embedding streams to produce the target token sequence. In addition to
investigating different fusion strategies, we explore the specific usage of
byte pair encoding (BPE), which compresses a phone sequence into a syllable-
like segmented sequence. Due to the conversion of symbols, a segmented
sequence represents not only pronunciation but also language-dependent
information lacking in phones. Experiments conducted on the Fisher Spanish-
English and Taigi-Mandarin drama corpora show that our method outperforms the
conformer-based baseline, and the performance is close to that of the existing
best method using source transcription.

Index Terms: speech translation, universal phone recognition, low-resource
learning

##  1 Introduction

Communication is the basic requirement of human daily life. However, more than
7,000 different languages are spoken in the world. Due to the growing demand
for international activities, it is very important to break the communication
barriers between different spoken languages. Unlike text-to-text machine
translation (MT), which takes text as input to generate text in another
language, the input of speech translation (ST) is just a speech signal stream.
Traditionally, the task of ST is divided into two connected processes:
automatic speech recognition (ASR) and MT [1]. The advantage of this cascading
approach is that it can make the best use of text and speech resources.
However, this approach is prone to error propagation, i.e., word errors
generated by ASR may lead to incorrect translations. Not only the source
speech and target translation, the training process of this approach also
requires the transcription of the source speech, which imposes restrictions on
all unwritten languages and some low-resource languages. Therefore, directly
translating the source speech to the target text is essential and has been
extensively developed in recent research [2, 3, 4]. Compared with the
cascading approach, the end-to-end approach avoids the problem of error-
propagation, reduces the inference time, and does not necessarily require
transcription of the source speech for training.

When the transcription of the source speech is available, the ST model that
implements the end-to-end method of joint training of ASR and MT components
can effectively translate the given source speech [5]. Source speech
recognition becomes a sub-task, which helps to obtain more phonetically
informative embeddings of the source speech. Moreover, the jointly trained
model can provide good initial model parameters for subsequent fine-tuning of
the ST-only task [6, 7, 8, 9]. On the other hand, in the cascading approach,
it has been proved that the phone sequence of the source speech derived from
the word sequence output of a well-trained ASR system has better performance
than the word sequence in ST [10, 11]. In other words, phone features are a
better substitute for the textual resources of a language. Salesky and Black
[11] further proved that phone features are very effective for both cascading
and end-to-end ST approaches under low-resource conditions. The results leave
us certain things to imagine: Can we obtain reliable phonetic information of a
speech utterance of an unwritten language? This is one of our motivations for
this study.

As far as we know, the end-to-end ST approach achieves promising results
because there are a large amount of resources available for both speech and
text. Although these methods work well under high-resource conditions, they
still fail in low-resource and unwritten languages. Due to the difficulty of
collecting the language-dependent lexicon and phone inventory, any method that
uses text resources is unrealistic for low-resource and unwritten languages.
For these languages, text resources usually do not exist or are in poor
condition. For example, over 3,000 languages have not yet developed a written
form, but are still in use. Therefore, how to directly translate and document
low-resource unwritten languages becomes a problem we are trying to solve [12,
13, 14]. In accordance with the actual situation, we follow the restrictions
to simulate a real-world low-resource unwritten language in our ST task: 1)
any lexicons, phone inventories, pre-trained ASR and MT models of the source
language cannot be used; 2) only approximately 40 hours of unlabeled source
speech are available; 3) external language-independent resources can be used.

Therefore, in this study, we propose a hybrid learning framework that does not
require any source language-dependent resources, such as word or phone
transcription of speech, lexicons, and phone inventories. In [15], Li et al.
proposed Allosaurus, a neural system for extracting a universal phone sequence
from a speech utterance. Allosaurus was built using the PHOIBLE dataset, a
curated phone dataset containing more than 2,000 languages. Owing to its
comprehensiveness and validity for the phones of various languages, we use it
as our phone recognizer for extracting phone features, regardless of whether
the decoded phone sequence is represented according to the phonetic
specification of the source language. Furthermore, we propose two methods to
integrate phone information into the acoustic features and the sequence-to-
sequence model for ST. Our experiments are conducted on two low-resource
corpora, namely the 40-hour subset of the Fisher Spanish-English corpus and
the 30-hour Taigi-Mandarin drama corpus. The results show that our method
outperforms the conformer-based end-to-end baseline [4], and the performance
is very close to that of the existing best approach using source
transcription. This work, named AlloST, is open-
sourced111https://github.com/jamfly/AlloST. The main contribution of this
study is two-fold. First, we confirm the potential of low-resource ST that
does not require source transcription; to the best of our knowledge, all
existing cascading methods require it. Second, as an improvement of the end-
to-end approach, we prove that the segmentation of phone information is very
helpful to boost the performance of ST.

Figure 1: The architecture of the encoder in the proposed model.

##  2 Proposed Model: AlloST

###  2.1 Allosaurus

The traditional phone recognition method either recognizes speech to the
international phonetic symbols (e.g., IPA) of all languages or the phones of a
specific language. Allosarus, a language-independent phone recognizer [15],
aims to increase the phone recognition accuracy under low-resource conditions.
Allosarus leverages knowledge from other languages to predict the phone
inventory of an unknown language. Unwritten languages may not have a pre-
defined phone inventory; however, spoken units are language-independent, and
they are just symbols of pronunciation. Allosaurus meets our need because it
can leverage the phone inventories across 2,000 different languages to
recognize speech in any language to language-independent or language-dependent
symbols.

Allosaurus has two execution modes: 1) producing language specific phone
sequences for known languages; 2) producing the IPA sequence for any language
(the default setting) with the phone inventory consisting of approximately 230
phones. In our experiments, we adopt the default setting because we assume
that the source language is unknown. Because Allosaurus’s training data
contain a Spanish corpus, it produces more accurate phone recognition results
for Spanish than Taigi. This difference is reflected in our experimental
results.

###  2.2 Methods

####  2.2.1 Phone features

We assume that phone labels can be used not only to express pronunciation, but
also to convey a certain level of semantics. Based on the idea that the
combination of phone labels can be regarded as a morpheme of the language, we
adopt BPE to segment a phone sequence to find possible morpheme units [16].
Originally, BPE was used to solve the problem of out-of-vocabulary (OOV).
However, it can also efficiently capture the relationship between
pronunciation and semantics within the corpus. We use trainable embeddings
with positional encoding to transfer semantic information from raw speech to
high-level presentations, and then fuse them with acoustic features. We will
introduce the strategies of integrating these presentations into existing
models in the following sections.

Figure 2: The architecture of the decoder in the proposed model.

####  2.2.2 Encoder fusion

As shown in Figure 1, we use two separate conformer encoders [4] to model
phone and acoustic features. Given a source speech utterance, Encoder 1 takes
the phone sequence as input while Encoder 2 takes the acoustic feature
(filter-bank) sequence as input.

Salesky et al. [10, 11] proposed two methods to integrate phone features and
acoustic features. The first one is to concatenate the trainable phone
embeddings and acoustic embeddings, and the second one is to use the phone
boundaries to segment and compress the acoustic embeddings. However, both
methods are not applicable here because Allosaurus’s phone sequence output
does not contain time information. To solve the problem, we propose to use
multi-head attention to learn the alignment between phone features and
acoustic features [17].

####  2.2.3 Decoder fusion

Inspired by recent research on using a second decoder to do post-editing [18,
19, 20, 21, 22], we use a similar structure to achieve the goal of
proofreading. As shown in Figure 2, we use the basic setting of the
transformer decoder [17], but add an additional stacked multi-head attention
layer after the original multi-head attention layer to deal with the phone
embedding of the source speech. We believe that the phone representation can
provide detailed and supplementary information about the source speech, and
when it is fused with the resulting representation after the first attention
layer that deals with the acoustic representation, it can correct “mishearing”
errors. The decoder generates one token at a time, and then feeds that token
back to the decoder to generate the next token until the stop symbol appears.
In addition to the stacked multi-head attention, we also tried to use the gate
(type (b)) in [23] to fuse phone and acoustic features; however, it did not
improve the quality of translation.

Table 1: BLEU scores on the Fisher Spanish-English corpus with BPE 1k. ($\star$) indicates that source transcription is used, and ($\ddagger$) means that additional Wikipedia resources are used. Method | fisher-dev | fisher-dev2 | fisher-test
---|---|---|---
Conformer [4] | 20.84 | 21.49 | 21.06
$\star$ Conformer w/ MTL [24] | 31.21 | 32.59 | 31.47
$\star$ phone Segment [10] | 21.00 | N/A | 19.80
$\star$ phone Concat. [11] | 34.50 | N/A | 33.00
$\ddagger$ Word Embedding [7] | 23.54 | N/A | 21.72
Encoder Fusion | 29.52 | 30.01 | 29.01
Decoder Fusion | 26.32 | 27.14 | 26.19
Encoder + Decoder Fusion | 28.95 | 30.18 | 29.49

##  3 Experimental Setup

###  3.1 Datasets

####  3.1.1 Fisher Spanish-English corpus

The Fisher Spanish-English speech translation corpus (LDC2014T23) is a widely
used corpus for ST research [25]. The corpus contains 160 hours of Spanish
telephone speech, including 138k utterances. All 160 hours of speech have
corresponding transcription and English translation. We selected 40 hours of
speech from the corpus to conduct experiments that simulate low-resource
conditions. We used the standard dev, dev2, and test sets to evaluate our
methods, each set containing approximately 4k utterances.

####  3.1.2 Taigi-Mandarin drama corpus

Taigi, also known as Taiwanese Minnan, is a variant of Southern Min spoken in
Taiwan. There are two main types of writing systems, “Pinyin” and “Han-Luo”.
However, most people are not familiar with any Taigi writing system. According
to the current situation, Taigi can be regarded as an unwritten language. Chen
et al. collected 9 TV series of Taigi dramas with Mandarin Chinese subtitles.
It contains about 870 hours of speech, of which 30 hours of speech are
manually labeled with Mandarin Chinese translation [26]. We used the 30-hour
manually labeled speech as the training set, and used approximately 6,000
speech utterances from different dramas as the test set.

###  3.2 Pre-processing

We followed the ESPnet-ST recipe to extract 80 log-mel filterbank coefficients
and 3-dimensional pitch features using the Kaldi toolkit as the acoustic
feature [24, 27]. Speed perturbation with the factor of 3 was applied to the
training source speech [28]. The punctuation marks in the translation were
removed. The training pairs with target translation longer than 400 characters
or source speech longer than 3,000 frames were discarded. Allosaurus was
applied to generate the phone sequence from source speech. BPE was applied to
both the source phone sequence and the target translation. For the two target
languages, the vocabulary size of BPE was fixed at 1k. For the phone sequences
of the two source languages, the vocabulary size ranged from 1k to 64k. BPE-
Dropout with dropout-rate 0.1 was only applied to the phone sequences [29].

###  3.3 Implementation Details

The conformer architecture implemented in ESPnet-ST was used as the baseline
system [24]. The encoder consists of 12 layers, and the decoder consists of 6
layers. We used 4-head self-attention with the dimension of 256 [17]. The
dimension of feed-forward was set to 2,048, and label-smoothing and dropout
were set to 0.1. The Adam optimizer [30] and Noam learning rate schedule [17]
with default settings were used. The proposed model was implemented on top of
the baseline. Different from the baseline system, dropout and label-smoothing
were set to 0.3. The additional encoder (Encoder 1 in Figure 1) consists of 6
layers. An additional stacked 4-head attention layer was used in the decoder.

For evaluation, the model parameters corresponding to the 5 best checkpoints
based on the performance of the validation set were averaged. For the Fisher
Spanish-English corpus, we reported the 4-reference BLEU score [25] because
there are 4 reference translations for each source utterance.

##  4 Experimental Results

Table 2: BLEU scores on the Taigi-Mandarin drama corpus with BPE 1k. Method | taigi-test | BLEU $\triangle$
---|---|---
Conformer | 18.01 |
Encoder Fusion | 22.69 | +4.68
Decoder Fusion | 22.94 | +4.93
Encoder + Decoder Fusion | 22.58 | +4.57
Table 3: The results of different ST models for two examples in the Fisher Spanish-English corpus. The text in green stands for correct translations in this example, and there are no missing words in other methods. The text in blue stands for correct translations in this example, but there are missing words in other methods. The text in red stands for wrong translations.  Method | Example 1 | Example 2
---|---|---
Ground Truth |  you buy music or have many cds |  now that’s going to be in a very little time my friend
Conformer |  that eh do you buy music or do you have lots | they’re doing it in puerto rico
Conformer w/ MTL |  that uh do you buy music or has a lot of cds |  it’s going to be a little time ago
Encoder + Decoder Fusion w/o BPE |  what eh you buy music or do you have much difficult |  it’s going to be a long time for me
Encoder + Decoder Fusion w/ BPE 1k |  that eh do you buy music or have many cds |  it’s going to be a little time for me
Table 4: Impact of BPE on the BLEU scores on the Fisher Spanish-English corpus. Method | fisher-dev | fisher-dev2 | fisher-test |  BLEU $\triangle$
---|---|---|---|---
w/o BPE | 26.52 | 26.92 | 26.12 |
BPE 1k | 28.95 | 30.18 | 29.49 | +3.33
BPE 10k | 29.26 | 29.90 | 29.37 | +3.25
BPE 16k | 28.89 | 30.18 | 29.73 | +3.61
BPE 32k | 29.19 | 31.05 | 30.30 | +4.18
BPE 48k | 29.37 | 30.26 | 29.70 | +3.58
BPE 64k | 29.44 | 30.73 | 29.12 | +3.00

###  4.1 Main Results

The BLEU scores evaluated on the Fisher Spanish-English corpus are shown in
Table 1. The proposed models (“Encoder Fusion”, “Decoder Fusion” and “Encoder
+ Decder Fusion”) are compared with several models including models that
leverage extra resources. “Conformer” was trained in an end-to-end manner
without using any text resources of the source language [4]. “Conformer w/
MTL” [24] was trained with multi-task learning of hybrid CTC/Attention ASR and
MT models [31], where the transcription of Spanish speech was used to train
the ASR model. “phone Segment” [10] and “phone Concat.” [11] were trained with
additional Spanish text resources (LDC96L16). “Word Embedding” [7] used
FastText that was pre-trained on Wikipedia word embeddings.

From Table 1, we can see that our models consistently outperform the baseline
conformer model (“Conformer”) on all the three test sets, e.g., the BLEU score
was improved from 21.06% to 29.49% on the fisher-test set. Our models also
outperform “phone Segment”, which used additional Spanish text resources, and
“Word Embedding”, which used pre-trained word embeddings as extra information.
We believe that compared with word embeddings, phone features are closer to
acoustic features, and it is rather easy to combine phone features with
acoustic features. Obviously, our models are slightly worse than “phone
Concat.” and “Conformer w/ MTL”. However, “phone Concat.” used segmented phone
features derived from ASR, while “Conformer w/ MTL” used the transcription of
the source speech to jointly train the ASR and MT models. The results show
that segmented phone features are more effective.

###  4.2 Ablation Studies

In order to further understand which strategy is more suitable for fusing
phone features and acoustic features, we conducted an ablation study. From
Table 1, the results show that fusing phone features and acoustic features in
both Encoder and Decoder at the same time is generally more effective than
fusing phone features and acoustic features in Encoder or Decoder alone.
Fusing the phone features with the acoustic features and the target tokens
that have been translated in Decoder alone are not as effective as fusing
phone features and acoustic features in Encoder alone. We guess this is
because the phone features are similar to the acoustic features in Encoder.
Therefore, fusing the phone features and the acoustic features in Encoder is
easier to find the alignment, so as to obtain better results.

The results in Table 1 indicate that the effectiveness of using the phone
features in Decoder to correct the translation is limited because the source
phone features and the target tokens are quite different. However, this may
only happen in the case when the source language and the target language are
very different. The second task of this study is ST for the Taigi-Mandarin
pair. The results are shown in Table 2. Taigi and Mandarin share many
vocabularies and have similar grammar. We can think of this task as
translation between two similar languages. From Table 2, we can see that
“Decoder Fusion” is more effective than “Encoder Fusion”, and is even more
effective than the combination of the two. The results show that using the
Taigi phone representations to correct the translated Mandarin text is an
effective way to improve the quality of translation.

###  4.3 Examples

Table 3 shows the translation results of different models for two examples in
the fisher-test set. The first row is ground truth, the second and third rows
are the translations of “Conformer” and “Conformer w/ MTL”. In the first
example, the translation of “Conformer” missed the word “cds”, and our
proposed model “Encoder + Decoder Fusion w/ BPE 1k” could accurately translate
this word. “Conformer w/ MTL” caused grammatical errors (“have” vs. “has”),
and our proposed models “Encoder + Decoder Fusion w/o BPE” and “Encoder +
Decoder Fusion w/ BPE 1k” did not make this error.

In the second example, “Conformer” failed to translate the entire sentence.
“Conformer w/ MTL” incorrectly translated “ago”, and because of this error,
the meaning of the sentence is completely different. Our proposed model
“Encoder + Decoder Fusion w/ BPE 1k” obviously output a better translation.

###  4.4 BPE vs. Raw Phone Sequence

Motivated by previous work that segmented speech into a phone sequence
according to phone change boundaries [10], we applied BPE with BPE-Dropout of
dropout-rate 0.1 [16, 29] to reduce the phone sequence length. As the BPE unit
inventory increases from 230 (the size of IPA) to 1k $\sim$ 64k, the phone
sequence length can be reduced by about 30% $\sim$ 60%. We believe that
combining BPE-based phone features will enable the model to have a deeper
understanding of sentences.

In order to analyze the effect of BPE segmentation, we conducted experiments
using different sizes of BPE vocabulary. As shown in Table 4, BPE-based phone
features consistently outperform raw phone features. As shown in Table 3, the
model that fused the raw phone features with the acoustic features (cf.
“Encoder + Decoder Fusion w/o BPE”) mistranslated “many cds” to “much
difficult” in example 1 and “little time” to “long time” in example 2, but the
model that fused the BPE-based phone features with the acoustic features (cf.
“Encoder + Decoder Fusion w/ BPE 1k”) did not make such errors. One possible
reason is that fusing the raw phone features with the acoustic features will
make the model pay too much attention to the pronunciation and ignore the
meaning of the sentence, thereby resulting in inaccurate translation.

##  5 Conclusions

In this study, we propose a framework called AlloST, which shows that
language-independent phone features generated by a universal phone recognizer
can improve low-resource speech translation tasks. We also show that fusing
the phone features with the acoustic features in the encoder, decoder or both
can generally improve the performance, but the best setting depends on the
language pairs. Generally, BPE-based phone sequence segmentation is of great
benefit to performance. Our conformer-based model only uses an additional
universal phone recognizer (i.e., Allosaurus). But it is superior or
comparable to several conformer-based models that use more other resources.

##  6 Acknowledgments

This work was supported in part by MOST-Taiwan under Grant:
110-2634-F-008-004.

## References

  * [1] F. Casacuberta, M. Federico, H. Ney, and E. Vidal, “Recent efforts in spoken language translation,” _IEEE Signal Processing Magazine_ , vol. 25, no. 3, pp. 80–88, 2008.
  * [2] A. Berard, O. Pietquin, L. Besacier, and C. Servan, “Listen and translate: A proof of concept for end-to-end speech-to-text translation,” in _Proc. NeurIPS_ , 2016.
  * [3] R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu, and Z. Chen, “Sequence-to-sequence models can directly translate foreign speech,” in _Proc. Interspeech_ , 2017.
  * [4] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in _Proc. Interspeech_ , 2020.
  * [5] A. Anastasopoulos and D. Chiang, “Tied multitask learning for neural speech translation,” in _Proc. NAACL_ , 2018.
  * [6] Y. Liu, H. Xiong, Z. He, J. Zhang, H. Wu, H. Wang, and C. Zong, “End-to-end speech translation with knowledge distillation,” in _Proc. Interspeech_ , 2019.
  * [7] S.-P. Chuang, T.-W. Sung, A. H. Liu, and H.-y. Lee, “Worse WER, but better BLEU? Leveraging word embedding as intermediate in multitask end-to-end speech translation,” in _Proc. ACL_ , 2020.
  * [8] C. Wang, Y. Wu, S. Liu, M. Zhou, and Z. Yang, “Curriculum pre-training for end-to-end speech translation,” in _Proc. ACL_ , 2020.
  * [9] C. Wang, Y. Wu, S. Liu, Z. Yang, and M. Zhou, “Bridging the gap between pre-training and fine-tuning for end-to-end speech translation,” in _Proc. AAAI_ , 2020.
  * [10] E. Salesky, M. Sperber, and A. W. Black, “Exploring phoneme-level speech representations for end-to-end speech translation,” in _Proc. ACL_ , 2019\.
  * [11] E. Salesky and A. W. Black, “Phone features improve speech translation,” in _Proc. ACL_ , 2020.
  * [12] G. Adda, S. Stüker, M. Adda-Decker, O. Ambouroue, L. Besacier, D. Blachon, H. Bonneau-Maynard, P. Godard, F. Hamlaoui, D. Idiatov, G. N. Kouarata, L. Lamel, E. M. Makasso, A. Rialland, M. Van De Velde, F. Yvon, and S. Zerbian, “Breaking the unwritten language barrier: The BULB Project,” _Procedia Computer Science_ , vol. 81, no. May, pp. 8–14, 2016.
  * [13] M. Muller, J. Franke, A. Waibel, and S. Stuker, “Towards phoneme inventory discovery for documentation of unwritten languages,” in _Proc. ICASSP_ , 2017.
  * [14] M. Z. Boito, A. Berard, A. Villavicencio, and L. Besacier, “Unwritten languages demand attention too! Word discovery with encoder-decoder models,” in _Proc. ASRU_ , 2017.
  * [15] X. Li, S. Dalmia, J. Li, M. Lee, P. Littell, J. Yao, A. Anastasopoulos, D. R. Mortensen, G. Neubig, A. W. Black, and F. Metze, “Universal phone recognition with a multilingual Allophone system,” in _Proc. ICASSP_ , 2020\.
  * [16] A. B. Rico Sennrich, Barry Haddow, “Neural machine translation of rare words with subword units,” in _Proc. ACL_ , 2016.
  * [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proc. NeurIPS_ , 2017.
  * [18] H. Le, J. Pino, C. Wang, J. Gu, D. Schwab, and L. Besacier, “Dual-decoder Transformer for joint automatic speech recognition and multilingual speech translation,” in _Proc. COLING_ , 2020.
  * [19] W. Lee, J. Shin, and J.-h. Lee, “Transformer-based automatic post-editing model with joint encoder and multi-source attention of decoder,” in _Proc. WMT_ , 2019.
  * [20] T.-W. Sung, J.-Y. Liu, H.-y. Lee, and L.-s. Lee, “Towards end-to-end speech-to-text translation with two-pass decoding,” in _Proc. ICASSP_ , 2019\.
  * [21] M. Sperber, G. Neubig, J. Niehues, and A. Waibel, “Attention-passing models for robust and data-efficient end-to-end speech translation,” _Transactions of the Association for Computational Linguistics_ , vol. 7, no. 1, pp. 313–325, 2019.
  * [22] J. Shin and J.-h. Lee, “Multi-encoder Transformer network for automatic post-editing,” in _Proc. WMT_ , 2018.
  * [23] J. Libovický, J. Helcl, and D. Mareček, “Input combination strategies for multi-source Transformer decoder,” in _Proc. WMT_ , 2018\.
  * [24] H. Inaguma, S. Kiyono, K. Duh, S. Karita, N. Yalta, T. Hayashi, and S. Watanabe, “ESPnet-ST: All-in-one speech translation toolkit,” in _Proc. ACL_ , 2020.
  * [25] M. Post, G. Kumar, A. Lopez, D. Karakos, C. Callison-Burch, and S. Khudanpur, “Improved speech-to-text translation with the Fisher and CallHome Spanish–English speech translation corpus,” in _Proc. IWSLT_ , 2013.
  * [26] P.-Y. Chen, C.-H. Wu, H.-S. Lee, S.-K. Tsao, M.-T. Ko, and H.-M. Wang, “Using Taigi dramas with Mandarin Chinese subtitles to improve Taigi speech recognition,” in _Proc. O-COCOSDA_ , 2020.
  * [27] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, “The Kaldi speech recognition toolkit,” in _Proc. ASRU_ , 2011\.
  * [28] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition,” in _Proc. Interspeech_ , 2015.
  * [29] I. Provilkov, D. Emelianenko, and E. Voita, “BPE-dropout: Simple and effective subword regularization,” in _Proc. ACL_ , 2020.
  * [30] D. P. Kingma and J. L. Ba, “Adam: a method for stochastic optimization,” in _Proc. ICLR_ , 2015.
  * [31] S. Watanabe, T. Hori, S. Kim, J. R. Hershey, T. Hayashi, S. S. Member, T. Hori, S. S. Member, S. Kim, S. S. Member, J. R. Hershey, and S. S. Member, “Hybrid CTC/Attention architecture for end-to-end speech recognition,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 11, no. 8, pp. 1240–1253, 2017.