# Sequence-to-Sequence Resources for Catalan

###### Abstract

In this work, we introduce sequence-to-sequence language resources for
Catalan, a moderately under-resourced language, towards two tasks, namely:
Summarization and Machine Translation (MT). We present two new abstractive
summarization datasets in the domain of newswire. We also introduce a parallel
Catalan$\leftrightarrow$English corpus, paired with three different brand new
test sets. Finally, we evaluate the data presented with competing state of the
art models, and we develop baselines for these tasks using a newly created
Catalan BART. We release the resulting resources of this work under open
license to encourage the development of language technology in Catalan.

Keywords: Summarization, Machine Translation, Catalan, Under-Resourced
Languages

languageresourceLanguage Resources

Sequence-to-Sequence Resources for Catalan

Ona de Gibert, Ksenia Kharitonova, Blanca Calvo Figueras,
---
Jordi Armengol-Estapé, Maite Melero
Barcelona Supercomputing Center
Plaça Eusebi Güell 1-3, Barcelona 08034, Spain
{ona.degibert, ksenia.kharitonova, blanca.calvo, jordi.armengol,
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

In recent years, the arrival of the transformers [Vaswani et al., 2017b] has
changed the landscape of Natural Language Processing (NLP). The potential of
this new technology has opened up new lines of research with a clear focus on
under-resourced languages [Zoph et al., 2016], since it has been shown that
pre-trained language models such as BERT [Devlin et al., 2019] can
successfully solve downstream tasks with much less data than what was needed
before.

For this reason, academia has made an effort on developing language-specific
pre-trained language models [Martin et al., 2020] and evaluation benchmarks
[Canete et al., 2020]. While this field is still dominated by big languages,
we can see how this is starting to change [Wang et al., 2020].

We focus on Catalan, a moderately under-resourced language, for which there
already exists a large monolingual corpus, a pre-trained encoder-only language
model, and a Natural Language Understanding (NLU) benchmark [Armengol-Estapé
et al., 2021a]. We explore the task of Natural Language Generation (NLG) by
developing resources for two Sequence-to-Sequence tasks, namely, Summarization
and Machine Translation (MT). These are complex tasks that require an encoder-
decoder architecture.

In this paper, we present new resources and models, diverse in sizes and
domains. The public release of these resources will allow the Natural Language
Processing (NLP) community to explore their algorithms in depth, as well as
the cross-lingual transfer-learning capabilities of their models. Our
contributions sum up to:

  * •

Two new datasets for abstractive Summarization

  * •

A high-quality dataset for English$\leftrightarrow$Catalan MT

  * •

Three new MT testsets for the English$\leftrightarrow$Catalan pair, one of
which is multilingual

  * •

A Catalan BART

We make these resources openly available at Github.111https://github.com/TeMU-
BSC/seq-to-seq-catalan

The rest of the paper is organized as follows. Section 2 provides an overview
of the previous work done in the field. Section 3 describes in detail the
resources presented. Section 4 describes the experiments and results obtained,
and finally, section 5 concludes our work and opens new future lines of
research.

## 2\. Related Work

Language resources for summarization are difficult to obtain and thus are
often built with automated processes from news websites, as it is easy to
interpret the body of an article and its title. There exist several resources
for English, both extractive [Lins et al., 2019] and abstractive [Narayan et
al., 2018] and recently there has been an effort on developing multilingual
resources [Scialom et al., 2020], with a focus on big languages. For the case
of Catalan, a minority language, there exists only one resource provided by
?), who collected newspaper articles from different sources for Spanish and
Catalan and trained a Transformer encoder-decoder for both languages. For
Catalan they created DACSA, a text summarization dataset composed of 725,184
sample pairs from 9 newspapers. They benchmark their results with two well
known multilingual models: mBART [Liu et al., 2020] and mT5 [Xue et al., 2021]
to investigate whether monolingual encoder-decoders are more beneficial than
multilingual ones.

In the case of MT, there has been much work on developing resources for under-
resourced languages. One of the main repositories of parallel data for MT is
OPUS tiedemann2012parallel, which includes many multilingual parallel datasets
ranging in domains, languages and sizes. Catalan is included in many of these
large web-crawled datasets, however, as ?) point out, most data coming from
online sources is of poor quality. Hence, the importance of high quality data
curation. In this work, we focus on the language pair English-Catalan, for
which, even if English is a lingua franca, there are not many publicly
available parallel resources. These are mostly from OPUS or from
Softcatala,222softcatala.org an non-profit organization that works for the
development of technologies in Catalan.

Over the past few years, large Transformer-based [Vaswani et al., 2017a]
models have shown to yield the best results on the majority of the sequence-
to-sequence tasks. The most common approach towards developing such models has
been to pre-train them on a vast amount of rich non-parallel text data with a
variety of objectives [Devlin et al., 2019], and afterwards fine-tune them on
a small amount of appropriate data for a required downstream task. This
approach can be used both with monolingual corpora for primarily monolingual
tasks, and with multilingual data. Following this approach, denoising
autoencoders expanded on an original sequence-to-sequence Transformer and they
proved to be especially useful for summarization due to their denoising
objective [Lewis et al., 2019]. Multilingual BART (mBART), pre-trained on a
multilingual concatenated non-parallel data, further showed performance gain
in a low resource language setting for machine translation [Liu et al., 2020].

## 3\. Language Resources

### 3.1. Summarization

We introduce two new datasets for summarization in Catalan. CaSum, which can
be used for training and evaluation, and VilaSum, an out-of-distribution (with
respect to CaSum) test set.

##### CaSum

is a new summarization dataset. It is extracted from a newswire corpus crawled
from the Catalan News Agency.333https://www.acn.cat/

The corpus consists of 217,735 instances that are composed by the headline and
the body. We obtained each headline and its corresponding body and applied the
following cleaning pipeline: deduplicating the documents, removing the
documents with empty attributes, and deleting some boilerplate sentences.

Since most summarization datasets are built automatically, there’s little
control over the final quality result. We perform a manual evaluation of the
dataset by taking 10,000 random samples and assessing whether the headline is
a valid summary for the article. The validation reports that 99,02 % of the
cases are correct. Therefore, we consider our dataset of high quality.

##### VilaSum

is a smaller summarization dataset, of 13,843 samples, which can be used as an
out-of-distribution (with respect to CaSum) test set. It has been obtained
from the digital newspaper Vilaweb.444https://www.vilaweb.cat/ To obtain the
final dataset, we followed the same procedure as in the CaSum dataset.
However, because of its smaller size, we were able to manually validate all
the samples of the dataset. The actual crawling returned 15,019 headline-body
pairs and the manual revision discarded 1,176 pairs, where the headline was
not a valid summary of the article.

The splits for both datasets can be seen in Table 1. Table 2 better describes
each dataset in terms of article and summary length, as well as novelty (how
many words in the summary do not belong to the article), compression ratio,
and vocabulary size. We note that VilaSum has longer articles, with shorter
summaries and more novel words in the summaries, making this test more
challenging than the one in CaSum.

Dataset | Train | Valid | Test
---|---|---|---
CaSum | 197,735 | 10,000 | 10,000
VilaSum | - | - | 13,843
Table 1: Splits of the Summarization datasets | CaSum | VilaSum
---|---|---
Article avg. sentences | 11.69 | 29.34
Summary avg. sentences | 1.00 | 1.01
Article avg. words | 338.64 | 647.69
Summary avg. words | 16.42 | 12.89
Novelty | 19.66 | 25.75
Compression ratio | 21.25 | 57.7
Vocabulary Size | 1,052,211 | 380,146
Table 2:  Statistics of the Summarization datasets

### 3.2. Machine Translation

In order to create a large dataset for CA$\leftrightarrow$EN machine
translation, we compile all available open-source parallel bilingual
CA$\leftrightarrow$EN corpora, plus a brand new high quality dataset, gEnCaTa.
In total, we use 20 different datasets to obtain a moderately large bilingual
corpus CA$\leftrightarrow$EN resulting in a dataset of over 11.3 million
aligned sentences. We release openly the gEnCaTa corpus. The characteristics
of the corpora can be found in Table 3.

ID | Dataset | Sentences | Source | Domain
---|---|---|---|---
1 | CCaligned | 5,787,682 |  elkishky_ccaligned_2020 | General
2 | COVID-19 Wikipedia | 1,531 |  tiedemann2012parallel | Health
3 | CoVost en-ca | 79,633 |  wang2020covost | General
4 | CoVost ca-en | 263,891 |  wang2020covost | General
5 | Eubookshop | 3,746 |  tiedemann2012parallel | Legislation
6 | Europarl | 1,965,734 |  tiedemann2012parallel | Legislation
7 | Global Voices | 21,342 |  tiedemann2012parallel | Newswire
8 | Gnome | 2,183 |  tiedemann2012parallel | Software
9 | JW300 | 97,081 |  tiedemann2012parallel | General
10 | KDE4 | 144,153 |  tiedemann2012parallel | Software
11 | Memories Lliures | 1,173,055 | Softcatalà | General
12 | Open Subtitles | 427,913 |  lison-tiedemann-2016-opensubtitles2016 | General
13 | Books | 4,580 |  tiedemann2012parallel | Narrative
14 | QED | 69,823 |  abdelali-etal-2014-amara | Education
15 | Tatoeba | 5,500 |  tiedemann2012parallel | General
16 | Tedtalks | 50,979 | Softcatalà | General
17 | Ubuntu | 6,781 |  tiedemann2012parallel | Software
18 | Wikimatrix | 977,466 |  schwenk2019wikimatrix | Wikipedia
19 | Wikimedia | 208,073 |  tiedemann2012parallel | Wikipedia
20 | GEnCaTa | 38,595 | New | General
| Total | 11,329,741 |  |
Table 3: Language resources for Machine Translation Training.

The compiled datasets originate from different sources and belong to different
domains. Mostly they come from OPUS [tiedemann2012parallel] and
Softcatalà.555https://github.com/Softcatala/parallel-catalan-corpus Most
datasets belong to the general domain, although some sources originate from
software translations or Wikipedia articles. Nonetheless, we are aware that
the quality of the datasets varies greatly, since an automatic alignment and
manual revision yield very different results. CCaligned, for instance, has
been shown to have poor quality [Kreutzer et al., 2021].

##### gEnCaTa

is a Catalan$\leftrightarrow$English parallel corpus composed of 38,595
segments. It has been compiled by leveraging parallel data from crawling the
gencat.cat domain and subdomains, belonging to the Catalan Government, both in
English and Catalan.

We used the cleaning pipeline in ?) pipeline to process the WARC files
obtained from the crawling. This allowed us to maintain the metadata and
retrieve the original URL per each visited page. We extracted the content of
the URLs that had data crawled in both languages and obtained 4,416 comparable
sites, which we consider documents.

To align the sentences in the documents we used both an automated alignment
algorithm (vecalign666https://github.com/thompsonb/vecalign), and a manual
revision. After the automated alignment and sentence deduplication there were
51,447 parallel segments. However, after the manual revision only 38,595
remained. We did a further analysis of these differences just to notice that
only 19.8% of the 5,000 highest scored segments ranked by vecalign were also
selected after the manual revision. This posits the question of how much can
we rely on alignment algorithms for building parallel corpora by only looking
at the given score.

#### 3.2.1. Machine Translation Evaluation

We also provide three new datasets for MT evaluation.

##### WMT2013-ca

consists of the Catalan translation of the WMT 2013 translation shared task
test set [Bojar et al., 2013], belonging to the newswire domain. We
commissioned the translation from Spanish to Catalan to a professional native
translator.

##### taCon

is a multilingual test set from the legal domain that includes all the
languages in which the Spanish Constitution exists, namely, Basque, Catalan,
Galician, Spanish and English. To obtain it, we downloaded the Spanish
Constitution from the website of the Agencia Estatal del Boletín Oficial del
Estado777www.boe.es in the corresponding languages in PDF format. We converted
it to plain text, fixed the broken sentences and finally aligned the sentences
manually.

##### Cyber-ca

is a brand new test set in Catalan, Spanish and English that belongs to the
cybersecurity domain. It is composed of cybersecurity alerts extracted from
the INCIBE Spanish-English corpus.888https://www.elrc-
share.eu/repository/browse/descripciones-de-vulnerabilidades-de-la-bbdd-
nvd/9dd01c02cfcc11e9913100155d02670670070560807f458f8e50bd5273778d7a/

Dataset | Languages | Domain | Sent.
---|---|---|---
WMT13-ca | ca, es, en | newswire | 3,003
Cyber-ca | ca, en, es | cybersecurity | 1,715
taCon |  ga, eu, ca, es, en | legal | 1,110
Table 4: Language resources for Machine Translation Evaluation

## 4\. Experiments and Results

### 4.1. Summarization

We develop a Catalan BART-base [Lewis et al., 2019], using the Catalan Textual
Corpus [Armengol-Estapé et al., 2021a], henceforth BART-Ca. We fine-tune it
with the CaSum training and validation sets during circa 4 epochs, using
performance during the validation as a stop signal. We evaluate our model on
the CaSum test set and also on VilaSum. We benchmark our results with mBART
[Liu et al., 2020], also fine-tuning it with CaSum during approximately the
same amount of epochs; and with NASCA,999Due to 512 token size limit, we can
only evaluate 6,742 and 7,199 samples of each testset for CaSum and VilaSum,
respectively. the pre-trained Catalan language model by ?). We report our
results with ROUGE [Lin, 2004] in Table 5.

Test set | Model | ROUGE-1 | ROUGE-L
---|---|---|---
CaSum | BART-Ca | 41.39 | 36.14
NASCA | 24.42 | 19.89
mBART | 43.95 | 38.11
VilaSum | BART-Ca | 35.04 | 29.70
NASCA | 23.18 | 19.09
mBART | 33.17 | 27.52
Table 5: Summarization results on CaSum and VilaSum

BART-Ca obtains the best results on the VilaSum dataset, whereas mBART
performs better for the CaSum test set itself. Therefore, BART-Ca shows better
performance on the out-of-distribution data, despite being considerably
smaller.101010BART-base vs. BART large, that is, 110M vs. 340M non-embedding
parameters. This may be attributed to the fact that BART-Ca is trained on a
more optimized vocabulary (language-specific), and pre-trained on data of
better quality. Results show that the NASCA model significantly underperforms
both fine-tuned BART-Ca and mBART, both on the test CaSum and the VilaSum
datasets. This can be expected, as NASCA has not been fine-tuned, and the
distribution of the data it was trained on may differ from our test sets.

With regard to good performance on the test CaSum dataset, we can assume that
this dataset is just a continuation of the training and validation datasets
that were used to fine-tune the models and, therefore, is exactly in the same
domain and in the same style. However, the performance on the VilaSum dataset
shows that the quality of the data used to fine-tune the starting pre-trained
model is of tantamount importance.

### 4.2. Machine Translation

To test the new evaluation sets, we use two supervised neural MT models:
Google Translate (GT) 111111https://translate.google.es and
Softcatalà,121212https://www.softcatala.org/traductor/ the latter one is open-
source and has established itself as a reference in the Catalan community. As
a reference, we also include the test set of Flores-101 goyal2021flores, a
multilingual test set for MT benchmarking. We report BLEU scores [Post, 2018]
in Table 6.

Direction | Test set | GT | Softcatalà
---|---|---|---
en $\rightarrow$ ca | WMT13 | 33.8 | 34.3
TaCon | 37.6 | 31.8
Cyber | 46.5 | 42.1
Flores-101 | 42.2 | 41.5
ca $\rightarrow$ en | WMT13 | 39.8 | 37.6
TaCon | 43.2 | 35.2
Cyber | 58 | 49.9
Flores-101 | 46.9 | 42.4
Table 6: BLEU scores for MT evaluation

Results show that MT achieves overall good results for the
CA$\leftrightarrow$EN language pair, with more mediocre results in certain
evaluation sets. As expected, GT outperforms Softcatalà, but the latter is
still a competitive baseline.

TaCon is the test set with the lowest scores, probably because of its
restricted language-specific domain. Surprisingly, the Cyber test set seems to
be the easiest one, although it is domain-specific. This is because it
contains numerous non-verbal segments that are kept untranslated, which boosts
the results, achieving 58 BLEU for GT in the CA$\rightarrow$EN direction. Both
Flores-101 and WMT13, which belong to the more general domain, present almost
no variability in the models’ performance.

## 5\. Conclusions & Future Work

In this work we presented resources for two sequence-to-sequence tasks,
specifically, Summarization and Machine Translation, and for a moderately
under-resourced language, Catalan. We describe in detail two new high-quality
Summarization datasets and several resources to exploit MT in Catalan. We
further explore our results by building baselines and comparing them with
state of the art models. We expect to encourage the development of more
complex language technologies for this language. As future work, we plan to
develop new training and evaluation resources for Catalan, noting that the
generative scenario is the one that currently lacks data.

## 6\. Acknowledgements

This work was funded by the MT4All CEF
project.131313https://ec.europa.eu/inea/en/connecting-europe-facility/cef-
telecom/2019-eu-ia-0031

## 7\. Bibliographical References

## References

  * Ahuir et al., 2021 Ahuir, V., Hurtado, L.-F., González, J. Á., and Segarra, E.  (2021).  Nasca and nases: Two monolingual pre-trained models for abstractive summarization in catalan and spanish.  Applied Sciences, 11(21):9872.
  * Armengol-Estapé et al., 2021a Armengol-Estapé, J., Carrino, C. P., Rodriguez-Penagos, C., de Gibert Bonet, O., Armentano-Oller, C., González-Agirre, A., Melero, M., and Villegas, M.  (2021a).  Are multilingual models the best choice for moderately under-resourced languages? a comprehensive assessment for catalan.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4933–4946.
  * Armengol-Estapé et al., 2021b Armengol-Estapé, J., Carrino, C. P., Rodriguez-Penagos, C., de Gibert Bonet, O., Armentano-Oller, C., Gonzalez-Agirre, A., Melero, M., and Villegas, M.  (2021b).  Are multilingual models the best choice for moderately under-resourced languages? A comprehensive assessment for Catalan.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4933–4946, Online, August. Association for Computational Linguistics.
  * Bojar et al., 2013 Bojar, O., Buck, C., Callison-Burch, C., Federmann, C., Haddow, B., Koehn, P., Monz, C., Post, M., Soricut, R., and Specia, L.  (2013).  Findings of the 2013 Workshop on Statistical Machine Translation.  In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 1–44, Sofia, Bulgaria, August. Association for Computational Linguistics.
  * Canete et al., 2020 Canete, J., Chaperon, G., Fuentes, R., and Pérez, J.  (2020).  Spanish pre-trained bert model and evaluation data.  PML4DC at ICLR, 2020.
  * Devlin et al., 2019 Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  (2019).  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  arXiv:1810.04805 [cs], May.  arXiv: 1810.04805.
  * Kreutzer et al., 2021 Kreutzer, J., Caswell, I., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., Setyawan, M., Sarin, S., Samb, S., Sagot, B., Rivera, C., Rios, A., Papadimitriou, I., Osei, S., Suárez, P. O., Orife, I., Ogueji, K., Rubungo, A. N., Nguyen, T. Q., Müller, M., Müller, A., Muhammad, S. H., Muhammad, N., Mnyakeni, A., Mirzakhalov, J., Matangira, T., Leong, C., Lawson, N., Kudugunta, S., Jernite, Y., Jenny, M., Firat, O., Dossou, B. F. P., Dlamini, S., de Silva, N., Çabuk Ballı, S., Biderman, S., Battisti, A., Baruwa, A., Bapna, A., Baljekar, P., Azime, I. A., Awokoya, A., Ataman, D., Ahia, O., Ahia, O., Agrawal, S., and Adeyemi, M.  (2021).  Quality at a glance: An audit of web-crawled multilingual datasets.
  * Lewis et al., 2019 Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L.  (2019).  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  arXiv preprint arXiv:1910.13461.
  * Lin, 2004 Lin, C.-Y.  (2004).  Rouge: A package for automatic evaluation of summaries.  In Text summarization branches out, pages 74–81.
  * Lins et al., 2019 Lins, R. D., Oliveira, H., Cabral, L., Batista, J., Tenorio, B., Ferreira, R., Lima, R., de França Pereira e Silva, G., and Simske, S. J.  (2019).  The cnn-corpus: A large textual corpus for single-document extractive summarization.  In Proceedings of the ACM Symposium on Document Engineering 2019, pages 1–10.
  * Liu et al., 2020 Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., and Zettlemoyer, L.  (2020).  Multilingual denoising pre-training for neural machine translation.  Transactions of the Association for Computational Linguistics, 8:726–742.
  * Martin et al., 2020 Martin, L., Muller, B., Suárez, P. J. O., Dupont, Y., Romary, L., de la Clergerie, É. V., Seddah, D., and Sagot, B.  (2020).  Camembert: a tasty french language model.  In ACL 2020-58th Annual Meeting of the Association for Computational Linguistics.
  * Narayan et al., 2018 Narayan, S., Cohen, S. B., and Lapata, M.  (2018).  Don’t give me the details, just the summary!  Topic-aware Convolutional Neural Networks for Extreme Summarization. In.
  * Post, 2018 Post, M.  (2018).  A call for clarity in reporting BLEU scores.  In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium, October. Association for Computational Linguistics.
  * Scialom et al., 2020 Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., and Staiano, J.  (2020).  Mlsum: The multilingual summarization corpus.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8051–8067.
  * Vaswani et al., 2017a Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I.  (2017a).  Attention is all you need.  In I. Guyon, et al., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  * Vaswani et al., 2017b Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  (2017b).  Attention is all you need.  CoRR, abs/1706.03762.
  * Wang et al., 2020 Wang, Z., Karthikeyan, K., Mayhew, S., and Roth, D.  (2020).  Extending multilingual bert to low-resource languages.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 2649–2656.
  * Xue et al., 2021 Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C.  (2021).  mt5: A massively multilingual pre-trained text-to-text transformer.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498.
  * Zoph et al., 2016 Zoph, B., Yuret, D., May, J., and Knight, K.  (2016).  Transfer learning for low-resource neural machine translation.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575, Austin, Texas, November. Association for Computational Linguistics.

## 8\. Language Resource References

lrec2022-bib languageresource