# Facial Action Unit Recognition With Multi-models Ensembling

Wenqiang Jiang These authors contributed equally to this work and should be
considered co-first authors Beijing Seek Truth Data Technology Co.,Ltd.
Yannan Wu These authors contributed equally to this work and should be
considered co-first authors Beijing Seek Truth Data Technology Co.,Ltd.
Fengsheng Qiao These authors contributed equally to this work and should be
considered co-first authors Beijing Seek Truth Data Technology Co.,Ltd.  Liyu
Meng  Beijing Seek Truth Data Technology Co.,Ltd.  Yuanyuan Deng  Beijing Seek
Truth Data Technology Co.,Ltd.  Chuanhe Liu  Beijing Seek Truth Data
Technology Co.,Ltd.

##  1 Abstract

The Affective Behavior Analysis in-the-wild (ABAW) 2022 Competition gives
Affective Computing a large promotion. In this paper, we present our method of
AU challenge in this Competition. We use improved IResnet100 as backbone. Then
we train AU dataset in Aff-Wild2 on three pertained models pretrained by our
private au and expression dataset, and Glint360K respectively. Finally, we
ensemble the results of our models. We achieved F1 score (macro) 0.731 on AU
validation set.

##  2 Introduction

As an important part of Artificial Intelligence and Human Interaction,
affective computing has arising more and more attention. Meanwhile, it has
lots of applications in many fields, such as customer satisfaction survey,
financial anti-fraud, and psychological analysis, etc.

The 3th ABAW Competition 2022 is a large- scale in the wild emotion database
which is held by Dimitrios Kollias[8][9][12][11], etc. It provides Aff-Wild2
which consists of three emotional database including categorical expression
(such as happy, angry, sad) , valence arousal and 12 facial action units. Aff-
Wild2 has 564 videos downloaded from YouTube. There are variety in ethnics,
poses and ages, etc.[20][14][13][10][7]

Different from seven basic categorical expressions and valence arousal, action
units(AU) describe facial muscle movements developed by Paul Ekman in
1970s[5]. Action units usually have concurrence. For example, AU 25 (lips part
) an AU 26 (jaw drop) often occur at the same time.

In this paper, we address AU task in ABAW 2022. In section 3, we present our
methods of data balancing, model structure and loss function design. In
section 4, we give the details about the datasets we used, experiment settings
and our model ensembling strategy.

##  3 Method

###  3.1 Data balancing

Facial action unit recognition is a multi-label visual task in deep learning.
There usually exist label imbalance problem in multi-label task and data
balancing is very difficult because of label concurrence. Several papers are
proposed to solve this problem. Wu Tong designed loss functions[19] to solve
this problem. Other scientists use data sampling to alleviate data unbalance.

In this paper, we make Aff-Wild2 AU dataset more balanced with ML-ROS
method[2].Another method we try to alleviate data unbalance problem is batch
sampling. When we train a batch from the training set. We wish all 12 Action
Units (negative and positive ) are included in a batch. First, we organize the
training data with 12 Action Units labels. Then when training is on, we sample
each batch from the organized data. By doing this we can get a more balanced
dataset.

###  3.2 Model Structure

We use IResnet100[4] as the backbone. In order to better extract facial
feature information, We choose pretrained weights on the Glint360K
dataset[1].During the experiment, We found that increasing the texture feature
information of the face is helpful for the classification of AU. Due to the
increase of the network depth, the semantic features are more abundant but the
texture features will be lost, so we added the FPN and SSH modules [3]to
increase the texture information and receptive field of the face.see Figure 1.

At the same time, we flatten the features of each layer passing through
FPN[15] and SSH[18] modules, and splicing the features of each layer to output
512 dimensions of features through a fully connected layer.And in order to
make the network pay more attention to a certain part, we added the Coordinate
Attention module[6] to the shallow and deep layers of the network.The
experimental results show that the feature information obtained in this way
contains more texture features, and the classification effect is better than
the previous AU classification.

Figure 1: Overview system of proposed method

###  3.3 Loss Function

For AU dataset in Aff-Wild2, we have counted the numbers of each AU in the
training set and verification set, and the distribution of data set is shown
in Figure 2.

Figure 2: Numbers of positive and negative AU in both training set and
validation set.

As shown in Table 1, the data distribution of training set and verification
set is extremely imbalanced, this will make AU15,AU23 and AU24 difficult to
train. Because Action Unit Detection is a multi-label problem, Data Augment
cannot solve the data imbalance problem. Therefore, we try to solve this
problem from loss function.

$bce\\_loss(x,y)=\mathrm{L}=\\{\mathrm{l}_{1},...,\mathrm{l}_{N}\\}.$ (1)

where L represents the sum of the 12 AU, N represents the number of AU.

$l_{i}=-\mathrm{w_{n}}[y_{i}\cdot\log\sigma(x_{i})+(1-y_{i})\cdot\log(1-\sigma(x_{i}))$
(2)

$W=\\{\mathrm{w}_{1},...,\mathrm{w}_{N}\\}$ (3)

$\sigma(x_{i})=\frac{1}{1+\mathrm{e}^{-x_{i}})}^{T}$ (4)

(2) is binary classification loss function, where W represents the loss weight
of each AU, in our method, W = [1, 2, 1, 1, 1, 1, 1, 6, 6, 5, 1, 5], where x
represents softmax output and in [0, 1] as (1), where y represents the target
and takes either 0 or 1.

$multi\\_label\\_loss(x,y)=-\mathrm{w_{n}}*\sum_{i}y[i]*\log((1+\exp(-x[i]))^{-1})+(1-y[i])\log\left(\frac{\exp(-x[i])}{(1+\exp(-x[i]))}\right)$
(5)

(5) is multi_label loss, where x represents softmax output and in [0, 1],
where y represents the target and takes either 0 or 1.

$total\\_loss=multi\\_label\\_loss(x,y)+ce\\_loss(x,y)$ (6)

Finally, add bce_loss and multi_label_loss together, as (6).

###  3.4 Post Processing

Considering this au challenge needs video sequence predictions, we smooth the
logits generated by the last layer of the network with a sliding window.

##  4 Experiments

###  4.1 Dataset

Three datasets are used to train the backbone network as pretrained models
respectively:

Glint360K: which is the largest and cleanest face recognition dataset and
contains 170M images of 360k IDs, baseline models trained on Glint360K can
easily achieve excellent performance.

Private commercial EXPR dataset: which contains 7K high definition images,
each of which is human annotated into one of 7 facial expression
categories(Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger and Other).

Private commercial AU dataset: which contains 7K high definition images, each
of which is human annotated into 15 face action unit categories(AU1, AU2, AU4,
AU5, AU6, AU7, AU9, AU10, AU11, AU12, AU15, AU17, AU20, AU24 and AU26).

After we get the pretrained models trained from the above three datasets, we
train AU dataset in Aff-Wild2[13] to get our final results.

###  4.2 Training and testing

In experiments, we use Iresnet100 to implement our framework. Our framework
input size is 112x112. The SGD optimizer is used with a learning rate of
0.001, momentum of 0.9 and weight decay of 5e-4 and with a batch size of 256.
The total training epoch is set as 15 in the ABAW training dataset. The
learning rate will be divided by a factor of 10 when training to the 4/6/8
epoch. The data augment we implement Color Jitter(30% chance of brightness,
30% chance of contrast, 30% chance of saturation and 30% chance of hue),
Random Horizontal Flip. Dropout of 0.6 is applied. All framework have been
implemented in PyTorch, and training on 4 RTX-3090 GPU.

r100 | glint | bal | mll | ca | sa | ls | b+m | fpn | bs256 | ssh | data | f1
---|---|---|---|---|---|---|---|---|---|---|---|---
✓ |  |  |  |  |  |  |  |  |  |  |  | 0.390
✓ | ✓ |  |  |  |  |  |  |  |  |  |  | 0.534
✓ | ✓ | ✓ |  |  |  |  |  |  |  |  |  | 0.549
✓ | ✓ | ✓ | ✓ |  |  |  |  |  |  |  |  | 0.570
✓ | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  |  |  | 0.614
✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  |  | 0.662
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  | 0.673
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  |  |  |  | 0.690
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  |  |  | 0.709
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  |  | 0.712
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  | 0.715
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 0.721
Table 1: All the effective methods are compared on the validation set.

###  4.3 Ablation study

We begin our oblation study by exploring the effectiveness of difference
operation in our framework(Table 1). All experiments are based on
Iresnet100(r100).Using glint360(glint) dataset pretrain can improve by 14.4%,
using data balance(ba): oversampling AU2/AU15/AU23/AU24/AU26 and downsampling
AU1/AU4/AU6/AU7/AU10/AU12/AU25 can improve by 1.5%, using multi label
loss(mll) can improve by 2.1%, adding coordinate attention(ca) module can
improve 4.4%, using label smooth(ls) can improve 4.8%, using bce loss + multi
label loss(b+m) can improve 1.1%, using feature pyramid networks(fpn) can
improve 1.9%, using bigger batch size 256 (bs256) can improve 0.3%, using
Single Stage Headless(shh) can improve 0.3%, adding additional AU data can
improve 0.6%.

###  4.4 Ensemble Model

In previous competitions[17][16] we have used ensemble. In numerous
experiments, we also adopted a model ensemble strategy, which ensemble the
model with the highest F1 scale for each AU in the experiment, and obtained a
higher F1 score on the validation set.see Figure 3.

Figure 3: F1 score results for each AU

##  5 Conclusion

For AU task in ABAW Competition 2022, we train the backbone on three different
datasets and get three pretrained models respectively. Using multi label loss
function, ML-ROS and batch sampling, the problem of data imbalancing is
alleviated. Then we train Aff-Wild2 AU dataset on the three pretrained models.
Finally, best checkpoint for each AU and each model are chosen and ensembled.

## References

  * [1] Xiang An et al.  “Partial FC: Training 10 Million Identities on a Single Machine”  In _Arxiv 2010.05222_ , 2020
  * [2] Francisco Charte, Antonio J Rivera, Marı́a J Jesus and Francisco Herrera  “Addressing imbalance in multilabel classification: Measures and random resampling algorithms”  In _Neurocomputing_ 163 Elsevier, 2015, pp. 3–16
  * [3] Jiankang Deng et al.  “RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [4] Ionut Cosmin Duta, Li Liu, Fan Zhu and Ling Shao  “Improved residual networks for image and video recognition”  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , 2021, pp. 9415–9422  IEEE
  * [5] Paul Ekman and Wallace V Friesen  “Facial action coding system”  In _Environmental Psychology & Nonverbal Behavior_, 1978
  * [6] Qibin Hou, Daquan Zhou and Jiashi Feng  “Coordinate Attention for Efficient Mobile Network Design”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 13713–13722
  * [7] D Kollias, A Schulc, E Hajiyev and S Zafeiriou  “Analysing Affective Behavior in the First ABAW 2020 Competition”  In _2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)(FG)_ , pp. 794–800
  * [8] Dimitrios Kollias  “ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges”  In _arXiv preprint arXiv:2202.10659_ , 2022
  * [9] Dimitrios Kollias, Viktoriia Sharmanska and Stefanos Zafeiriou  “Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study”  In _arXiv preprint arXiv:2105.03790_ , 2021
  * [10] Dimitrios Kollias, Viktoriia Sharmanska and Stefanos Zafeiriou  “Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network”  In _arXiv preprint arXiv:1910.11111_ , 2019
  * [11] Dimitrios Kollias and Stefanos Zafeiriou  “Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework”  In _arXiv preprint arXiv:2103.15792_ , 2021
  * [12] Dimitrios Kollias and Stefanos Zafeiriou  “Analysing affective behavior in the second abaw2 competition”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 3652–3660
  * [13] Dimitrios Kollias and Stefanos Zafeiriou  “Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace”  In _arXiv preprint arXiv:1910.04855_ , 2019
  * [14] Dimitrios Kollias et al.  “Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond”  In _International Journal of Computer Vision_ Springer, 2019, pp. 1–23
  * [15] Tsung-Yi Lin et al.  “Feature pyramid networks for object detection”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125
  * [16] Chuanhe Liu, Wenqiang Jiang, Minghao Wang and Tianhao Tang  “Group level audio-video emotion recognition using hybrid networks”  In _Proceedings of the 2020 International Conference on Multimodal Interaction_ , 2020, pp. 807–812
  * [17] Chuanhe Liu, Tianhao Tang, Kui Lv and Minghao Wang  “Multi-feature based emotion recognition for video clips”  In _Proceedings of the 20th ACM International Conference on Multimodal Interaction_ , 2018, pp. 630–634
  * [18] Mahyar Najibi, Pouya Samangouei, Rama Chellappa and Larry S Davis  “Ssh: Single stage headless face detector”  In _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 4875–4884
  * [19] Tong Wu et al.  “Distribution-balanced loss for multi-label classification in long-tailed datasets”  In _European Conference on Computer Vision_ , 2020, pp. 162–178  Springer
  * [20] Stefanos Zafeiriou et al.  “Aff-wild: Valence and arousal ‘in-the-wild’challenge”  In _Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on_ , 2017, pp. 1980–1987  IEEE


#  RomanSetu: Efficiently unlocking multilingual capabilities of Large
Language Models models via Romanization

Jaavid Aktar Husain1 Raj Dabre2 Aswanth Kumar3
Ratish Puduppully4 Anoop Kunchukuttan5

IIT Madras, India1,5 IIIT D&M Kancheepuram1 Flipkart, India3
National Institute of Information and Communications Technology, Kyoto, Japan2
Institute for Infocomm Research (I2R), A∗STAR, Singapore4 Microsoft, India5
AI4Bharat1,5
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This study addresses the challenge of extending Large Language Models (LLMs)
to non-English languages, specifically those using non-Latin scripts. We
propose an innovative approach that utilizes the romanized form of text as an
interface for LLMs, hypothesizing that its frequent informal use and shared
tokens with English enhance cross-lingual alignment. Focusing on Hindi, we
demonstrate through Hindi-to-English translation and sentiment analysis tasks
that romanized text not only significantly improves inference efficiency due
to its lower fertility compared to native text but also achieves competitive
performance with limited pre-training. Additionally, our novel multi-script
prompting approach, which combines romanized and native texts, shows promise
in further enhancing task performance. These findings suggest the potential of
romanization in bridging the language gap for LLM applications, with future
work aimed at expanding this approach to more languages and tasks.

11footnotetext: Work done during internship at AI4Bharat, IIT Madras.

##  1 Introduction

Large language models (LLMs) demonstrate remarkable proficiency across a broad
spectrum of natural language processing (NLP) tasks, as evidenced by various
studies (Liu et al., 2023; Chung et al., 2022; Chowdhery et al., 2022; Wei et
al., 2022; Goyal et al., 2022). They excel not only in tasks for which they
were explicitly trained but also in those they were not trained for. This
achievement is mainly due to the availability of corpora (Wenzek et al., 2019;
Abadji et al., 2021; Suárez et al., 2019) as well as the advancements in LLMs
that leverage these datasets for pretraining (Touvron et al., 2023; Workshop
et al., 2022; Chowdhery et al., 2022). Despite their proficiency in English,
these models typically demonstrate reduced effectiveness when applied to non-
English languages, highlighting a significant challenge in extending their
benefits to non-English languages.

The English-heavy LLMs (Touvron et al., 2023; Jiang et al., 2023; Zhang et
al., 2022) still have some representation from other languages due to data
leakage while creating the pre-training dataset, particularly for languages
which use the same script as English i.e. Latin script. This script sharing
enables cross-lingual transfer and bestows some of these LLM capabilities to
these languages. For languages using non-Latin scripts, the data
representation is very limited to non-existent. Tokenization of text in the
languages exhibits high fertility (Ács, 2020) and byte-level representation
(Artetxe et al., 2019). Hence, these LLMs perform poorly on most of these
languages, and the inefficient tokenization also leads to high inference
latency. This disparity in performance raises a critical question: how can we
extend the capabilities of LLMs to the languages written in non-Latin scripts?

A widely explored solution is extension of the tokenizer vocabulary to
incorporate new languages and continual pre-training on native language data
(Cui et al., 2023; Nguyen et al., 2023; Minixhofer et al., 2021). This
approach is computationally demanding, since the models need to be continually
pre-trained long enough to effectively learn the new embeddings and align the
representations of English and the new language. Furthermore, this approach
requires the availability of large volumes of text corpora.

In this work, we explore an alternative approach for more efficient knowledge
transfer. Instead of utilizing the native script, we use the romanized form of
text as the interface to the LLMs. The adoption of romanized representation is
justified for several reasons. In many languages, romanized text has been
frequently used in informal settings and on social media in recent history.
This usage creates the potential for the inclusion of some romanized data
representation in the pre-training corpus. Additionally, code-mixing with
English is a common occurrence, and the romanized form shares tokens with
English. This leads us to hypothesize that the romanized form is better
aligned with English than the native script, thereby facilitating a more
effective transfer from English.

We present our initial results using romanization to extend the capabilities
of LLMs to languages not represented in the Latin script. Taking the case of
Hindi, an Indo-Aryan language written in the Devanagari script and spoken by
over 500 million people, we conduct experiments in two tasks: Hindi-to-English
translation and sentiment analysis. Our experiments and analysis indicate
that:

  * •

The fertility of the romanized text is 2x times lower than the native text,
making the romanized form far more efficient than the native script.

  * •

Continually pre-training on romanized data is key to unlocking the performance
on non-Latin script languages. A model continually pre-training with limited
romanized data is competitive with the base model using native text. Hence,
inference efficiency improves significantly without any significant negative
effect on task performance.

  * •

Romanized representation can complement the native representation. We propose
a multi-script prompting approach that jointly prompts with romanized and
native text to improve task performance.

We plan to extend our experiments to more tasks and languages to provide
stronger validation of our hypothesis.

##  2 Related Work

Transliteration refers to the conversion of text written in one script to
another. Romanization is a specific instance of transliteration, where the
target script is the Roman/Latin script. Romanization has special significance
since Roman script is by far the most widely adopted script in the world and
many language models are primarily trained for English, which is written in
the Roman script. Transliteration is typically used to represented different
languages in the same script to enable cross-lingua transfer.

Transliteration has been used to improve cross-lingual transfer in NMT.
Amrhein and Sennrich (2020) show that transliteration shows improvements for
low-resource languages with different scripts by transferring from related
high resource languages that use different scripts. Goyal et al. (2020) show
that transliteration helps even when only contact relationship exists between
the languages involved. Song et al. (2020) show that transliteration during
pre-training stage for NMT also helps cross-lingual transfer.

Transliteration has also been used for cross-lingual transfer in the context
of pre-trained language models. Some works (Khemchandani et al., 2021;
Dhamecha et al., 2021; Moosa et al., 2022; Purkayastha et al., 2023) employ
transliteration to a common script during the pretraining phase to enable
cross-lingual transfer. Other studies (Dabre et al., 2022; Muller et al.,
2021; Chau and Smith, 2021) adopt transliteration during the fine-tuning.
Transliteration could be done to a common non-Latin script (Khemchandani et
al., 2021; Dhamecha et al., 2021; Dabre et al., 2022; Doddapaneni et al.,
2023) or to the Latin script Muller et al. (2021); Moosa et al. (2022);
Purkayastha et al. (2023).

While transliteration has been explored for language modeling as described
above, our work differs from previous work in some impoertant aspects:

  * •

Previous work explored cross-lingual transfer using transliteration with
multilingual language models. We focus our attention on English LLMs, and try
to achieve cross-lingual transfer via romanization using English LLMs. This is
a challenging scenario, since the language can be unrelated to English, and
very little (if any) native or romanized data in the language under study
might be seen during pre-training of the English LLM. At the same time, this
is a very practical need since most best performing LLMs are English-heavy and
hence cross-lingual transfer via romanization is an important direction to
explore.

  * •

We investigate the utility of transliteration in decoder-only language models,
which is the standard architecture for LLMs. In contrast, all previous work
explored cross-lingual transfer with transliteration in the context of
encoder-only models (the exception is Dabre et al. (2022) which use encoder-
decoder models).

  * •

Additionally, we delve into multi-script prompting, using both native and
romanized inputs, along with continual pretraining and supervised fine-tuning,
in a language model pretrained on mostly-English data.

To the best of our knowledge, no prior research has investigated the
leveraging of romanization for mostly-English LLMs in the context of cross-
lingual transfer to non-English languages.

Figure 1: Overview of our proposed approach consisting of romanization,
continual-pretraining and fine-tuning. Figure 2: Overview of our romanization-
based multi-script prompting. The setup on the left shows the usual
inferencing using the native script, the middle setup utilizes the
transliteration tool and constructs prompts using transliterated text. On the
right is a multi-script setup that constructs prompts using both native and
transliterated scripts. This is a 1-shot with a newline as a delimiter between
in-context examples and input.

##  3 Utilizing Romanized Data to make LLMs Multilingual

Our proposed approach aims to enhance the capabilities of mostly-English LLMs
for non-English languages by leveraging romanized data. We first continually
pretrain the LLM with both romanized and English data to create a base LM that
is romanization-aware. Subsequently, we extend the process by fine-tuning the
continually pretrained model for the specific task. The framework of our
approach is illustrated in Figure 1, encompassing the stages of data
romanization, pre-training, and fine-tuning.

###  3.1 Romanization Scheme

A variety of romanization schemes are available, each driven by multiple
considerations. One key factor is the resemblance of the romanized
representation to the way people typically write romanized text. This is
particularly advantageous if the pre-training data includes romanized text, as
it might aid in aligning with English. Another important aspect is the
fertility achieved by the romanization scheme when considering the original
LLM’s tokenizer. A third consideration is whether the transliteration scheme
is lossy or lossless. A lossless scheme is preferable when the objective is to
convert the output back to the native script. Typically, deterministic
transliteration schemes are lossless, whereas natural transliteration schemes
tend to be lossy.

In this work, we evaluated two romanization schemes for Hindi: (a) the
extended ITRANS scheme Kunchukuttan (2020), which defines a fixed, reversible
mapping between Devanagari and Roman characters, and (b) the IndicXlit scheme
Madhani et al. (2023), which generates romanizations as commonly used by Hindi
speakers in informal contexts and is learned from parallel transliteration
corpora. These mappings are inherently lossy. The IndicXlit romanization
demonstrates lower fertility compared to ITRANS romanization (2.98 vs. 3.91).
Preliminary prompting experiments with the LLaMa2 model also indicated that
IndicXlit outperforms ITRANS in romanized Hindi to English translation.
Consequently, we selected IndicXlit as our romanization scheme for this
project. However, it is important to note that the transliterations are not
reversible. With continued pre-training, ITRANS transliterations might
eventually achieve similar task performance to IndicXlit while ensuring script
reversibility. We leave this exploration for future work.

###  3.2 Continual Pretraining

To render the base model aware of romanization, we continue to pretrain the
base LLM with romanized document-level data. To prevent any catastrophic
forgetting of English capabilities, we also incorporate an equal amount of
English data into the pre-training mix. Additionally, we explore the inclusion
of native script data in the pre-training mixture as part of our multi-script
prompting approach. It is important to note that native script pre-training is
not necessary for romanized prompting. We maintain the use of the original
tokenizer embedding and do not expand the vocabulary for either the romanized
or native script text.

###  3.3 Supervised Fine-tuning

The LLM is subsequently fine-tuned using task-specific data pairs (input,
output) to align the model for performing the specified task. Following
standard supervised fine-tuning practices, the cross-entropy loss is
calculated solely on the output tokens. In this work, we fine-tune the model
with romanized SFT data. Moreover, we also undertake fine-tuning with native
script SFT data and multi-script SFT data. In the current work, we fine-tune
the model separately for different tasks. The exploration of multi-task fine-
tuning is reserved for future work.

###  3.4 Prompting

In this study, we investigate both zero-shot and few-shot prompting
techniques. For the romanization-aware models, we employ prompts in romanized
text. In the case of other model variations, native script prompting is also
explored for comparative purposes. Additionally, we introduce the concept of
multi-script prompting. In this method, the model receives prompts in both the
native script and its romanized form. The rationale behind this approach is
that the native script may have been more prevalent in the original pre-
training data, while the romanized script potentially offers better alignment
with English. Our aim is that these two representations will mutually enhance
the model’s performance. Figure 2 depicts the various prompting methods under
exploration.

##  4 Experimental Settings

We conducted experiments using LLaMA2 7B (Touvron et al., 2023) as the base
model under various settings. To assess the effectiveness of romanization, we
conducted experiments for two tasks: Hindi-to-English machine translation and
Hindi sentiment analysis.

###  4.1 Datasets

Continual Pretraining: For continual pretraining, we sourced approximately 100
million words of document-level data from web-crawled Hindi corpora
(Doddapaneni et al., 2023). To generate the romanized dataset, we
transliterated this Hindi dataset using the IndicXlit model (Madhani et al.,
2023). Both the original Hindi dataset and its romanized counterpart were then
used for continual pretraining.

Supervised Fine-tuning: In the supervised fine-tuning phase for machine
translation, Hindi-English pairs were obtained from the BPCC-H-Wiki and BPCC-
H-Daily seed data within the BPCC corpus (Gala et al., 2023), comprising
roughly 40,000 parallel sentences. For sentiment analysis, we employed the
SST2 dataset (Socher et al., 2013), which includes about 67,300 instances of
sentiment analysis data in the training split. It is important to note that
for sentiment analysis, we fine-tuned the model using English data but
evaluated its performance on Hindi.

Evaluation Data: The FLORES-200 test set (Costa-jussà et al., 2022) was used
for evaluating machine translation, and the IndicSentiment (Doddapaneni et
al., 2023) for sentiment analysis. For machine translation, we employed the
dev set for model checkpoint selection and the devtest set for evaluation. The
first three examples from the dev set were used as few-shot exemplars. A
similar approach was applied to the dev and test sets of IndicSentiment.

###  4.2 Training and Finetuning Details

We adapt the _open-instruct_ 111https://github.com/allenai/open-
instruct/tree/main (Wang et al., 2023) for our continual pre-training and
fine-tuning experiments. The models are fully-finetuned during continual pre-
training as well supervised finetuning. Detailed information about the
hyperparameters used for both continual pretraining and supervised fine-tuning
can be found in Appendix A. The optimal hyperparameters were identified based
on their performance in the validation set.

###  4.3 Decoding Details

For machine translation generation, we use greedy decoding with the bfloat16
precision. Throughout all experiments, we maintained a consistent batch size
of 8 to ensure uniformity and reliability across various experimental setups.

###  4.4 Evaluation Metrics

The evaluation metric for machine translation in our experiments is BLEU
(Papineni et al., 2002) computed using the SacreBLEU toolkit (Post,
2018).222nrefs:1—case:mixed—eff:no—tok:13a—smooth:exp—version:2.3.1 For the
sentiment analysis task, we report accuracy scores.

###  4.5 Models compared

We use the 7B parameter LLAaMA2 base model as our starting point for all our
experiments. We evaluate the base model and 2 continually pre-trained models:
(a) using romanized + English data, (b) romanized + native script + English
data. We also explored 4 variants of SFT models, depending on the script
configuration of the Hindi SFT data: (a) romanized examples, (b) native script
examples, (c) multi-script examples, (d) aggregating examples of all the three
types (a-c).

|  |  |  | Prompting Data
---|---|---|---|---
# |  CPT Data | SFT Data | #shots | H | RH | H+RH
1 |  ✗ | ✗ | 1 | 16.7 | 6.4 | 7.2
2 | 3 | 16.2 | 6.5 | 17.0
3 | H$\rightarrow$E | 0 | 25.2 | - | -
4 | RH$\rightarrow$E | 0 | - | 26.0 | -
5 | H+RH$\rightarrow$E | 0 | - | - | 28.4
6 | All | 0 | 22.6 | 25.4 | 27.4
7 |  RH+ E | ✗ | 1 | - | 15.7 | -
8 | 3 | - | 15.4 | -
9 | H$\rightarrow$E | 0 | 25.3 | - | -
10 | RH$\rightarrow$E | 0 | - | 28.6 | -
11 | H+RH$\rightarrow$E | 0 | - | - | 28.8
12 | All | 0 | 23.7 | 26.6 | 28.0
13 |  H+ RH+ E | ✗ | 1 | 24.1 | 16.7 | 23.0
14 | 3 | 22.3 | 17.0 | 23.5
15 | H$\rightarrow$E | 0 | 26.5 | - | -
16 | RH$\rightarrow$E | 0 | - | 28.1 | -
17 | H+RH$\rightarrow$E | 0 | - | - | 29.8
18 | All | 0 | 24.9 | 26.5 | 27.3
Table 1: BLEU scores for the machine translation of Hindi (H) to English (E)
using different types of continual pre-training (CPT) data, supervised fine-
tuning (SFT) data, number of shots (#shots), and the type of script used,
namely, native Hindi (N), Romanized Hindi (RH) and multi-script (H+RH)
methods. The highest scores, for each type of CPT data used, are in bold text.
We prompt non-SFT models with 1 and 3 shots, whereas we prompt SFT models in 0
shot manner.

###  4.6 Prompting Methods compared

We experimented with 0-shot, 1-shot and 3-shot prompting. We also experimented
with native script, romanized and multi-script prompting. Below is a prompt
template for the $k$-shot setting, incorporating both native and
transliterated text for machine translation task:

     Translate the following sentences from
     Hindi to English. The output should be
     in English and no other language.
        Hindi: [x_1]
        Hindi Transliterated: [t_1]
        English: [y_1]

        ...

        Hindi: [x_k]
        Hindi Transliterated: [t_k]
        English: [y_k]

        Hindi: [x]
        Hindi Transliterated: [t]
        English:

The prompt templates for various prompting methods and the sentiment analysis
task are provided in the Appendix B.

##  5 Results

Table 1 shows the results for the machine translation experiments from Hindi
to English, while Table 3 presents the results for the sentiment analysis task
in Hindi.

#### Few-Shot Prompting of Base Model

In the few-shot prompting of the base LLaMA2 model, prompting with native
scripts outperforms prompting with romanized data (see Rows 1 and 2). A
combination of native script and romanized data in a multi-script approach
enhances the effectiveness over using native data alone for 3-shot prompting.
The comparatively lower performance when using solely romanized Hindi is
understandable, given that the base LLaMA2 model is less familiar with the
vocabulary in the romanized script compared to that in the native script.

#### Continual Pre-training on Romanized Data Enhances Task Performance

We see that continual pre-training on limited amount of romanized data
significantly enhances translation quality in few-shot prompting scenarios.
The Continual Pre-Training (CPT) model using romanized data prompting achieves
performance comparable to that of the base model prompted with native script
text, as demonstrated by the comparison between Rows 7 and 8 with Rows 1 and
2.

#### Efficiency Gains with Romanized Prompting

Table 2 presents the fertility scores for romanized and native script text in
the LLaMA2 model. It is observed that the sequence length for native script
text is nearly twice that of the romanized text, indicating that processing
romanized text is significantly more efficient than native text. Coupled with
the previous results, this suggests that Continual Pre-Training (CPT) with
romanized data not only achieves translation quality comparable to native
script but also offers substantial efficiency gains at inference time.

Language | Fertility Score
---|---
Hindi | 7.36
Romanized Hindi (IndicXlit) | 2.98
Table 2: Fertility scores of LLaMA2 model computed on FLORES-200 dev set dataset. |  |  |  | Prompting Data
---|---|---|---|---
# |  CPT Data | SFT Data | #shots | H | RH | H+RH
1 |  ✗ | ✗ | 1 | 0.51 | 0.51 | 0.51
2 | 3 | 0.75 | 0.52 | 0.69
3 | E$\rightarrow$S | 0 | 0.91 | 0.89 | 0.88
4 |  RH+E | ✗ | 1 | 0.51 | 0.51 | 0.52
5 | 3 | 0.62 | 0.72 | 0.77
6 | E$\rightarrow$S | 0 | 0.89 | 0.89 | 0.93
7 |  H+RH+E | ✗ | 1 | 0.51 | 0.51 | 0.51
8 | 3 | 0.75 | 0.56 | 0.84
9 | E$\rightarrow$S | 0 | 0.94 | 0.95 | 0.91
Table 3: Accuracies for sentiment analysis of Hindi (H) data to get sentiments
(S) using different types of continual pre-training (CPT) data, number of
shots (#shots), and the type of script used, namely, native Hindi (N),
Romanized Hindi (RH) and multi-script (H+RH) methods. Supervised fine-tuning
(SFT) is done on English(E) sentiment data. The highest scores, for each type
of CPT data used, are in bold text. We prompt non-SFT models with 1 and 3
shots, whereas we prompt SFT models in 0 shot manner.

#### Supervised Fine-tuning on Romanized Data

Supervised fine-tuning of the base model using romanized text has been found
to significantly enhance translation quality, achieving equivalence to native
text translation (see Rows 3 and 4). Furthermore, supervised fine-tuning (SFT)
on the Continual Pre-Training (CPT) model with romanized data results in
further improvements in translation performance (refer to Rows 9 and 10). In
fact, romanized translation outperforms native script translation, suggesting
that romanized model can better benefit from the English LLM due to better
cross-lingual overlap. Consequently, romanized SFT presents itself as an
efficient and effective alternative to native script SFT.

#### Multiscript Enhances Performance of MostlyEnglish LLMs

A substantial improvement in translation quality has been observed through the
use of multiscript prompting. This finding indicates that including
transliterated text along with native script significantly boosts the model’s
ability to produce superior machine translations. Notably, this method
surpasses native script prompting by as much as 3.3 BLEU points (see Rows 5,
11 and 17 versus preceding rows in their corresponding block), emphasizing the
beneficial role of transliterated text in enhancing machine translation
performance.

#### SFT on multiple data formats

If we finetune the model on multiple prompting formats, we see an decline in
performance compared to a model finetuned on just a single format (comparison
between Rows 5, 11 and 17 with Rows 6, 12, and 18.)

#### Sentiment Analysis Performance

In sentiment analysis, we observed that supervised fine-tuning (SFT) with an
English dataset enhances performance across various settings. This includes
the base LLaMA2 model, the model continually pretrained with both romanized
and English scripts, and the model pretrained with a combination of romanized,
native, and English scripts (Rows 3, 6 and 9). The romanized and native script
performances are comparable, suggesting that cross-lingual transfer work well
for romanized data inputs as well. This is in line with results for machine
translation presented earlier.

##  6 Conclusion and Future Work

In this study, we proposed the use of romanization to enhance the performance
of LLMs primarily trained in English. Our approach successfully unlocks LLM
capabilities for non-English languages while maintaining task performance. We
have empirically demonstrated the effectiveness of this strategy through
experiments involving few-shot prompting, continual pretraining, and
supervised fine-tuning in tasks such as Hindi-to-English machine translation
and Hindi sentiment analysis. Additionally, our results indicate that
leveraging romanized data significantly improves inference efficiency.

Looking forward, we aim to expand our experiments to encompass more languages
and explore a broader range of NLP tasks, particularly generation tasks. Our
primary focus will be on training models using larger text corpora that span
multiple languages. This will involve a special emphasis on cross-lingual
transfer and cross-task transfer, broadening the scope and impact of our
research.

## Limitations

Our proposed approach involves utilizing romanization to unlock the
capabilities of mostly-English LLMs for non-English languages. While we have
demonstrated the effectiveness of romanization using the LLaMA2 model in
conjunction with continual pretraining and fine-tuning, the generalizability
of our findings to other multilingual language models remains uncertain.
Additionally, the pretraining data utilized in our experiments constitutes
only a small portion, and the model might benefit further from exposure to a
larger dataset. Resource constraints led us to limit our experiments to a 7B
LLaMA model, but obtaining a more extensive understanding could be possible
with a larger model. Evaluation on a wider set of tasks will help to better
understand the generalization of this approach.

## Ethics Statement

We use romanization as a way to align English with other languages using non-
Latin scripts. Given the current state of LLMs, this seems like a practical
direction to extend capabilities of the best LLMs to other languages. The
intention is not to supplant the use of a native script (which is widely
adopted and has a rich literary tradition) with romanized script. Further
advancments are needed to extend this line of research to improve native
script performance efficiently.

This work does not involve any new data collection and does not employ any
annotators for data collection. We utilize publicly available datasets for the
experiments reported in this work. Some of these datasets originate from web
crawls, and we do not explicitly attempt to identify any biases within these
datasets, using them in their original form.

## References

  * Abadji et al. (2021) Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2021.  Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus.  In _CMLC 2021-9th Workshop on Challenges in the Management of Large Corpora_.
  * Amrhein and Sennrich (2020) Chantal Amrhein and Rico Sennrich. 2020.  On Romanization for model transfer between scripts in neural machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2461–2469, Online. Association for Computational Linguistics.
  * Artetxe et al. (2019) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019.  On the cross-lingual transferability of monolingual representations.  _arXiv preprint arXiv:1910.11856_.
  * Chau and Smith (2021) Ethan C. Chau and Noah A. Smith. 2021.  Specializing multilingual language models: An empirical study.  In _Proceedings of the 1st Workshop on Multilingual Representation Learning_ , pages 51–61, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Costa-jussà et al. (2022) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.  No language left behind: Scaling human-centered machine translation.  _arXiv preprint arXiv:2207.04672_.
  * Cui et al. (2023) Yiming Cui, Ziqing Yang, and Xin Yao. 2023.  Efficient and effective text encoding for chinese llama and alpaca.  _arXiv preprint arXiv:2304.08177_.
  * Dabre et al. (2022) Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh Khapra, and Pratyush Kumar. 2022.  IndicBART: A pre-trained model for indic natural language generation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1849–1863, Dublin, Ireland. Association for Computational Linguistics.
  * Dhamecha et al. (2021) Tejas Dhamecha, Rudra Murthy, Samarth Bharadwaj, Karthik Sankaranarayanan, and Pushpak Bhattacharyya. 2021.  Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8584–8595, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Doddapaneni et al. (2023) Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023.  Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12402–12426.
  * Gala et al. (2023) Jay Gala, Pranjal A Chitale, A K Raghavan, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar M, Janki Atul Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023.  Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages.  _Transactions on Machine Learning Research_.
  * Goyal et al. (2022) Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022.  News summarization and evaluation in the era of gpt-3.  _arXiv preprint arXiv:2209.12356_.
  * Goyal et al. (2020) Vikrant Goyal, Anoop Kunchukuttan, Rahul Kejriwal, Siddharth Jain, and Amit Bhagwat. 2020.  Contact relatedness can help improve multilingual NMT: Microsoft STCI-MT @ WMT20.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 202–206, Online. Association for Computational Linguistics.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Khemchandani et al. (2021) Yash Khemchandani, Sarvesh Mehtani, Vaidehi Patil, Abhijeet Awasthi, Partha Talukdar, and Sunita Sarawagi. 2021.  Exploiting language relatedness for low web-resource language model adaptation: An Indic languages study.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1312–1323, Online. Association for Computational Linguistics.
  * Kunchukuttan (2020) Anoop Kunchukuttan. 2020.  The IndicNLP Library.  https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf.
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Computing Surveys_ , 55(9):1–35.
  * Madhani et al. (2023) Yash Madhani, Sushane Parthan, Priyanka Bedekar, Gokul Nc, Ruchi Khapra, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Khapra. 2023.  Aksharantar: Open Indic-language transliteration datasets and models for the next billion users.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 40–57, Singapore. Association for Computational Linguistics.
  * Minixhofer et al. (2021) Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2021.  Wechsel: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.  _arXiv preprint arXiv:2112.06598_.
  * Moosa et al. (2022) Ibraheem Muhammad Moosa, Mahmud Elahi Akhter, and Ashfia Binte Habib. 2022.  Does transliteration help multilingual language modeling?  _arXiv preprint arXiv:2201.12501_.
  * Muller et al. (2021) Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021.  When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 448–462, Online. Association for Computational Linguistics.
  * Nguyen et al. (2023) Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, et al. 2023.  Seallms–large language models for southeast asia.  _arXiv preprint arXiv:2312.00738_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting bleu scores.  _arXiv preprint arXiv:1804.08771_.
  * Purkayastha et al. (2023) Sukannya Purkayastha, Sebastian Ruder, Jonas Pfeiffer, Iryna Gurevych, and Ivan Vulić. 2023.  Romanization-based large-scale adaptation of multilingual language models.  _arXiv preprint arXiv:2304.08865_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Song et al. (2020) Haiyue Song, Raj Dabre, Zhuoyuan Mao, Fei Cheng, Sadao Kurohashi, and Eiichiro Sumita. 2020.  Pre-training via leveraging assisting languages for neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 279–285, Online. Association for Computational Linguistics.
  * Suárez et al. (2019) Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019.  Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures.  In _7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7)_. Leibniz-Institut für Deutsche Sprache.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2023) Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023.  How far can camels go? exploring the state of instruction tuning on open resources.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_.
  * Wenzek et al. (2019) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2019.  Ccnet: Extracting high quality monolingual datasets from web crawl data.  _arXiv preprint arXiv:1911.00359_.
  * Workshop et al. (2022) BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.
  * Ács (2020) Judit Ács. 2020.  Exploring BERT’s vocabulary.  http://juditacs.github.io/2019/02/19/bert-tokenization-stats.html.

##  Appendix A Hyperparameter Details

Hyperparameter | Value
---|---
Batch Size (tokens) | 1M, 2M
Learning Rate | 0.00005
Number of Epochs | 1, 2, 3, 4
Maximum Sequence Length | 2,048
Table 4: The range of hyperparameters used for continual pretraining. Hyperparameter | Value
---|---
Batch Size (examples) | 128
Learning Rate | 0.00005
Number of Epochs | 1, 2, 3, 4
Maximum Sequence Length | 2,048333Although this is the maximum permissible length, most examples fall far below this length during fine-tuning.
Table 5: The range of hyperparameters used for supervised fine-tuning.

##  Appendix B Prompts

Example of one-shot prompting used in the machine translation task using
native script.

Example of one-shot prompting used in the machine translation task using
transliterated script.

Example of one-shot prompting used in the machine translation task using multi
script.

Example of one-shot prompting used in the sentiment analysis task using native
script.

Example of one-shot prompting used in the sentiment analysis task using
transliterated script.

Example of one-shot prompting used in the sentiment analysis task using multi
script.