# AngOFA: Leveraging OFA Embedding
Initialization and Synthetic Data for
Angolan Language Model

Osvaldo Luamba Quinjica
Department of Computer Science
Beijing Institute of Technology
Beijing, China
<EMAIL_ADDRESS>
&David Ifeoluwa Adelani
Department of Computer Science
University College London
United Kingdom
<EMAIL_ADDRESS>
###### Abstract

In recent years, the development of pre-trained language models (PLMs) has
gained momentum, showcasing their capacity to transcend linguistic barriers
and facilitate knowledge transfer across diverse languages. However, this
progress has predominantly bypassed the inclusion of very-low resource
languages, creating a notable void in the multilingual landscape. This paper
addresses this gap by introducing four tailored PLMs specifically finetuned
for Angolan languages, employing a Multilingual Adaptive Fine-tuning (MAFT)
approach. In this paper, we survey the role of informed embedding
initialization and synthetic data in enhancing the performance of MAFT models
in downstream tasks. We improve baseline over SOTA AfroXLMR-base (developed
through MAFT) and OFA (an effective embedding initialization) by 12.3 and 3.8
points respectively.

##  1 Introduction

Significant advancements have marked the progress of language models and
evaluation datasets across various global languages (Devlin et al., 2019;
Conneau et al., 2020; Workshop et al., 2023; Xue et al., 2021). Nevertheless,
this progress has often bypassed numerous African languages, creating a
significant gap. Simultaneously, the majority of African-centric language
models have overlooked the inclusion of Angolan languages (Dossou et al.,
2022; Alabi et al., 2022; Ogueji et al., 2021). Efforts within the AfricaNLP
community have been commendable in broadening downstream evaluation datasets
(Adelani et al., 2021; 2022; Muhammad et al., 2023; Ma et al., 2023). However,
despite these initiatives, Angolan languages still lack representation.

In the pursuit of developing a multilingual pre-trained language model (PLM),
there are two primary approaches. The first entails building a model from
scratch, training it directly on multiple languages, employing a specific
self-supervised learning such as masked language modeling (Devlin et al.,
2019). An alternative approach is multilingual adaptive fine-tuning (MAFT)
which involves adapting an existing multilingual pretrained language model to
a new set of languages (Alabi et al., 2022; Wang et al., 2022; ImaniGooghari
et al., 2023). MAFT gains favor for its resource efficiency, especially in
scenarios where computational budgets pose constraints amid the escalating
model sizes (Tay et al., 2022; Gupta et al., 2023). The performance of MAFT
can be further enhanced by introducing new vocabulary tokens for the
additional languages and employing non-Gaussian embedding initialization
(Minixhofer et al., 2022; Dobler & de Melo, 2023; Liu et al., 2023a).

In this paper, we introduce the first set of multilingual PLMs tailored for
five Angolan languages using the MAFT approach. We compare PLMs developed
through MAFT with and without informed embedding initialization, denoted as
AngOFA and AngXLM-R , respectively. Leveraging OFA approach to perform
embedding initialization before performing MAFT, our findings reveal that
AngOFA significantly outperforms AngXLM-R and OFA, underscoring the
substantial performance improvements achievable through the incorporation of
informed embedding initialization and synthetic data.

##  2 Angolan Languages

Boasting a rich linguistic landscape comprising more than 40 languages and a
population of 32 million people, Angolan languages include Portuguese, some
Khoisan languages, and mostly Bantu languages from the Niger-Congo family.
Despite this linguistic diversity, there is a notable scarcity of literature,
radio, or television programming in native Angolan languages. All languages in
Angola are written using the Latin script, and many share common digraphs. Due
to data scarcity, our focus will primarily revolve around the five most spoken
Angolan languages: Umbundu, Kimbundu, Kikongo, Chokwe, and Luba-Kasai. See
Table 1 for more details.

| Bantu | No. | NLLB | Synthetic | Combined | Combined
---|---|---|---|---|---|---
Language | Zone | Speakers | Corpus (MB) | Corpus (MB) | Corpus (MB) | No. Sentences
Chokwe (cjk) | Zone K | 0.5M | 11.3 | 108.2 | 119.5 | 878,824
Kimbundu (kmb) | Zone H | 1.7M | 10 | 98.5 | 108.5 | 800,603
Kikongo (kon) | Zone H | 2M | 112.1 | 107.9 | 220 | 2,189,413
Luba-Kasai (lua) | Zone L | 0.06M | 133.2 | 113.9 | 247.1 | 2,415,794
Umbundu (umb) | Zone R | 6M | 15.1 | 98\. 5 | 113.6 | 902,961
Total |  | 10.2M | 281.6 | 527 | 808.6 | 7,187,595

Table 1: Language Information and Statistics: Summary of language, language
family, number of speakers, number of sentences. All languages belong to the
Niger-Congo/Bantu family, we state the Bantu Zones according to (Smith, 1949).
Synthetic corpus was generated using NLLB-600M machine translation model

##  3 Approaches to improve MAFT

###  3.1 Vocabulary Expansion

PLMs are proned to Out-of-Vocabulary (OOV) tokens for languages or scripts
uncovered during pre-training. The situation is more pronounced for unseen
scripts (Adelani et al., 2021; Pfeiffer et al., 2021), one of the most
effective way of dealing with this is to expand the vocabulary of the PLM to
cover new tokens (Wang et al., 2019). Glot-500 (ImaniGooghari et al., 2023)
was created by first expanding the vocabulary of XLM-R from 250K to 400K
before MAFT. However, the new tokens added were randomly initialized.

###  3.2 OFA: Embedding Factorization

OFA addresses two problems of adapting PLMs to new languages (1) the random
initialization of embeddings for new subwords fails to exploit the lexical
knowledge encoded in the source model (2) the introduction of additional
parameters poses potential obstacles to the efficient training of the
finetuned model (Liu et al., 2023a). OFA solves these problems by leveraging
both external multilingual embeddings and embeddings in the source PLM to
initialize the embeddings of new subwords. In its approach, OFA factorizes the
embeddings matrix of the source PLM into two smaller matrices as replacements.
Within a lower-dimensional space, the embeddings of non-overlapping new
subwords are expressed as combinations of source PLM subword embeddings. These
combinations are weighted by similarities derived from well-aligned external
multilingual embeddings, i.e., ColexNet+ (Liu et al., 2023b), covering more
than one thousand languages. Overlapping subword embeddings are directly
copied. This approach ensures that embeddings for subwords shared between the
source PLM and the extended vocabulary are integrated, preserving continuity
in representation. To complete the process, OFA duplicates all non-embedding
parameters from the source PLM model, and the source tokenizer is substituted
with the target tokenizer post-vocabulary extension.

###  3.3 Synthetic data for language modeling

For languages lacking sufficient pre-training data, synthetic data can be
generated through dictionary augmentation (Reid et al., 2021) or machine
translation (MT) model—an approach very popular in MT research known as back-
translation is an effective way to improve MT model for low-resource languages
(Sugiyama & Yoshinaga, 2019; Xia et al., 2019). In this paper, we utilize
synthetic data obtained through machine translation as described in (Adelani
et al., 2023). The authors generated machine-translated data for 34 African
languages(including Angolan languages) with less than 10MB of data, using the
English news commentary dataset (Kocmi et al., 2022), which contains over 600K
sentences.

##  4 Data

###  4.1 Training data

We leveraged the NLLB dataset (NLLB-Team et al., 2022), excluding English
translations, and focused solely on Kimbundu, Umbundu, Kikongo, Chokwe, and
Luba-Kasai. These languages were concatenated into a single file as our pre-
training corpus. Additionally, we added synthetic data generated through NLLB.
Table 1 shows the details of the monolingual data.

###  4.2 Evaluation data

In our work, we evaluated on SIB-200 (Adelani et al., 2023), a text
classification dataset that provides train/dev/test sets with 7 classes in
more than 200 African languages and dialects. The distribution of the classes
are: science/technology (252), travel (198), politics (146), sports (122),
health (110), entertainment (93), geography (83). SIB-200 is the only
benchmark dataset that covers Angolan languages. We evaluated only on the
subset of Angolan languages covered in this work.

##  5 Experimental Setup

We utilized the cross-lingual capabilities of XLM-R (Conneau et al., 2020) for
training, resulting in the creation of a novel set of PLMs111Models available
at https://github.com/zuela-ai/ANGOFA: AngXLM-R and AngOFA. These models,
underwent distinct fine-tuning processes. Specifically, AngXLM-R underwent
fine-tuning using the MAFT approach outlined in Alabi et al. (2022), with two
variants—one trained solely on monolingual data (281.6 MB), and the other
incorporating both monolingual and synthetic data (808.7 MB).

Similarly, AngOFA also underwent two variations of fine-tuning, utilizing the
datasets in the same manner as AngXLM-R . However, AngOFA followed the
configurations outlined for ofa-multi-768, as described in (Liu et al.,
2023a). We opted to maintain 768 as the only latent dimension in our
experiments based on insights from (ImaniGooghari et al., 2023; Liu et al.,
2023a) and further supported by preliminary results from our own experiments.
These findings revealed evidence of information loss in lower dimensions,
particularly noticeable in tasks such as text classification. This dataset
partitioning approach aimed to investigate the effects of the MAFT and OFA
approaches, both with and without synthetic data, on model performance.

We compared our new models to the following baseline models:

  1. 1.

XLM-R (Conneau et al., 2020): an encoder-only model that underwent pre-
training on 100 languages through a masked language model objective. XLM-R
does not cover any language evaluated in this work.

  2. 2.

Serengeti (Adebara et al., 2023): trained on 500 African languages, including
10 high-resource ones. It includes Kimbundu, Umbundu, and Chokwe.

  3. 3.

Glot-500 (ImaniGooghari et al., 2023): derived from XLM-R, was extended to
cover 500 languages by expanding its vocabulary from 250K to 400K, thus
accommodating new tokens representing 400 languages previously absent in
XLM-R. Glot-500 covers all Angolan languages used in our evaluation.

  4. 4.

AfroXLMR-base (Alabi et al., 2022): developed using the MAFT approach, it
covers 20 languages with a monolingual corpus of at least 50MB. Angolan
languages are not included.

  5. 5.

AfroXLMR-base-76L (Adelani et al., 2023): developed using the MAFT approach,
it covers languages with at least 10MB of data on the web. It expands coverage
to include more languages, notably those listed in the NLLB-200 MT model.
Synthetic data was also generated for approximately 30 languages with limited
data, including all five Angolan languages. In total, it covers 76 languages.

  6. 6.

OFA (Liu et al., 2023a): integrates OFA embedding initialization alongside
MAFT using Glot500-c (ImaniGooghari et al., 2023), thus including all
languages addressed in this work.

##  6 Results and Discussion

| Pre-trained (scratch) | MAFT |  MAFT \+ syn. data | OFA | OFA + syn
---|---|---|---|---|---
|  |  | Glot | Afro | Ang | Afro | Ang | Ang | OFA |
Lang. | XLM-R | Serengeti | 500 | XLMR | XLM-R | XLMR76 | XLM-R | OFA | 500 | AngOFA
cjk | 41.3 | 43.2 | 42.9 | 51.3 | 43.6 | 55.6 | 51.7 | 46.3 | 52.8 | 58.4
kmb | 44.8 | 46.9 | 43.5 | 50.6 | 50.2 | 58.5 | 56.6 | 58.5 | 63.2 | 64.7
kon | 67.8 | 69.1 | 72.6 | 65.7 | 72.5 | 77.2 | 76.1 | 78.8 | 76.9 | 82.4
lua | 54.5 | 57.9 | 54.7 | 62.5 | 65.4 | 64.4 | 73.2 | 69.1 | 68.6 | 73.5
umb | 50.4 | 51.7 | 40.3 | 50.5 | 54.9 | 61.0 | 56.8 | 54.3 | 61.8 | 63.3
Ave. | 51.8 | 53.7 | 50.7 | 56.1 | 57.3 | 63.3 | 62.8 | 61.4 | 64.6 | 68.4

Table 2: Benchmark results: comparing the effectiveness of OFA to random
initialization before multilingual adaptive fine-tuning (MAFT)

Table2 shows the performance of our baseline models using the weighted F1
metric. We discuss our key findings below:

##### Region-specific PLMs are better than those pre-trained from scratch with
many languages

Our results shows that AngXLM-R created with MAFT performed better than XLM-R,
AfroXLMR, Serengeti and Glot-500 with $+5.5$, $+1.2$, $+3.6$, $+6.6$ points
respectively. The last two PLMs have been pre-trained on 500+ languages with
few Angolan languages but performed worse than AfroXLMR (adapted through MAFT
to 20 languages), and AngXLM-R (adapted to five Angolan languages). This shows
that region-specific PLMs covering related languages within the same language
family can be more effective.

##### MAFT results can be boosted by leveraging synthetic monolingual data

By incorporating additional synthetic data, AngXLM-R (+SYN data) performance
improved by $+5.5$ over the AngXLM-R without synthetic data. However, it
failed to beat the performance of AfroXLMR-base-76L that has been trained on
76 African languages including all Angolan languages except for Luba-Kasai
with the largest data. Our experiment showed that the adapted PLM to 76
languages performed better than Serengeti pre-trained on 500 languages, which
further shows that we can create better PLMs to cover more languages through
adaptation without the expensive process of pre-training from scratch.

##### OFA embedding initialization with larger data is more effective

Models initialized with OFA demonstrated a consistent improvement compared
with other baselines. This indicates that OFA, which explicitly leverages
information encoded in the embeddings of the source model and external
multilingual embeddings, is superior to random initialization. Notably,
AngOFA’s advantage over OFA is accentuated by its access to a significantly
larger corpus of data for the respective languages through the use of
synthetic data. Without the additional synthetic data AngOFA performed worse
than OFA pre-trained on 500 languages with a drop of $-3.2$. However, when we
trained on the synthetic data, AngOFA achieved the best overall performance
with $+16.6$ over XLM-R, $+12.3$ over AfroXLMR, and $+5.6$ over AngXLM-R (with
synthetic data).

##  7 Conclusion and Future work

This paper introduces four multilingual PLMs models tailored for Angolan
languages. Our experimental findings illustrate that employing informed
embedding initialization significantly enhances the performance of a MAFT
model in downstream tasks. While models initialized with OFA exhibit superior
results compared to their counterparts, even in the case where AngXLM-R
finetuned on a larger corpus of data for the respective languages performs
poorly as compared to OFA finetuned on a smaller corpus. Nevertheless, the
specific factors contributing to AngXLM-R ’s superiority over OFA, especially
in the context of Luba-Kassai, raise intriguing questions about the primary
determinants influencing the performance of models in downstream tasks,
including considerations like dataset size versus informed embedding
initialization. These questions are left for future investigation.
Furthermore, we aim to expand the application of OFA to more African languages
for further exploration.

#### Acknowledgments

This work was supported in part by Oracle Cloud credits and related resources
provided by Oracle. David Adelani acknowledges the support of DeepMind
Academic Fellowship programme.

## References

  * Adebara et al. (2023) Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Alcoba Inciarte.  SERENGETI: Massively multilingual language models for Africa.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Findings of the Association for Computational Linguistics: ACL 2023_ , pp. 1498–1537, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-acl.97.  URL https://aclanthology.org/2023.findings-acl.97.
  * Adelani et al. (2022) David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Kabore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh-Buabeng, Victoire Memdjokam Koagne, Allahsera Auguste Tapo, Tebogo Macucwa, Vukosi Marivate, Mboning Tchiaze Elvis, Tajuddeen Gwadabe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabende, Neo Lerato Mokono, Ignatius Ezeani, Chiamaka Chukwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow.  MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 4488–4508, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.298.  URL https://aclanthology.org/2022.emnlp-main.298.
  * Adelani et al. (2021) David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei.  MasakhaNER: Named entity recognition for African languages.  _Transactions of the Association for Computational Linguistics_ , 9:1116–1131, 2021.  doi: 10.1162/tacl˙a˙00416.  URL https://aclanthology.org/2021.tacl-1.66.
  * Adelani et al. (2023) David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and Annie En-Shiun Lee.  Sib-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects, 2023.  URL https://arxiv.org/abs/2309.07445.
  * Alabi et al. (2022) Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow.  Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning.  In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), _Proceedings of the 29th International Conference on Computational Linguistics_ , pp. 4336–4349, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.  URL https://aclanthology.org/2022.coling-1.382.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 8440–8451, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.747.  URL https://aclanthology.org/2020.acl-main.747.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Dobler & de Melo (2023) Konstantin Dobler and Gerard de Melo.  FOCUS: Effective embedding initialization for monolingual specialization of multilingual models.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 13440–13454, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.829.  URL https://aclanthology.org/2023.emnlp-main.829.
  * Dossou et al. (2022) Bonaventure F. P. Dossou, Atnafu Lambebo Tonja, Oreen Yousuf, Salomey Osei, Abigail Oppong, Iyanuoluwa Shode, Oluwabusayo Olufunke Awoyomi, and Chris Emezue.  AfroLM: A self-active learning-based multilingual pretrained language model for 23 African languages.  In Angela Fan, Iryna Gurevych, Yufang Hou, Zornitsa Kozareva, Sasha Luccioni, Nafise Sadat Moosavi, Sujith Ravi, Gyuwan Kim, Roy Schwartz, and Andreas Rücklé (eds.), _Proceedings of The Third Workshop on Simple and Efficient Natural Language Processing (SustaiNLP)_ , pp. 52–64, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.sustainlp-1.11.  URL https://aclanthology.org/2022.sustainlp-1.11.
  * Gupta et al. (2023) Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort.  Continual pre-training of large language models: How to re-warm your model?  In _Workshop on Efficient Systems for Foundation Models @ ICML2023_ , 2023.  URL https://openreview.net/forum?id=pg7PUJe0Tl.
  * ImaniGooghari et al. (2023) Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze.  Glot500: Scaling multilingual corpora and language models to 500 languages.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1082–1117, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.61.  URL https://aclanthology.org/2023.acl-long.61.
  * Kocmi et al. (2022) Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović.  Findings of the 2022 conference on machine translation (WMT22).  In Philipp Koehn, Loïc Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pp. 1–45, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics.  URL https://aclanthology.org/2022.wmt-1.1.
  * Liu et al. (2023a) Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schütze.  Ofa: A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining, 2023a.  URL https://arxiv.org/abs/2311.08849.
  * Liu et al. (2023b) Yihong Liu, Haotian Ye, Leonie Weissweiler, Renhao Pei, and Hinrich Schuetze.  Crosslingual transfer learning for low-resource languages based on multilingual colexification graphs.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023b.  URL https://openreview.net/forum?id=Tn5hALAaA4.
  * Ma et al. (2023) Chunlan Ma, Ayyoob ImaniGooghari, Haotian Ye, Ehsaneddin Asgari, and Hinrich Schütze.  Taxi1500: A multilingual dataset for text classification in 1500 languages, 2023.  URL https://arxiv.org/abs/2305.08487.
  * Minixhofer et al. (2022) Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz.  WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.  In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 3992–4006, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.naacl-main.293.  URL https://aclanthology.org/2022.naacl-main.293.
  * Muhammad et al. (2023) Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Seid Muhie Yimam, David Ifeoluwa Adelani, Ibrahim Said Ahmad, Nedjma Ousidhoum, Abinew Ali Ayele, Saif Mohammad, Meriem Beloucif, and Sebastian Ruder.  SemEval-2023 task 12: Sentiment analysis for African languages (AfriSenti-SemEval).  In Atul Kr. Ojha, A. Seza Doğruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, and Elisa Sartori (eds.), _Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)_ , pp. 2319–2337, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.semeval-1.315.  URL https://aclanthology.org/2023.semeval-1.315.
  * NLLB-Team et al. (2022) NLLB-Team, Marta Ruiz Costa-jussà, James Cross, Onur cCelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Alison Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon L. Spruit, C. Tran, Pierre Yves Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm’an, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang.  No language left behind: Scaling human-centered machine translation.  _ArXiv_ , abs/2207.04672, 2022.  URL https://api.semanticscholar.org/CorpusID:250425961.
  * Ogueji et al. (2021) Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin.  Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages.  In Duygu Ataman, Alexandra Birch, Alexis Conneau, Orhan Firat, Sebastian Ruder, and Gozde Gul Sahin (eds.), _Proceedings of the 1st Workshop on Multilingual Representation Learning_ , pp. 116–126, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.mrl-1.11.  URL https://aclanthology.org/2021.mrl-1.11.
  * Pfeiffer et al. (2021) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder.  UNKs everywhere: Adapting multilingual language models to new scripts.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 10186–10203, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.800.  URL https://aclanthology.org/2021.emnlp-main.800.
  * Reid et al. (2021) Machel Reid, Junjie Hu, Graham Neubig, and Yutaka Matsuo.  AfroMT: Pretraining strategies and reproducible benchmarks for translation of 8 African languages.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 1306–1320, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.99.  URL https://aclanthology.org/2021.emnlp-main.99.
  * Smith (1949) Edwin W. Smith.  The classification of the bantu languages. by malcolm guthrie, ph.d. published for the international african institute by the oxford university press, 1948. pp. 91. map. 8s. 6d. net.  _Africa_ , 19(1):73–74, 1949.  doi: 10.2307/1156267.
  * Sugiyama & Yoshinaga (2019) Amane Sugiyama and Naoki Yoshinaga.  Data augmentation using back-translation for context-aware neural machine translation.  In Andrei Popescu-Belis, Sharid Loáiciga, Christian Hardmeier, and Deyi Xiong (eds.), _Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)_ , pp. 35–44, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-6504.  URL https://aclanthology.org/D19-6504.
  * Tay et al. (2022) Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler.  Scale efficiently: Insights from pretraining and finetuning transformers.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=f2OYVDyfIB.
  * Wang et al. (2019) Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, and Dong Yu.  Improving pre-trained multilingual model with vocabulary expansion.  In Mohit Bansal and Aline Villavicencio (eds.), _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pp. 316–327, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/K19-1030.  URL https://aclanthology.org/K19-1030.
  * Wang et al. (2022) Xinyi Wang, Sebastian Ruder, and Graham Neubig.  Expanding pretrained models to thousands more languages via lexicon-based adaptation.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 863–877, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.61.  URL https://aclanthology.org/2022.acl-long.61.
  * Workshop et al. (2023) BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Daniel McDuff, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf.  Bloom: A 176b-parameter open-access multilingual language model, 2023\.  URL https://arxiv.org/abs/2211.05100.
  * Xia et al. (2019) Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, and Graham Neubig.  Generalized data augmentation for low-resource translation.  In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 5786–5796, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1579.  URL https://aclanthology.org/P19-1579.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mT5: A massively multilingual pre-trained text-to-text transformer.  In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 483–498, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.41.  URL https://aclanthology.org/2021.naacl-main.41.


11institutetext: Center for Nuclear Theory, Department of Physics and
Astronomy, Stony Brook University, Stony Brook, New York 11794â3800, USA

# Exact Polarization of Particles of Any Spin at Global Equilibrium

Andrea Palermo 11<EMAIL_ADDRESS>
###### Abstract

The polarization of the $\Lambda$ particle offers the unique opportunity to
study the hydrodynamic gradients in the Quark-Gluon Plasma formed in heavy-ion
collisions. However, the theoretical formula commonly used to calculate
polarization is only a linear order expansion in thermal vorticity and
neglects higher-order corrections. Here, I present an exact calculation to all
orders in (constant) thermal vorticity at global equilibrium, obtaining the
analytic form of the spin density matrix and the polarization vector for
massive particles of any spin. Finally, I extend these results to local
equilibrium and assess their phenomenological impact by numerically
calculating the polarization vector in a 3+1 hydrodynamic simulation.

##  1 Introduction

The experimental measurement of the polarization vector of $\Lambda$ particles
in heavy ion collisions paved a new way in the hydrodynamic studies of the QGP
STAR:2017ckg ; STAR:2019erd ; STAR:2020xbm ; ALICE:2021pzu ; ALICE:2019onw ;
Florkowski:2017ruc ; Florkowski:2018fap ; Weickgenannt:2019dks ;
Weickgenannt:2020aaf ; Gao:2020lxh ; Becattini:2020ngo ; Becattini:2022zvf ;
Gao:2020vbh ; Hongo:2021ona . In fact, with spin polarization one can access
the thermodynamic gradients in the plasma at freeze-out. According to the
hydrodynamic model and based on a statistical calculation, it was found early
on that the polarization vector of massive spin-S fields is Becattini:2013fla
; Fang:2016vpj ; Becattini:2019ntv :

$S^{\mu}(p)=-\frac{S(S+1)}{3}\frac{1}{2m}\epsilon^{\mu\nu\rho\sigma}p_{\sigma}\frac{\int{\rm
d}\Sigma\cdot p\,\varpi_{\nu\rho}n(p)[1+(-1)^{2S}n(p)]}{\int{\rm d}\Sigma\cdot
p\,n(p)},$ (1)

where $p$ is the four-momentum of the particle at hand, the integral is
calculated on the decoupling hypersurface, $n(p)=[{\rm e}^{\beta\cdot
p}+(-1)^{2S+1}]^{-1}$ is the Fermi-Dirac or the Bose-Einstein distribution
function and $\varpi$ is the thermal vorticity:

$\varpi_{\mu\nu}=-\frac{1}{2}(\partial_{\mu}\beta_{\nu}-\partial_{\nu}\beta_{\mu}),$
(2)

$\beta^{\mu}$ being the four-temperature vector.

Additional contributions to spin polarization have been found later on,
including the thermal shear Becattini:2021suc ; Becattini:2021iol ; Fu:2021pok
; Liu:2021uhn  and the spin hall effect Fu:2022myl ; Ivanov:2020udj ;
Buzzegoli:2022qrr , which I will not discuss further here. It is important to
stress that all the terms mentioned so far are the leading order contributions
to the spin vector in linear response theory. Even in the case of global
equilibrium, where the only contribution to the spin vector comes from the
constant thermal vorticity, there are no results beyond the first order of
approximation in $\varpi$. Here, I show how to compute the spin vector to all
orders in $\varpi$ for fields of any spin at global equilibrium
Palermo:2023cup . I will also provide a finite and analytic expression of the
spin density matrix, which can be used to compute any spin observable,
including the vector meson spin alignment.

##  2 Spin density matrix and polarization vector

To compute the polarization vector or any other spin-related quantity, one
uses the spin density matrix $\Theta$. For massive fields of spin $S$, the
spin density matrix is a $(2S+1)\times(2S+1)$ hermitian matrix, characterized
by $S^{2}-1$ real numbers. The definition of the spin density matrix adopted
in the context of relativistic heavy-ion collisions is
becattini2020polarization :

$\displaystyle\Theta_{sr}(p)=\frac{{\rm
Tr}\left({\widehat{\rho}}\,\widehat{a}^{\dagger}_{r}(p)\widehat{a}_{s}(p)\right)}{\sum_{l}{\rm
Tr}\left({\widehat{\rho}}\,\widehat{a}^{\dagger}_{l}(p)\widehat{a}_{l}(p)\right)}=\frac{\langle\widehat{a}^{\dagger}_{r}(p)\widehat{a}_{s}(p)\rangle}{\sum_{l}\langle\widehat{a}^{\dagger}_{l}(p)\widehat{a}_{l}(p)\rangle}.$
(3)

In the above equation, ${\widehat{\rho}}$ is the statistical operator, and
$\widehat{a}$ and $\widehat{a}^{\dagger}$ are the annihilation and creation
operators of states with momentum $p$ and spin $r$, $s$.

The spin density matrix is related to the spin vector $S^{\mu}$ through
becattini2020polarization :

$\displaystyle S^{\mu}(p)=\sum_{i=1}^{3}[p]^{\mu}_{\ i}{\rm
tr}\left(\Theta(p)D^{S}(\mathrm{J}^{i})\right),$ (4)

where $D^{S}(\mathrm{J}^{i})$ is the $i$-th generator of $\text{SO}(3)$ in the
$S$-representation. The transformation denoted as $[p]$ is the so-called
_standard boost_ : a Lorentz transformation that maps the rest frame momentum
$\mathfrak{p}=(m,\bm{0})$ to the momentum $p=(\varepsilon,\bm{p})$, i.e.
$p^{\mu}=[p]^{\mu}_{\ \nu}\mathfrak{p}^{\nu}$. Other formulae for the
polarization vector, such as those involving the Wigner function, are always
derived from Eqs. (3) and (4).

##  3 Exact spin physics at global equilibrium

From equation (4), the polarization vector can be computed once the spin
density matrix is known. Here, I will compute exactly the spin density matrix
for fields of any spin at global equilibrium.

I will consider the following density operator:

${\widehat{\rho}}=\frac{1}{Z}\exp\left[-b\cdot{\widehat{P}}+\frac{\varpi:{\widehat{J}}}{2}+\zeta{\widehat{Q}}\right],$
(5)

where ${\widehat{Q}}$ is the charge operator, and ${\widehat{P}}$ and
${\widehat{J}}$ are the generators of the PoincarÃ group: the four-momentum
and the angular momentum-boost operator, respectively. The vector $b^{\mu}$
and antisymmetric tensor $\varpi^{\mu\nu}$, that is the thermal vorticity, are
constants, and they define the four-temperature vector as the Killing vector:

$\displaystyle\beta^{\mu}(x)=b^{\mu}+\varpi^{\mu\nu}x_{\nu}.$

In addition, $\zeta=\mu/T$ is also constant, $\mu$ being the chemical
potential. The proper temperature is
$T=\left(\sqrt{\beta\cdot\beta}\right)^{-1}$.

In Refs. Becattini:2020qol ; Palermo:2021hlf , it was realized that
factorizing the operator (5) and performing the analytic continuation
$\varpi\mapsto-i\phi$ provides a powerful technique to compute exact
expectation values. This method allows the calculation of
$\langle\widehat{a}^{\dagger}_{r}(p)\widehat{a}_{s}(p)\rangle$:

$\langle\widehat{a}^{\dagger}_{s}(p)\widehat{a}_{t}(p^{\prime})\rangle=2\varepsilon^{\prime}\,\displaystyle\sum_{n=1}^{\infty}(-1)^{2S(n+1)}\delta^{3}(\Lambda^{n}{\bf{p}}-{\bf{p}^{\prime}})D(W(\Lambda^{n},p))_{ts}{\rm
e}^{-\widetilde{b}\cdot\sum_{k=1}^{n}\Lambda^{k}p}{\rm e}^{n\zeta},$ (6)

where $\Lambda=\exp\left[-i\phi:J/2\right]$ is the Lorentz transformation with
parameter $\phi$ equal to the _imaginary_ vorticity and $W(\Lambda,p)=[\Lambda
p]^{-1}\Lambda[p]$ is the so-called Wigner rotation. The vector
$\tilde{b}(\phi)$ is:

$\tilde{b}^{\mu}(\phi)=\sum_{k=0}^{\infty}\frac{1}{(k+1)!}\phi^{\mu}_{\
\nu_{1}}\phi^{\nu_{1}}_{\ \nu_{2}}\dots\phi^{\nu_{k-1}}_{\
\nu_{k}}b^{\nu_{k}},$ (7)

and its form is a consequence of the fact that the PoincarÃ group is non-
Abelian.

Plugging eq. (6) into eq. (3), yields:

$\Theta_{rs}(p)=\frac{\sum_{n=1}^{\infty}(-1)^{2S(n+1)}{\rm
e}^{-\widetilde{b}\cdot\sum_{k=1}^{n}\Lambda^{k}p}{\rm
e}^{n\zeta}D_{rs}(W(\Lambda^{n},p))\delta^{3}(\Lambda^{n}p-p)}{\sum_{n=1}^{\infty}(-1)^{2S(n+1)}{\rm
e}^{-\widetilde{b}\cdot\sum_{k=1}^{n}\Lambda^{k}p}{\rm e}^{n\zeta}{\rm
tr}[D(W(\Lambda^{n},p))]\delta^{3}(\Lambda^{n}p-p)}.$ (8)

To proceed further in the calculation, and as long as the thermal vorticity is
imaginary, the constraint $\Lambda p=p$ must be enforced. Indeed, from eq.
(6), one sees that if this is not the case
$\langle\widehat{a}^{\dagger}_{s}(p)\widehat{a}_{t}(p)\rangle=0$, and the spin
density matrix would become trivial.

Imposing the constraint $\Lambda p=p$, it follows that:

$\displaystyle\phi^{\mu\nu}=\epsilon^{\mu\nu\rho\sigma}\xi_{\rho}\frac{p_{\sigma}}{m},$
$\displaystyle\xi^{\rho}=-\frac{1}{2m}\epsilon^{\rho\mu\nu\sigma}\phi_{\mu\nu}p_{\sigma}.$
(9)

Thanks to this simplification, and after some algebra (see Ref.
Palermo:2023cup  for the full derivation), the spin density matrix becomes:

$\displaystyle\Theta(p)=\frac{\sum_{n=1}^{\infty}(-1)^{2S(n+1)}{\rm
e}^{-nb\cdot p}{\rm e}^{n\zeta}{\rm e}^{-in\bm{\xi}_{0}\cdot
D^{S}(\textbf{J})}}{\sum_{n=1}^{\infty}(-1)^{2S(n+1)}{\rm e}^{-nb\cdot p}{\rm
e}^{n\zeta}{\rm tr}\left({\rm e}^{-in\bm{\xi}_{0}\cdot
D^{S}(\textbf{J})}\right)},$

where $\xi_{0}^{\mu}={[p]^{-1}}^{\mu}_{\ \nu}\xi^{\nu}$, $\bm{\xi}_{0}$ being
its spatial part, and $D^{S}(\textbf{J})$ is the three-vector of the
generators of the rotation group in the $S$-representation. As long as the
absolute value of the eigenvalues of the matrix $\exp(-b\cdot
p+\zeta)\exp\left[-i\bm{\xi}_{0}\cdot D^{S}(\textbf{J})\right]$ is less than
one (that is $b\cdot p>\zeta$) the series can be summed as a geometric series
of matrices hubbard2002vector . Defining:

$\theta^{\mu}=-\frac{1}{2m}\epsilon^{\mu\nu\rho\sigma}\varpi_{\nu\rho}p_{\sigma},$
(10)

the spin density matrix after the analytic continuation $\phi\mapsto i\varpi$
reads:

$\Theta(p)=\frac{\left[(-1)^{2S}\mathbb{I}-{\rm e}^{-b\cdot
p+\bm{\theta_{0}}\cdot
D^{S}(\textbf{J})}\right]^{-1}-(-1)^{2S}\mathbb{I}}{\sum_{k=-S}^{S}\left({\rm
e}^{b\cdot p-k\sqrt{-\theta^{2}}}-(-1)^{2S}\right)^{-1}}.$ (11)

From eq. (11), one can compute all spin-related quantities, including spin
polarization and spin alignment. In particular, the mean spin vector reads,
using eq. (4):

$S^{\mu}(p)=\frac{\theta^{\mu}}{\sqrt{-\theta^{2}}}\frac{\sum_{k=-S}^{S}k\left[{\rm
e}^{b\cdot
p-\zeta-k\sqrt{-\theta^{2}}}-(-1)^{2S}\right]^{-1}}{\sum_{k=-S}^{S}\left[{\rm
e}^{b\cdot p-\zeta-k\sqrt{-\theta^{2}}}-(-1)^{2S}\right]^{-1}}.$ (12)

This formula reproduces the previous literature, including the small vorticity
approximation and the case of Boltzmann statistics. In the case of very large
thermal vorticity, it predicts a polarization equal to one Palermo:2023cup .

##  4 Polarization in heavy-ion collisions

One can now evaluate the effect of the formula (12) in phenomenological
applications. Although the derivation holds in global equilibrium, eq. (12)
can be generalized to local equilibrium by promoting the thermal vorticity to
be a local variable, i.e. $\varpi\mapsto\varpi(x)$, and integrating eq. (12)
over the decoupling hypersurface.

$S^{\mu}_{LE}(p)=\frac{\int{\rm d}\Sigma\cdot p\,n(p)\,S^{\mu}(p)}{\int{\rm
d}\Sigma\cdot p\,n(p)}.$ (13)

Figure 1: The components of the polarization vector along the total angular
momentum (left panel) and the beam direction (right panel) as a function of
the azimuthal angle. The blue dashed line is the linear approximation (1),
whereas the solid orange line corresponds to Eq. (13).

Here, in Fig. 1, I report the result of Eq. (13) compared to that of Eq. (1)
in Au-Au collisions at $\sqrt{s_{NN}}=30$ GeV in the centrality class 10-60%,
simulated using the code vHLLE Karpenko:2013wva  for the hydrodynamic
evolution. The initial state is an averaged entropy density profile generated
by GLISSANDO v.2.702 Rybczynski:2013yba . For the figure, I consider the case
of Dirac fermions at midrapidity. I remind the reader that the polarization is
related to the spin vector by $P^{\mu}=S^{\mu}/S$, where $S$ is the spin. One
can see that the two formulae give the same result, and it follows that the
linear approximation is a good one, at least for $\sqrt{s_{NN}}>30$ GeV.

##  5 Conclusions

In this work, I have shown how to calculate the spin density matrix and the
polarization vector for massive spin-S fields to all orders in thermal
vorticity and obtained an analytic finite expression for both quantities.
Using the spin density matrix, all spin observables can be computed, including
alignment. The polarization vector is a finite sum, representing the average
of the spin state weighted by Bose or Fermi distribution functions.

I have shown a numerical calculation assessing the importance of high-order
corrections in vorticity to the phenomenology of heavy ion collisions. As it
turns out, the exact solution and the linear approximation precisely coincide
for AuAu collisions at $30$ GeV. Yet, a small discrepancy may arise in lower-
energy collisions, where the thermal vorticity is larger. This work shows that
the linear approximation of the polarization vector is an excellent one in
most practical applications to heavy-ion collisions.

## References

  * (1) L. Adamczyk et al. (STAR), Nature 548, 62 (2017), 1701.06657
  * (2) J. Adam et al. (STAR), Phys. Rev. Lett. 123, 132301 (2019), 1905.11917
  * (3) J. Adam et al. (STAR), Phys. Rev. Lett. 126, 162301 (2021), 2012.13601
  * (4) S. Acharya et al. (ALICE), Phys. Rev. Lett. 128, 172005 (2022), 2107.11183
  * (5) S. Acharya et al. (ALICE), Phys. Rev. C 101, 044611 (2020), [Erratum: Phys.Rev.C 105, 029902 (2022)], 1909.01281
  * (6) W. Florkowski, B. Friman, A. Jaiswal, E. Speranza, Phys. Rev. C 97, 041901 (2018), 1705.00587
  * (7) W. Florkowski, A. Kumar, R. Ryblewski, Prog. Part. Nucl. Phys. 108, 103709 (2019), 1811.04409
  * (8) N. Weickgenannt, X.L. Sheng, E. Speranza, Q. Wang, D.H. Rischke, Phys. Rev. D 100, 056018 (2019), 1902.06513
  * (9) N. Weickgenannt, E. Speranza, X.l. Sheng, Q. Wang, D.H. Rischke, Phys. Rev. Lett. 127, 052301 (2021), 2005.01506
  * (10) J.H. Gao, Z.T. Liang, Q. Wang, X.N. Wang, Lect. Notes Phys. 987, 195 (2021), 2009.04803
  * (11) F. Becattini, M.A. Lisa, Ann. Rev. Nucl. Part. Sci. 70, 395 (2020), 2003.03640
  * (12) F. Becattini, Rept. Prog. Phys. 85, 122301 (2022), 2204.01144
  * (13) J.H. Gao, G.L. Ma, S. Pu, Q. Wang, Nucl. Sci. Tech. 31, 90 (2020), 2005.10432
  * (14) M. Hongo, X.G. Huang, M. Kaminski, M. Stephanov, H.U. Yee, JHEP 11, 150 (2021), 2107.14231
  * (15) F. Becattini, V. Chandra, L. Del Zanna, E. Grossi, Annals Phys. 338, 32 (2013), 1303.3431
  * (16) R.h. Fang, L.g. Pang, Q. Wang, X.n. Wang, Phys. Rev. C 94, 024904 (2016), 1604.04036
  * (17) F. Becattini, G. Cao, E. Speranza, Eur. Phys. J. C 79, 741 (2019), 1905.03123
  * (18) F. Becattini, M. Buzzegoli, A. Palermo, Phys. Lett. B 820, 136519 (2021), 2103.10917
  * (19) F. Becattini, M. Buzzegoli, G. Inghirami, I. Karpenko, A. Palermo, Phys. Rev. Lett. 127, 272302 (2021), 2103.14621
  * (20) B. Fu, S.Y.F. Liu, L. Pang, H. Song, Y. Yin, Phys. Rev. Lett. 127, 142301 (2021), 2103.10403
  * (21) S.Y.F. Liu, Y. Yin, JHEP 07, 188 (2021), 2103.09200
  * (22) B. Fu, L. Pang, H. Song, Y. Yin (2022), 2201.12970
  * (23) Y.B. Ivanov, Phys. Rev. C 103, L031903 (2021), 2012.07597
  * (24) M. Buzzegoli (2022), 2211.04549
  * (25) A. Palermo, F. Becattini, Eur. Phys. J. Plus 138, 547 (2023), 2304.02276
  * (26) F. Becattini, Lect. Notes Phys. 987, 15 (2021), 2004.04050
  * (27) F. Becattini, M. Buzzegoli, A. Palermo, JHEP 02, 101 (2021), 2007.08249
  * (28) A. Palermo, M. Buzzegoli, F. Becattini, JHEP 10, 077 (2021), 2106.08340
  * (29) J. Hubbard, B. Hubbard, _Vector Calculus, Linear Algebra, and Differential Forms: A Unified Approach_ (Prentice Hall, 2002), ISBN 9780130414083, https://books.google.com/books?id=bW92QgAACAAJ
  * (30) I. Karpenko, P. Huovinen, M. Bleicher, Comput. Phys. Commun. 185, 3016 (2014), 1312.4160
  * (31) M. Rybczynski, G. Stefanek, W. Broniowski, P. Bozek, Comput. Phys. Commun. 185, 1759 (2014), 1310.5475


# Magnifying Subtle Facial Motions for Effective 4D Expression Recognition

Qingkai Zhen  School of Computer Science and
Engineering, Beihang University, China
Email<EMAIL_ADDRESS>Di Huang  School of Computer Science and
Engineering, Beihang University, China
Email<EMAIL_ADDRESS>Yunhong Wang  School of Computer Science and
Engineering, Beihang University, China
Email<EMAIL_ADDRESS>Hassen Drira  Institut Mines-Télécom/Télécom Lille,
CRIStAL (UMR CNRS 9189), France
Email<EMAIL_ADDRESS>Boulbaba Ben Amor  Institut Mines-
Télécom/Télécom Lille,
CRIStAL (UMR CNRS 9189), France
Email<EMAIL_ADDRESS>Mohamed Daoudi  Institut Mines-
Télécom/Télécom Lille,
CRIStAL (UMR CNRS 9189), France
Email<EMAIL_ADDRESS>
###### Abstract

In this paper, an effective pipeline to automatic 4D Facial Expression
Recognition (4D FER) is proposed. It combines two growing but disparate ideas
in Computer Vision – computing the spatial facial deformations using tools
from Riemannian geometry and magnifying them using temporal filtering. The
flow of 3D faces is first analyzed to capture the spatial deformations based
on the recently-developed Riemannian approach proposed in [1], where
registration and comparison of neighboring 3D faces are led jointly. Then, the
obtained temporal evolution of these deformations are fed into a magnification
method in order to amplify the facial activities over the time. The latter,
main contribution of this paper, allows revealing subtle (hidden) deformations
which enhance the emotion classification performance. We evaluated our
approach on BU-4DFE dataset, the state-of-art 94.18% average performance and
an improvement that exceeds 10% in classification accuracy, after magnifying
extracted geometric features (deformations), are achieved.

##  I Introduction

Facial expressions are important non-verbal ways for human beings to
communicate their feeling and affective states. In recent years, as a major
topic of affective computing, Facial Expression Recognition (FER) has
attracted increasing interests due to its potential in many applications, such
as psychological analysis [2], transport security (driver fatigue), computer
graphics, human-machine interaction, animation of 3D avatars etc.

Sun and Yin, the pioneers of 4D FER, tracked the change of a generic
deformable model to extract a Spatio-Temporal (ST) descriptor of the face from
dynamic sequences of 3D scans [3]. The vertex flow tracking was applied to
each frame to form a set of motion trajectories of the 3D face video. The
spatio-temporal features and two-dimensional HMM were used for classification.
In [4], they proposed a tracking-model-based approach for vertex registration
and motion trajectory estimation in 4D FER. The 2D intermediary generated
through conformal mapping and a generic model adaptation algorithm were
employed to establish the correspondence across frames. ST-HMM which
incorporates 3D surface characterization was utilized to learn the spatial and
temporal information of faces. Canavan et al [5] described the 3D dynamic
surface by the surface curvature-based shape-index information, and the
surface features are characterized in local regions along the temporal axis. A
dynamic curvature descriptors is constructed from local regions as well as
temporal domains, and 3D tracking model based method was applied to locate the
local regions across 3D dynamic sequences. Sandbach et al [6] exploited 3D
motion-based features (Free-Form Deformation, FFD) between neighboring 3D
facial geometry frames for FER. A feature selection step was applied to
localize the features of each of the onset and offset segments of the
expression. The HMM classifier was used to model the full temporal dynamics of
each expression. In their another work [7], the entire expressive sequence is
modelled to contain an Onset followed by an Apex and an Offset. Feature
selection methods are applied in order to extract features for each of the
onset and offset segments of the expression. These features are then used to
train GentleBoost classifiers and build an HMM in order to model the full
temporal dynamics of the expression.

Ben Amor et al [1, 8, 9] presented the facial by collections of radial curves,
then Riemannian shape analysis was applied to quantify dense deformations and
extract motion from successive 3D frames. Two different classification schema
were performed, a HMM-based classifier and a mean deformation-based
classifier. Xue et al [10] extracted local depth patch-sequences from
consecutive expression frames based on the automatically detected facial
landmarks. Three dimension discrete cosine transform (3D-DCT) is then applied
on these patch-sequence to extract spatio-temporal features for facial
expression dynamic representation. Berretti et al [11] presented a fully-
automatic and real-time approach for 4D FER. A set of 3D facial landmarks were
automatically detected firstly, the local characteristics of the face around
those landmarks and their mutual distances were used to model the facial
deformation. The work of Fang et al [12, 13] emphasized 4D face data
registration and dense corresponding between 3D meshes along the temporal
line, and a variant of Local Binary Patters on Three Orthogonal Plane (LBP-
TOP) was introduced for static and dynamic feature extraction to predict
expression labels.

Even though the performance of FER has been substantially boosted by 4D data
in recent years, there still exist an unsolved problem, that is the reputed
similar expressions are difficult to distinguish since the facial deformations
are sometimes really slight [14]. To handle this issue, this paper present a
novel and effective approach to amplify the subtle facial deformation, the
contribution of the paper are two-folds:

  * •

A comprehensive pipeline of spatio-temporal processing for effective facial
expression recognition from 4D data.

  * •

A method to amplify subtle movements on facial surfaces which contributes to
distinguish similar expressions.

The rest of the paper is structured as follows. In section II the background
of the used feature is introduced. The magnification of subtle facial
deformation is introduced in Section III. Our comprehensive experimental study
is presented in Section IV, followed in Section V we conclude the paper.

##  II Background on Dense Scalar Fields

Figure 1: Top row: facial texture images of an individual with different
expressions. Bottom row: facial deformations in Riemannian space. Where, warm
colors are associated to the high $\chi$ and correspond to facial regions with
high deformations, cold colors reflect the most static parts of the 3D face.

Following the geometric approach recently-developed in [1], we represent 3D
facial surfaces by collections of radial curves emanating from the tip of the
nose. It is a new parameterization imposed for 3D face analysis, registration,
comparison, etc.The amount of deformation from one shape into another (across
the 3D video) is computed using tools from differential geometry through
analyzing shapes of 3D radial curves as explained in the following. In the
pre-processing step, the 3D mesh in each frame is first aligned to the first
one and then cropped. The facial surfaces are then approximated by indexed
collections of radial curves $\beta_{\alpha}$, where the index $\alpha$
denotes the angle formed by the curve with respect to a reference radial
curve. These curves are then uniformally resampled. Given a radial curve
$\beta$ of the face with an arbitrary orientation $\alpha\in[0,2\pi]$, it can
be parameterized as $\beta:I\to\mathbb{R}^{3}$, with $I=[0,1]$, and
mathematically represented using the SRVF, denoted by $q(t)$, according to:

$q(t)=\frac{\dot{\beta}(t)}{\sqrt{\|\dot{\beta}(t)\|}},t\in I.$ (1)

Figure 2: (a) Overview of 3D video magnification. The original facial
deformation features are first decomposed into different spatial frequencies,
and the temporal filter is applied to all the frequency bands. The filtered
spatial bands are then amplified by a given factor $\zeta$, added back to the
original signal, and collapsed to the output sequence. (b) An example of
facial expression deformation (norm of the velocity vector) before (green) and
after (red) magnification.

This special representation has the advantage of capturing the shape of the
curve and makes the calculus simpler. While there are several ways to analyze
shapes of curves, an elastic analysis of the parametrized curves is
particularly appropriate in our application – face analysis under facial
expression variations. This is because (1) such analysis uses the square-root
velocity function representation which allows us to compare local facial
shapes in presence of deformations, (2) this method uses a square-root
representation under which the elastic metric is reduced to the standard
$\mathbb{L}^{2}$ metric and thus simplifies the analysis, (3) under this
metric the group of re-parametrization acts by isometry on the curves
manifold, thus a Riemannian re-parametrization metric can be set between two
facial curves. Shown in Fig. 1 are examples of apex frames taken from the 3D
videos of the BU-4DFE dataset as well as the dense 3D deformations computed
with respect to the neutral frame. Let us define the space of the SRVFs as

$\mathcal{C}=\\{q:I\to\mathbb{R}^{3},\|q\|=1\\}\subset\mathbb{L}^{2}(I,\mathbb{R}^{3}),$
(2)

with$\|\cdot\|$ indicating the $\mathbb{L}^{2}$ norm. With the
$\mathbb{L}^{2}$ metric on its tangent space, $\mathcal{C}$ becomes a
Riemannian manifold. Basically, with this parametrization each radial curve is
represented on the manifold $\mathcal{C}$ by its SRVF. Accordingly, given the
SRVF $q_{1}$ and $q_{2}$ of two curves, the shortest path $\psi^{\ast}$ on the
manifold $\mathcal{C}$ between $q_{1}$ and $q_{2}$ (called geodesic path) is a
critical point of the following energy function:

$E(\psi)=\frac{1}{2}\int\|\dot{\psi}(\tau)\|^{2}d\tau,$ (3)

where $\psi$ denotes a path on the manifold $\mathcal{C}$ between $q_{1}$ and
$q_{2}$, $\tau$ is the parameter for traveling along the path $\psi$,
$\dot{\psi}(\tau)\in T_{\psi(\tau)}(\mathcal{C})$ is the tangent vector field
on the curve $\psi(\tau)\in\mathcal{C}$, and $\|\cdot\|$ denotes the
$\mathbb{L}^{2}$ norm on the tangent space. Since elements of $\mathcal{C}$
have a unit $\mathbb{L}^{2}$ norm, $\mathcal{C}$ is an hypersphere in the
Hilbert space $\mathbb{L}^{2}(I,\mathbb{R}^{3})$. As a consequence, the
geodesic path between any two points $q_{1}$ and $q_{2}\in\mathcal{C}$ is
given by the minor arc of the great circle connecting them. The tangent vector
field on this geodesic between the curves $\beta_{1}$ and $\beta_{2}$ making
the angle $\alpha$ with the reference curve is parallel along the geodesic and
one can represent it with the initial velocity vector (called also shooting
vector), without any loss of information.

$\frac{d\psi^{\ast}_{\alpha}}{d\tau}_{\arrowvert_{\tau=0}}=\frac{\theta}{\sin(\theta)}(q_{2}-\cos(\theta)q_{1}),\
(\theta\neq 0).$ (4)

where $\theta=d_{\mathcal{C}}(q_{1},q_{2})=cos^{-1}(\langle
q_{1},q_{2}\rangle)$ represent the length of the geodesic path connecting
$q_{1}$ to $q_{2}$. In practice, the curves are re-sampled to a discrete
number of points, say $T$, and the face is approximated by a collection of
$\arrowvert\Lambda\arrowvert$ curves. The norm of the quantity at each
discrete point $r$ is computed to represent the amount of 3D deformations at
each point of the surface parameterized by the pair $(\alpha,r)$, termed Dense
Scalar Fields (or DSFs). The final feature vector is of size
$T\times\arrowvert\Lambda\arrowvert$. We will refer to this quantity at a
given time $t$ of the 3D video by $\chi(t)$ (see bottom row of Fig. 1 for
illustrations). It provides the amplitude of the deformations between two
facial surfaces in a dense way.

##  III Subtle Facial Deformation Magnification

As described in Section II, $\chi$ reveals the shape difference of two facial
surfaces by deforming one mesh into another through an accurate registration
step. However, there exist another challenge to capture certain facial
movements, especially the slight ones, with low spatial amplitude, reflected
by the limited performance in distinguishing similar 3D facial expressions in
the literature. To solve this problem, we propose a novel approach to
highlight the subtle geometry changes of the facial surface in $\chi$ by
adapting the Eulerian spatio-temporal processing [15] to the 3D domain. This
method and its application to 3D face videos are presented in the subsequent.
The Eulerian spatio-temporal processing was introduced for motion
magnification in 2D videos, and has proved its effectiveness in [15]. Its
basic idea is to amplify the variation of pixel values over time, in a
spatially-multiscale manner, without explicitly estimating motion but rather
exaggerating motion by amplifying temporal color changes at fixed positions.
It relies on a linear approximation related to the brightness constancy
assumption that forms the basis of the optical flow algorithm. However, the
case is not that straightforward in 3D, because the vertex correspondence
across frames cannot be achieved as easy as that in 2D. Fortunately, during
the computation of $\chi$, such correspondence is established by surface
registration and remeshing. We can thus adapt Eulerian spatio-temporal
processing to 3D face video. We take into account the values of the time
series $\chi$ at any spatial location and highlight the differences in a given
temporal frequency band of interest. It thus combines spatial and temporal
processing to emphasize subtle changes in a 3D face video.

The process is illustrated in Fig. 2. Specifically, the video sequences are
first decomposed into different spatial frequency bands by Gaussian pyramid,
and these bands might be magnified differently. We consider that the time
series correspond to the values of $\chi$ on the mesh surfaces in a frequency
band and apply a band pass filter to extract the frequency bands of interest.
The temporal processing, $\mathfrak{T}$, is uniform for all spatial levels,
and for all $\chi$ within each level. We then multiply the extracted band
passed signal by a magnification factor $\zeta$, and add the magnified signal
to the original and collapse the spatial pyramid to obtain the final output.

Input: $\chi$, $l$-Gaussian pyramid levels, $\zeta$-amplification factor,
$\mathcal{\xi}$-sample rate, $\gamma$-attenuation rate, $\mathfrak{f}$-video
frequency

Step1. Spatial Processing

for  _$i=1;i\leq n$_ do

$\mathfrak{D}$($i,:,:,:$) = decompose the $\chi(i)$, with $l$ level Gaussian
pyramid.

Step2. Temporal Processing

$\mathfrak{S}$ = $\mathfrak{T}$($\mathfrak{D}$, $\mathfrak{f}$,
$\mathcal{\xi}$)

Step3. Magnification

for  _$i=1;i\leq 3$_ do

$\mathfrak{S(:,:,:,i)}$ = $\mathfrak{S(:,:,:,i)}\ast\zeta\ast\gamma$

Step3. Reconstruction

for  _$i=1;i\leq n$_ do

$\hat{\chi}(i)$ = $\mathfrak{S}(i,:,:,:)$ \+ $\chi(i)$

Output: $\hat{\chi}(t)$

Algorithm 1 Online 3D Deformation Magnification

For the translational motion of the facial mesh, we express the observed
$\chi(s,t)$ value with respect to a displacement function $\delta(t)$, such
that $\chi(s,t)=\chi(s)+\delta(t)$ and $\chi(s,0)=\chi(s)$. By using a first-
order Taylor series expansion, at time $t$, $\chi(s+\delta(t))$ in a first-
order Taylor expansion about $s$, as

$\chi(s,t)\thickapprox\chi(s)+\delta(t)\frac{\partial{\chi(s)}}{\partial{s}}$
(5)

Let $\phi(s,t)$ be the result of applying a broadband temporal band pass
filter to $\chi(s)$ at each position $(s)$. Assume that the motion signal
$\delta(t)$ is within the pass band of the temporal band pass filter.

$\phi(s,t)=\delta(t)\frac{\partial{\chi(s)}}{\partial{s}}$ (6)

Amplify the band pass signal by factor $\zeta$ and add it back to $\chi(s)$.

$\hat{\chi}(s,t)=\chi(s,t)+\zeta\phi(s,t)$ (7)

By combining Eqn. 5, 6, and 7, we reach

$\hat{\chi}(s,t)\thickapprox\chi(s)+(1+\zeta)\delta(t)\frac{\partial{\chi(s)}}{\partial{s}}$
(8)

Assuming that the first-order Taylor expansion holds for the amplified larger
perturbation $(1+\zeta)\delta(t)$, the motion magnification of 3D face video
can be simplified as follows:

$\hat{\chi}(s,t)\thickapprox\chi(s+(1+\zeta)\delta(t))$ (9)

This shows that the spatial displacement $\delta(t)$ of the $\chi(s)$ at time
$t$, is amplified to a magnitude of $(1+\zeta)$. Sometimes $\delta(t)$ is not
entirely within the pass band of the temporal filter. In this case, let
$\delta_{k}(t)$, indexed by $k$, represent the different temporal spectral
components of $\delta(t)$. This results in a band pass signal,

$\phi(s,t)=\sum_{k}\gamma_{k}\delta_{k}(t)\frac{\partial{\chi(s)}}{\partial{s}}$
(10)

where $\gamma$ is an attenuation factor. Temporal frequency dependent
attenuation can be equivalently interpreted as a frequency-dependent motion
magnification factor, $\zeta_{k}=\gamma\zeta$, and the amplified output signal

$\hat{\chi}(s,t)\thickapprox\chi(s+\sum_{k}(1+\zeta{k})\delta_{k}(t))$ (11)

this procedure, and Fig. 2 displays an example of facial deformation
trajectory before (green) and after (red) magnification.

##  IV Experimental Results

###  IV-A Dataset Description and Experimental Settings

The BU-4DFE dataset [3] is a dynamic 3D facial expression dataset which
consists of 3D facial sequences of $58$ females and $43$ males. It includes in
total $606$ 3D sequences according to the $6$ universal expressions. Each 3D
sequence captures a facial expression at a rate of $25$ fps (frames per
second) and lasts approximately $3$-$4$ seconds.

In our experiments, at a time $t$, the 3D face model $f^{t}$ is approximated
by a set of $200$ elastic radial curves originating from the tip of the nose,
a total of 50 sampled vertices on each curve is considered. Based on this
parameterization, the 3D face shapes along the video sequence are compared to
a reference frame $f^{0}$ to derive the $\chi(t)$ at each time $t$. Then,
within the spatial processing step, a Gaussian pyramid decomposition is used
to decompose $\chi$ into $4$ band levels. Finally, a temporal processing to
all the bands is applied. The factor $\zeta$ is set to $10$, the sample rate
$\mathcal{\xi}$ is set to $25$, $\mathfrak{f}\in[0.3,0.4]$, and the
attenuation rate $\gamma$ is set to $1$. Our experiments are conducted on the
following sub-pipelines – (1) the whole video sequence (denoted by WV), (2)
the magnified whole video sequence (denoted as MWV).

A multi-class Support Vector Machine (SVM) is considered here where
$\bar{\chi}$ is treated as a feature vector to predict the video label in the
Subtle Deformation Magnification experiment (using SVM Classifier). We also
adopt HMM to encode the temporal behavior of each expression, and get the
expression type. To allow fair comparison with previous studies, we randomly
select $60$ subjects from the BU-4DFE dataset to perform our experiments under
a $10$-fold cross-validation protocol.

###  IV-B Classification Performance

TABLE I: Average accuracy (and standard deviations) achieved by SVM and HMM on whole video sequence before and after magnification. Algorithm | Magnification? |  Whole Sequence (%)
---|---|---
SVM on $\bar{\chi}$ | N | $82.49\pm 3.11$
Y | $93.39\pm 3.54$
HMM on $\chi(t)$ | N | $83.19\pm 2.84$
Y | $94.18\pm 2.46$
TABLE II: The confusion matrices (WV, MWV) achieved by using SVM and HMM classifiers. SVM on $\bar{\chi}$ | Whole Video (WV) | Magnified Whole Video (MWV)
---|---|---
% | AN | DI | FE | HA | SA | SU | AN | DI | FE | HA | SA | SU
AN | 73.86 | 9.18 | 6.49 | 1.75 | 6.11 | 2.51 | 91.07 | 2.73 | 2.01 | 1.59 | 2.08 | 0.51
DI | 8.76 | 71.27 | 9.29 | 3.51 | 4.84 | 2.21 | 2.05 | 92.62 | 2.63 | 1.07 | 1.38 | 0.24
FE | 5.79 | 5.37 | 73.14 | 4.59 | 5.39 | 5.61 | 1.66 | 1.53 | 92.33 | 1.31 | 1.54 | 1.62
HA | 0.81 | 1.18 | 2.42 | 93.6 | 1.08 | 0.88 | 0.91 | 0.88 | 2.36 | 94.29 | 0.97 | 0.58
SA | 2.54 | 2.27 | 2.99 | 1.63 | 88.75 | 1.77 | 1.36 | 1.22 | 1.62 | 0.9 | 93.93 | 0.96
SU | 0.74 | 0.88 | 1.91 | 0.75 | 1.38 | 94.32 | 0.51 | 0.61 | 1.29 | 0.52 | 0.96 | 96.11
Average | 82.49 $\pm$ 3.10 | 93.39 $\pm$ 3.54
HMM on $\chi(t)$ |  |
% | AN | DI | FE | HA | SA | SU | AN | DI | FE | HA | SA | SU
AN | 75.29 | 5.88 | 7.31 | 1.14 | 8.17 | 2.21 | 91.87 | 1.91 | 2.41 | 0.38 | 2.69 | 0.73
DI | 10.42 | 71.55 | 11.43 | 1.82 | 4.27 | 0.5 | 2.11 | 94.22 | 2.32 | 0.29 | 0.86 | 0.19
FE | 5.07 | 6.86 | 73.69 | 3.33 | 8.06 | 2.99 | 1.37 | 1.86 | 92.85 | 0.91 | 2.19 | 0.81
HA | 0.48 | 0.87 | 1.54 | 94.93 | 1.81 | 0.37 | 0.47 | 0.77 | 1.43 | 95.3 | 1.67 | 0.35
SA | 3.71 | 1.01 | 4.17 | 0.65 | 89.19 | 1.26 | 1.84 | 0.51 | 2.07 | 0.33 | 94.61 | 0.63
SU | 0.49 | 0.33 | 2.79 | 0.32 | 1.59 | 94.47 | 0.33 | 0.22 | 1.89 | 0.22 | 1.08 | 96.25
Average | 83.19 $\pm$ 2.83 | 94.18 $\pm$ 2.46

Table I provides a first summary of our results, it shows that the
magnification procedure an improvement that exceeds 10% in classification
accuracy is achieved. Before magnification, our approach achieves 82.49% and
83.19% as correct classification rate using SVM and HMM classifiers
respectively, when consider full video. In particular, the SU and HA sequences
are well classified with a high classification rate. This is mainly due to the
high intensities and clear patterns of their deformations. However, remaining
expressions (DI, FE, AN and SA) are harder to distinguish. We believe that two
major reasons induce this difficulty: (1) the intra-class variability which
makes confusing similar classes such as DI/AN/FE; (2) the low magnitude of the
deformations exhibited when performing these expressions. After the
magnification, the overall improvement exceeds 10% both all settings, which
highlights the interest of the magnification procedure to reveal invisible
deformations. It can be clearly seen from these confusion matrices, a
significant improvement in distinguishing the AN, the DI and the FE
expressions which have been difficult to recognize before. Table II shows the
confusion matrices (WV, MWV) achieved by using SVM and HMM classifiers. Fig. 3
illustrations of the deformation magnification on sequences of the same
subject.

###  IV-C Comparison with state-of-the-art

Several research groups have reported FER results on the BU-4DFE dataset,
however they differ in their experimental settings. In this section, we
compare our results with existing approaches when considering these
differences.

TABLE III: A comparative study of the proposed approach with the state-of-the-art on BU-4DFE. Method | Experimental Settings | Accuracy
---|---|---
Sun et al. [3] | 6E, 60S, 10-CV, Win=6 | 90.44%
Sun et al. [4] | 6E, 60S, 10-CV, Win=6 | 94.37%
Reale et al. [16] | 6E, 100S, –, Win=15 | 76.9%
Sandb. et al. [7] | 6E, 60S, 6-CV, Win | 64.6%
Fang et al. [13] | 6E, 100S, 10-CV, – | 74.63%
Le et al. [17] | 3E, 60S, 10-CV, Full seq. | 92.22%
Xue et al. [10] | 6E, 60S, 10-CV, Full seq. | 78.8%
Berretti et al. [11] | 6E, 60S, 10-CV, Full seq. | 79.4%
Berretti et al. [11] | 6E, 60S, 10-CV, Win=6 | 72.25%
Ben Amor et al. [1] | 6E, 60S, 10-CV, Full seq. | 93.21%
Ben Amor et al. [1] | 6E, 60S, 10-CV, Win=6. | 93.83%
This work – SVM on $\bar{\chi}$ | 6E, 60S, 10-CV, Full seq. |  93.39%
This work – HMM on $\chi(t)$ | 6E, 60S, 10-CV, Full seq. |  94.18%

Most of the results reported on BU-4DFE are shown in Table III. In this table,
$\\#$E means the number of expressions, $\\#$S is the number of subjects,
$\\#$-CV provides the number of cross-validation used, Full Seq./Win means the
decision is made based on the analysis of the full sequence or on sub-
sequences captured using a sliding window. The studies [3, 4] report one of
the highest accuracy when using a sliding window of $6$ frames, nevertheless,
the approach requires manual annotation of 83 landmarks on the first frame.
Moreover, the vertex-level dense tracking scheme is time consuming. In a more
recent work from the same group developed by Reale et al. [16], the authors
propose 4D (Space-Time) features termed Nebula computed on a fixed-size window
of $15$ frames. The best accuracy reported is 76.9% using sequences of $100$
subjects, when the 3D video segmentation to limit the expressive time interval
is performed manually. In [13], Fang et al. obtained an accuracy of 74.63%
with $507$ sequences of $100$ subjects. Le et al. [17] evaluate their
algorithm consisting of level curves and HMMs on $60$ subjects sequences on
three expressions (HA, SA and SU) and display an accuracy of 92.22%. It should
be noted that the proposed approach is evaluated when considering full
sequences which is a major difference to the works [3, 4, 16, 7]. It is
pointed out in [11] that the problem of the window-based evaluation protocol
is to label all sub-sequences from the neutral intervals as one of the six
expressions which can bias the final result. Compared to the results listed in
Table III, the proposed approach outperforms existing approaches, where (1) no
landmark detection is required; (2) no dimensionality reduction or feature
selection techniques are applied; and (3) A vertex-level 4D dense registration
and quantification of the deformations are led jointly through a Riemannian
approach. The temporal filtering amplifies these deformations and consequently
reveals hidden (subtle) 4D facial motions.

Figure 3: Illustrations of the deformation magnification on sequences of the
same subject performing six expressions. One can appreciate the magnification
effects on 3D deformations compared to those of the original features. From up
to bottom, each row presents the texture image, the original features, and the
amplified features, respectively.

##  V Conclusions

In this paper, a spatio-temporal processing approach for effective 4D FER is
presented. It focus on an important issue – expression magnification to reveal
subtle deformations. After a preprocessing step, the flow of 3D faces is
analyzed to capture the spatial deformations based on the Riemannian method
where registration and comparison are achieved jointly. Then, the obtained
deformations are amplified using the temporal filter over the 3D face video.
The combination of these two ideas performs accurate vertex-level registration
of 4D faces and reveal hidden (subtle) deformations in 3D facial sequences.
Experiments on BU-4DFE dataset demonstrate the effectiveness of the proposed
method.

## References

  * [1] B. Ben Amor, H. Drira, S. Berretti, M. Daoudi, and A. Srivastava, “4-d facial expression recognition by learning geometric deformations,” _IEEE Transactions on Cybernetics_ , vol. 44, no. 12, pp. 2443–2457, 2014.
  * [2] H. Meng, D. Huang, H. Wang, H. Yang, M. AI-Shuraifi, and Y. Wang, “Depression recognition based on dynamic facial and vocal expression features using partial least square regression,” in _International Workshop on Audio/Visual Emotion Challenge_ , 2013, pp. 21–30.
  * [3] Y. Sun and L. Yin, “Facial expression recognition based on 3d dynamic range model sequences,” in _Europeon Conference on Computer Vision_ , 2008, pp. 58–71.
  * [4] Y. Sun, X. Chen, M. Rosato, and L. Yin, “Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis,” _IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans_ , vol. 40, no. 3, pp. 461–474, 2010.
  * [5] S. Canavan, Y. Sun, X. Zhang, and L. Yin, “A dynamic curvature based approach for facial activity analysis in 3d space,” in _Computer Vision and Pattern Recognition Workshops_ , 2012, pp. 14–19.
  * [6] G. Sandbach, S. Zafeiriou, M. Pantic, and D. Rueckert, “A dynamic approach to the recognition of 3d facial expressions and their temporal models,” in _IEEE International Conference on Automatic Face Gesture Recognition and Workshops_ , 2011, pp. 406–413.
  * [7] ——, “Recognition of 3d facial expression dynamics,” _Image and Vision Computing_ , vol. 30, no. 10, pp. 762–773, 2012.
  * [8] H. Drira, B. B. Amor, M. Daoudi, A. Srivastava, and S. Berretti, “3d dynamic expression recognition based on a novel deformation vector field and random forest,” in _International Conference on Pattern Recognition_ , 2012, pp. 1104–1107.
  * [9] M. Daoudi, H. Drira, B. B. Amor, and S. Berretti, “A dynamic geometry-based approach for 4d facial expressions recognition,” in _European Workshop on Visual Information Processing_ , 2013, pp. 280–284.
  * [10] M. Xue, A. Mian, W. Liu, and L. Li, “Automatic 4d facial expression recognition using dct features,” in _IEEE Winter Conference on Applications of Computer Vision_ , pp. 199–206.
  * [11] S. Berretti, A. Del Bimbo, and P. Pala, “Automatic facial expression recognition in real-time from dynamic sequences of 3d face scans,” _The Visual Computer_ , vol. 29, no. 12, pp. 1333–1350, 2013.
  * [12] T. Fang, X. Zhao, S. K. Shah, and I. A. Kakadiaris, “4d facial expression recognition,” in _IEEE International Conference on Computer Vision Workshops_ , 2011, pp. 1594–1601.
  * [13] T. Fang, X. Zhao, O. Ocegueda, S. K. Shah, and I. A. Kakadiaris, “3d/4d facial expression analysis: An advanced annotated face model approach,” _Image and Vision Computing_ , vol. 30, no. 10, pp. 738–749, 2012.
  * [14] X. Yang, D. Huang, Y. Wang, and L. Chen, “Automatic 3d facial expression recognition using geometric scattering representation,” in _IEEE International Conference on Automatic Face and Gesture Recognition and Workshops_ , vol. 1, 2015, pp. 1–6.
  * [15] H.-Y. Wu, M. Rubinstein, E. Shih, J. Guttag, F. Durand, and W. Freeman, “Eulerian video magnification for revealing subtle changes in the world,” _ACM Trans. Graph._ , vol. 31, no. 4, pp. 1–8, 2012.
  * [16] M. Reale, X. Zhang, and L. Yin, “Nebula feature: A space-time feature for posed and spontaneous 4D facial behavior analysis.” in _IEEE International Conference on Automatic Face and Gesture Recognition_ , 2013, pp. 1–8.
  * [17] V. Le, H. Tang, and T. Huang, “Expression recognition from 3d dynamic faces using robust spatio-temporal shape features,” in _IEEE International Conference on Automatic Face Gesture Recognition and Workshops_ , 2011, pp. 414–421.

## References

  * [1] B. Ben Amor, H. Drira, S. Berretti, M. Daoudi, and A. Srivastava, “4-d facial expression recognition by learning geometric deformations,” _IEEE Transactions on Cybernetics_ , vol. 44, no. 12, pp. 2443–2457, 2014.
  * [2] H. Meng, D. Huang, H. Wang, H. Yang, M. AI-Shuraifi, and Y. Wang, “Depression recognition based on dynamic facial and vocal expression features using partial least square regression,” in _International Workshop on Audio/Visual Emotion Challenge_ , 2013, pp. 21–30.
  * [3] Y. Sun and L. Yin, “Facial expression recognition based on 3d dynamic range model sequences,” in _Europeon Conference on Computer Vision_ , 2008, pp. 58–71.
  * [4] Y. Sun, X. Chen, M. Rosato, and L. Yin, “Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis,” _IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans_ , vol. 40, no. 3, pp. 461–474, 2010.
  * [5] S. Canavan, Y. Sun, X. Zhang, and L. Yin, “A dynamic curvature based approach for facial activity analysis in 3d space,” in _Computer Vision and Pattern Recognition Workshops_ , 2012, pp. 14–19.
  * [6] G. Sandbach, S. Zafeiriou, M. Pantic, and D. Rueckert, “A dynamic approach to the recognition of 3d facial expressions and their temporal models,” in _IEEE International Conference on Automatic Face Gesture Recognition and Workshops_ , 2011, pp. 406–413.
  * [7] ——, “Recognition of 3d facial expression dynamics,” _Image and Vision Computing_ , vol. 30, no. 10, pp. 762–773, 2012.
  * [8] H. Drira, B. B. Amor, M. Daoudi, A. Srivastava, and S. Berretti, “3d dynamic expression recognition based on a novel deformation vector field and random forest,” in _International Conference on Pattern Recognition_ , 2012, pp. 1104–1107.
  * [9] M. Daoudi, H. Drira, B. B. Amor, and S. Berretti, “A dynamic geometry-based approach for 4d facial expressions recognition,” in _European Workshop on Visual Information Processing_ , 2013, pp. 280–284.
  * [10] M. Xue, A. Mian, W. Liu, and L. Li, “Automatic 4d facial expression recognition using dct features,” in _IEEE Winter Conference on Applications of Computer Vision_ , pp. 199–206.
  * [11] S. Berretti, A. Del Bimbo, and P. Pala, “Automatic facial expression recognition in real-time from dynamic sequences of 3d face scans,” _The Visual Computer_ , vol. 29, no. 12, pp. 1333–1350, 2013.
  * [12] T. Fang, X. Zhao, S. K. Shah, and I. A. Kakadiaris, “4d facial expression recognition,” in _IEEE International Conference on Computer Vision Workshops_ , 2011, pp. 1594–1601.
  * [13] T. Fang, X. Zhao, O. Ocegueda, S. K. Shah, and I. A. Kakadiaris, “3d/4d facial expression analysis: An advanced annotated face model approach,” _Image and Vision Computing_ , vol. 30, no. 10, pp. 738–749, 2012.
  * [14] X. Yang, D. Huang, Y. Wang, and L. Chen, “Automatic 3d facial expression recognition using geometric scattering representation,” in _IEEE International Conference on Automatic Face and Gesture Recognition and Workshops_ , vol. 1, 2015, pp. 1–6.
  * [15] H.-Y. Wu, M. Rubinstein, E. Shih, J. Guttag, F. Durand, and W. Freeman, “Eulerian video magnification for revealing subtle changes in the world,” _ACM Trans. Graph._ , vol. 31, no. 4, pp. 1–8, 2012.
  * [16] M. Reale, X. Zhang, and L. Yin, “Nebula feature: A space-time feature for posed and spontaneous 4D facial behavior analysis.” in _IEEE International Conference on Automatic Face and Gesture Recognition_ , 2013, pp. 1–8.
  * [17] V. Le, H. Tang, and T. Huang, “Expression recognition from 3d dynamic faces using robust spatio-temporal shape features,” in _IEEE International Conference on Automatic Face Gesture Recognition and Workshops_ , 2011, pp. 414–421.