# Multilingual Large Language Models and Curse of Multilinguality

Daniil Gurgurov1,2 Tanja Bäumel2 Tatiana Anikina 2
1Universität des Saarlandes
2German Research Center for Artificial Intelligence (DFKI)
{daniil.gurgurov, tanja.baeumel<EMAIL_ADDRESS>
###### Abstract

Multilingual Large Language Models (LLMs) have gained large popularity among
Natural Language Processing (NLP) researchers and practitioners. These models,
trained on huge datasets, show proficiency across various languages and
demonstrate effectiveness in numerous downstream tasks. This paper navigates
the landscape of multilingual LLMs, providing an introductory overview of
their technical aspects. It explains underlying architectures, objective
functions, pre-training data sources, and tokenization methods. This work
explores the unique features of different model types: encoder-only (mBERT,
XLM-R), decoder-only (XGLM, PALM, BLOOM, GPT-3), and encoder-decoder models
(mT5, mBART). Additionally, it addresses one of the significant limitations of
multilingual LLMs - the curse of multilinguality - and discusses current
attempts to overcome it.

Multilingual Large Language Models and Curse of Multilinguality

Daniil Gurgurov1,2 Tanja Bäumel2 Tatiana Anikina 2 1Universität des Saarlandes
2German Research Center for Artificial Intelligence (DFKI) {daniil.gurgurov,
tanja.baeumel<EMAIL_ADDRESS>

##  1 Introduction

Large Language Models (LLMs) Devlin et al. (2019); Lewis et al. (2020); Liu et
al. (2019) have made a significant impact on the field of Natural Language
Processing (NLP), showing effectiveness in various tasks. The remarkable
aspect of LLMs is their capacity to learn a language during pre-training and
enhance their expertise for specific tasks during fine-tuning. Pre-training
involves the acquisition of knowledge, where a model grasps language
structures by analyzing huge datasets. Fine-tuning, on the other hand,
specializes the model by adjusting its parameters so that it can perform
specific downstream tasks using a smaller set of examples compared to those
used in pre-training.

Another significant advancement involves teaching a model to comprehend
multiple languages, leading to the concept of multilingual LLMs Devlin et al.
(2018); Pires et al. (2019); Conneau et al. (2020); Xue et al. (2021); Liu et
al. (2020). While monolingual LLMs focus on understanding patterns within a
single language, multilingual LLMs simultaneously learn from multiple
languages. This is accomplished by exposing these models to data from various
languages during the pre-training phase. Furthermore, variations in the
architectures of multilingual LLMs contribute to their strengths in certain
tasks while potentially limiting their effectiveness in others.

This paper aims to provide a brief overview of the architectures of the most
prominent multilingual LLMs, including details such as their pre-training
objective functions, data sources, tokenization schemas, the number of
languages supported, and the peculiarities of each individual multilingual
LLM. Subsequently, the primary challenge facing multilingual LLMs, known as
the "curse of multilinguality" Conneau et al. (2020), and the current attempts
to solve it, are discussed.

Model | Architecture | Training Data Sources | Languages | Tokenization Schema
---|---|---|---|---
mBERT | Encoder | Wikipedia | 104 | WordPiece
XLM-R | Encoder | CC | 100 | SentencePiece
mBART | Encoder-Decoder | CC25 | 25 | SentencePiece
mT5 | Encoder-Decoder | C4 | 101 | SentencePiece
XGLM | Decoder | CC100-XL | 134 | SentencePiece
PALM | Decoder | Wikipedia, books, webpages, social media & source code | 124 | SentencePiece
BLOOM | Decoder | ROOTS, OSCAR | 46 | Byte Pair Encoding
GPT-3 | Decoder | CC, Wikipedia, WebText2, Books1&2 | >95 | Byte Pair Encoding
Table 1: Comparison of Multilingual Large Language Models

##  2 Technicalities

This section delves into the technical details behind multilingual LLMs,
providing a solid foundation for thorough conceptual understanding of their
inner workings.

###  2.1 Architectures

LLMs, including multilingual LLMs, are typically constructed using the
Transformer architecture Vaswani et al. (2017). These models can be
categorized into three main architectural types: Encoder-only, Decoder-only,
and Encoder-Decoder.

Figure 1: The Transformer Architecture Vaswani et al. (2017)

The original Transformer implementation was intended for machine translation
purposes and was of the Encoder-Decoder type, since it included both an
Encoder and a Decoder. The architecture is illustrated in figure 1 and
described as follows.

The first stage of the architecture involves an Encoder. Initially, the input
sentence is split into tokens, which are mapped to their vocabulary indices
and converted into embeddings within the "Input embeddings" layer.
Simultaneously, positional encoding vectors Gehring et al. (2017) are added to
the original token embeddings to capture token position information. The
embeddings then undergo a multi-head self-attention mechanism, allowing the
model to focus on relevant information while generating the output and
enhancing the ability to capture long-range dependencies. This is achieved by
calculating attention scores based on the similarity between each token and
all other tokens in the sequence. The attention step is followed by the "Add &
Norm" layer Ba et al. (2016), serving as a normalization procedure and
enhancing gradient flow by combining initial token embeddings to the multi-
head attention output. Subsequently, the outputs from the previous step pass
through a two-layer Feed Forward Neural Network (FFNN) Bebis and Georgiopoulos
(1994), followed by another "Add & Norm" layer. FFNN refines and enriches the
token representations coming from the self-attention mechanism, helping the
model to capture both local and global contextual information. Finally, the
representations are fed into a multi-head self-attention mechanism of the
decoder.

The decoder follows a similar structure to the encoder but with some important
differences. The input sequence is tokenized, converted into embeddings, and
positionally encoded. The decoder’s self-attention is applied in two steps:
masked self-attention and subsequent multi-head self-attention. The masked
self-attention mechanism ensures that each position in the encoder can only
attend to positions before it, which prevents information leakage from future
tokens. After the first attention steps, "Add & norm" is applied, and outputs
move to a decoder-encoder attention layer, enabling the encoder to attend to
the decoder output. This mechanism facilitates the alignment of input and
output sequences, helping with information capture for token generation. Then,
"Add & Norm" is used, and token embeddings pass through another two-layer
FFNN, followed by "Add & Norm". At this stage, the representations for output
are generated, going through a final layer and producing a probability
distribution over the vocabulary. The model outputs the token with the highest
probability for each position. During training, the model is optimized using
backpropagation and an optimization algorithm Dreyfus (1990).

####  2.1.1 Encoder-Decoder

In the context of multilingual LLMs, the Encoder-Decoder architecture is
utilized for tasks such as machine translation. An exemplary model employing
this architecture is the mBART model Liu et al. (2020). mBART uses the
Transformer architecture in an Encoder-Decoder style, where the encoder
processes the input text, and the decoder generates the corresponding
translated output. This architecture allows the model to effectively capture
language-specific details during translation.

####  2.1.2 Encoder-only

Multilingual Encoder-only architectures are designed for tasks that require
understanding input sequences without the need for sequence generation. mBERT
Devlin et al. (2018) is a prominent example of an Encoder-only model. Trained
on a large set of languages, mBERT’s encoder is capable of extracting
contextualized representations of input. This makes it valuable for various
downstream tasks, such as sentiment analysis and named entity recognition,
where understanding language context is important.

####  2.1.3 Decoder-only

In specific multilingual cases, a Decoder-only architecture is beneficial.
Models like XGLM Lin et al. (2022a) follow a Decoder-only configuration. This
architecture is particularly useful for tasks where the focus is on generating
sequences, such as language modeling or text completion. XGLM is a powerful
decoder model that generates coherent sequences in various languages without
the need for an encoder.

###  2.2 Objective Functions

In this section, the most popular objective functions used for pre-training
multilingual LLMs are described: Masked Language Modeling (MLM) Devlin et al.
(2019), Causal Language Modeling (CLM) Conneau and Lample (2019), Next
Sentence Prediction (NSP) Devlin et al. (2019), and Translation Language
Modeling (TLM) Conneau and Lample (2019). These pre-training objective
functions serve as foundational tasks for training language models.

####  2.2.1 MLM

Inspired by a Cloze task Taylor (1953), Masked Language Modeling involves
randomly masking certain words in a sentence and training the model to predict
these masked words based on the context provided by the surrounding words.
This objective function teaches the model to understand the relationships and
dependencies between different words in a sentence, promoting a deeper
understanding of syntax, semantics, and contextual nuances within the text.

####  2.2.2 CLM

Causal Language Modeling is designed for autoregressive models, whose focus is
on the next token generation. It involves predicting the next word in a
sequence given the preceding context. This pre-training objective is
particularly effective for tasks where the order of the input sequence is
important, as the model learns to capture dependencies and temporal
relationships between words.

####  2.2.3 NSP

Next Sentence Prediction trains models to predict whether a given pair of
sentences is contiguous or not. The models learn to understand the coherence
and logical flow between sentences. This pre-training objective is useful for
tasks requiring comprehension of discourse and context, such as question-
answering and document summarization.

####  2.2.4 TLM

Translation Language Modeling extends the pre-training objective to involve
parallel data in different languages. The sentences in different languages are
concatenated, and words are randomly masked in each of them. Predicting the
missing words in this case requires the model to understand the semantic
relationships between sentences in different languages and helps in capturing
cross-lingual representations. TLM is particularly beneficial for multilingual
models, enabling them to transfer knowledge across languages.

###  2.3 Pre-training data

In the pre-training phase, multilingual LLMs leverage two primary types of
data: large monolingual corpora in individual languages and parallel corpora
across languages. The data for most models is taken from one of the following
datasets: Wikipedia Vrandečić and Krötzsch (2014), Common Crawl (CC) Corpus
Wenzek et al. (2020), ROOTS Laurençon et al. (2023), OSCAR Abadji et al.
(2022), WebText2 Gao et al. (2020), Books1, and Books2 Brown et al. (2020).
The choice of training corpora varies among multilingual LLMs. For instance,
mBERT leverages Wikipedia as its training data, while XLM-R uses the more
extensive CC corpus.

####  2.3.1 Wikipedia

The Wikipedia dataset includes cleaned articles in various languages, sourced
from Wikipedia dumps. Each language has a separate subset, with examples
representing entire articles. The content is cleaned by removing markdown and
extra sections.

####  2.3.2 Common Crawl

The Common Crawl corpus is a vast collection of web data accumulated over more
than 10 years through web crawling. It includes raw web page data, metadata
extracts, and text extracts. This non-curated dataset includes web pages in
numerous languages, providing a rich source for multilingual pre-training.

####  2.3.3 ROOTS

The Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus is
a comprehensive dataset developed by the BigScience workshop, an international
and multidisciplinary initiative focusing on researching and LLMs with an
emphasis on ethics, harm, and governance. Spanning 1.6 terabytes, the ROOTS
corpus serves as a foundational resource for training large-scale language
models.

####  2.3.4 OSCAR

The OSCAR project is an open-source initiative aimed at providing web-based
multilingual data for ML and AI applications. It offers large quantities of
raw data obtained through high-performance data pipelines. OSCAR focuses on
improving data quality, particularly for low-resource languages.

####  2.3.5 WebText2

OpenWebText2 is an improved version of the original OpenWebTextCorpus. It
covers all Reddit posts from 2005 to April 2020, with additional months
becoming available over time. This dataset is another contribution to the
diverse training material for multilingual language models.

####  2.3.6 Books1 and Books2

Books1 and Books2 comprise internet-based book corpora, featuring a random
sampling of public domain books, as well as modern published literature in
e-book format. The contents are drawn from available online books, offering a
mix of historical and contemporary literature. It was used for pre-training
GPT-3 Brown et al. (2020) and does not seem to be publicly available.

###  2.4 Languages

Multilingual LLMs demonstrate diversity in terms of the number of languages
they support. Models like XLM-R show huge multilingual capabilities,
facilitating approximately 100 languages. In contrast, models like mBART
target a smaller set of languages, as shown in Table LABEL:table:multilingual.
Managing the imbalance in pre-training data among languages, especially when
dealing with a large number of them, is a challenge. For example, high-
resource languages like English have significantly more data available
compared to lower-resource languages like Maltese or Odia Joshi et al. (2020).
To address these imbalances, multilingual LLMs often employ exponentially
smoothed weighting Devlin et al. (2018). This approach ensures relatively fair
representation of low-resource languages in the model’s training data,
preventing them from being underrepresented in the overall vocabulary.

###  2.5 Tokenization Techniques

Tokenization is a critical part in language processing, and several techniques
have been developed to represent words effectively. In this section, three
widely used tokenization methods are introduced: Byte Pair Encoding (BPE),
WordPiece, and SentencePiece.

####  2.5.1 Byte Pair Encoding

BPE Sennrich et al. (2016) is a subword tokenization technique that builds a
vocabulary by iteratively merging the most frequent pairs of consecutive
bytes. This method is particularly effective in representing both common words
and rare subword units in a wide range of languages and character sets.

####  2.5.2 WordPiece

WordPiece Schuster and Nakajima (2012) is another subword tokenization
algorithm that starts with a vocabulary of individual characters and
iteratively merges the most frequent character pairs. This process creates a
vocabulary of subword units, allowing the model to represent words as
combinations of subword tokens.

####  2.5.3 SentencePiece

SentencePiece Kudo and Richardson (2018) is a text tokenizer that works at the
subword level. Unlike the previous tokenizers, it takes the input sequence as
a raw stream, including the space in the collection of characters. It employs
a unigram language model to tokenize text into pieces, making it suitable for
various languages. It was introduced as a solution for the problem posed by
languages that do not use spaces to separate words.

##  3 Multilingual Large Language Models

This section presents prominent examples of multilingual LLMs categorized into
three types: Encoder-Only, Decoder-Only, and Encoder-Decoder, along with their
intricacies. Table LABEL:table:multilingual summarizes the important points
regarding all presented models.

###  3.1 Encoder-Only Models

###  3.2 mBERT

The first notable example of a genuinely multilingual neural LLM is the
multilingual Bidirectional Encoder Representations from Transformers (mBERT)
Devlin et al. (2019); Wenzek et al. (2020). This model adapts the architecture
similar to the original BERT Devlin et al. (2019), consisting of 12
Transformer layers and operating as an encoder-only model. The distinguishing
factor lies in the model being trained on the entire Wikipedia data covering
104 languages Pires et al. (2019), in contrast to the original BERT, which was
trained only on English. Additionally, mBERT employs a shared vocabulary for
all languages based on the WordPiece tokenization schema. Pre-training mBERT
involves two unsupervised tasks: MLM and NSP.

###  3.3 XLM-R

Cross-lingual Language Modeling RoBERTa (XLM-R) Conneau et al. (2020) is a
12-layers Transformer-based multilingual model that adopts an encoder-only
architecture following the XLM Conneau and Lample (2019) approach. It
integrates RoBERTa, introducing simple improvements to the learning procedure.
Unlike the original XLM, which employed both MLM and TLM objectives for pre-
training, XLM-R primarily utilizes the MLM objective. XLM-R is available in
two configurations - XLM-R Base (270M parameters) and XLM-R XL (550M
parameters). Both variants are trained on the clean CommonCrawl corpus
covering 100 languages. The tokenization method utilized by the model is
SentencePiece.

###  3.4 Decoder-Only Models

####  3.4.1 XGLM

XGLM Lin et al. (2022b), or Cross-lingual Generative Language Model, is a
family of multilingual models trained on the extensive CC100-XL dataset, this
is CommonCrawl snapshots including data from 2013 to 2020. The models vary in
size and configuration, with parameters ranging from 564 million to 7.5
billion, language coverage from 30 to 134, and number of layers from 24 to 48.
They utilize the SentencePiece tokenizer and focus on CLM during pre-training,
aiming to predict the next token given the previous ones. The largest XGLM
model, with 7.5 billion parameters, achieved state-of-the-art performance in
few-shot learning across multiple languages at the time of the release,
surpassing GPT-3 in tasks like commonsense reasoning and natural language
inference.

####  3.4.2 PALM

PaLM Chowdhery et al. (2023), or Pathways Language Model, is a multilingual
model trained on a diverse corpus consisting of content from Wikipedia, books,
web pages, social media, and source code. While exact details on the training
data are not provided, it is stated that PaLM is trained on 124 languages
using the SentencePiece tokenizer. Pre-training is done using Pathways, a
system enabling highly efficient pre-training across multiple TPUs. The
objective function used for pre-training is autoregressive language modeling
(CLM). The model comes in a few configurations: 8B parameters and 32
Transformer layers, 62B parameters and 64 Transformer layers, and 540B
parameters and 118 transformer layers. The largest variant, PaLM 540B,
achieves competitive results across various tasks, despite being trained on
approximately 22% of non-English data out of the total 780 billion training
tokens.

####  3.4.3 BLOOM

BLOOM Scao et al. (2023), standing for Bloom Language Model, is a significant
step towards open-sourcing LLMs. With a capacity of 176 billion parameters and
70 Transformer layers, BLOOM is an open-access LLM developed through
collaboration among hundreds of researchers. BLOOM is a decoder-only
Transformer language model trained on the ROOTS corpus, which consists of data
from hundreds of sources, including OSCAR, covering 46 natural languages and
13 programming languages. Similar to other models, BLOOM utilizes CLM during
training and uses BPE tokenization. Despite being open-source, BLOOM
demonstrates competitive performance across a wide range of benchmarks.
Moreover, its performance can be further enhanced through multitask prompted
fine-tuning. To support future research and applications using LLMs, the
models and code for BLOOM are publicly released, which promotes transparency
and collaboration in the field of NLP.

####  3.4.4 GPT-3

GPT-3 Brown et al. (2020), short for Generative Pre-trained Transformer 3, is
a widely known LLM developed by OpenAI. It is a decoder-only Transformer
architecture, consisting of 175 billion parameters and 96 Transformer layers.
GPT-3 is trained using the CLM task on a diverse dataset covering Common Crawl
(CC), Wikipedia, WebText2, and Books1 & 2\. These sources provide a wide range
of textual data, enabling GPT-3 to understand and generate human-like text
across various domains. The model covers over 95 languages, which is not
explicitly stated in the original report and estimated by users. For
tokenization, GPT-3 employs BPE. Additionally, GPT-3 incorporates advanced
techniques such as few-shot learning, enabling it to perform new tasks with
minimal instructions. Despite its impressive performance and wide adoption,
GPT-3 is not openly accessible for training or fine-tuning by external
researchers. However, its capabilities have captured significant interest,
serving as a benchmark for the development of new language models.

###  3.5 Encoder-Decoder Models

###  3.6 mBART

The next model is the multilingual Bidirectional Autoregressive Transformer
(mBART) - a multilingual sequence-to-sequence (Seq2Seq) denoising model Liu et
al. (2020). It stands out as one of the pioneering models employing both an
encoder and a decoder. The model is trained by applying the BART objectives
Lewis et al. (2020) on extensive monolingual corpora across a variety of
languages. This Seq2Seq model is pre-trained by denoising full texts in
various languages, and it is primarily intended for translation tasks. During
pre-training, the model tackles noise in the input by masking phrases and
introducing sentence order permutations, which is a variant of MLM.
Consequently, it learns to reconstruct the masked words based on context,
developing an understanding of word relationships, and to correctly order
sentences, thereby capturing relationships between sentences. A single
Transformer model with 12 encoder and 12 decoder layers consisting of 680
million parameters is used to execute training. The pre-training data is the
CC25 dataset extracted from the Common Crawl, which comprises of 25 languages
and is tokenized using the SentencePiece schema.

###  3.7 mT5

The Multilingual Text-to-text Transfer Transformer, mT5 Xue et al. (2021), is
the model that leverages the T5 architecture Raffel et al. (2020). Its
training data comes from Multilingual C4 (mC4) dataset, comprising data in 101
languages sampled from Common Crawl and tokenized using the SentencePiece
schema. An advantageous feature of the T5 architecture lies in its unified
"Seq2seq" format, where the model generates text conditioned on given text
inputs. Employing this "text-to-text" structure, mT5 uses a standard Encoder-
Decoder Transformer architecture. For pre-training, a variation of MLM,
involving the replacement of consecutive spans of input text with a masked
token and predicting them, is used. Notably, mT5 exists in several variants,
distinguished by the quantity of data used for training and the number of
parameters in the model. These variants include mT5-Small, mT5-Base,
mT5-Large, mT5-XL, and mT5-XXL. The number of parameters ranges from 300
million to 13 billion and number of layers - from 12 to 24 in both encoder and
decoder.

##  4 Curse of Multilinguality

This chapter addresses one of the biggest challenges faced by multilingual
LLMs - curse of multilinguality - and explores its potential solutions to
overcome it.

###  4.1 Motivation

The curse of multilinguality was introduced by Conneau et al. and refers to
the challenges and limitations that arise when developing multilingual LLMs
Conneau et al. (2020). As the number of languages increases, a transfer-
dilution trade-off occurs, diminishing the per-language capacity and
consequently impacting model performance. Initially, adding similar higher-
resource languages during pre-training can enhance the performance of low-
resource languages. However, beyond a certain point, the curse of
multilinguality starts to manifest, leading to decreased performance across
all languages Aharoni et al. (2019). This emphasises the complexity of
achieving universality in multilingual models and points at the need to
address key factors contributing to the curse of multilinguality.

####  4.1.1 Linguistic Diversity

Languages show significant diversity in grammar, syntax, vocabulary, and
cultural nuances, posing a significant challenge for multilingual model
development Conneau et al. (2020). Accommodating such diversity while
maintaining model efficiency and performance becomes increasingly challenging
with multiple languages, worsening the effects of the curse of
multilinguality.

####  4.1.2 Data Sparsity

The availability of training data varies widely across languages, with some
languages possessing a lot of resources while others being severely under-
resourced Joshi et al. (2020). The impact of data sparsity on training
efficient multilingual models is large, as highlighted by Conneau et al.
Conneau et al. (2020). Models may struggle to generalize well to languages
with limited training data, further complicating the curse of multilinguality.

####  4.1.3 Model Complexity and Scale

Multilingual models require increased complexity and scale to accommodate the
diverse linguistic characteristics of multiple languages Conneau et al.
(2020). Balancing model size and computational resources is crucial, as overly
complex models may become impractical for real-world deployment. Conneau et
al. emphasize the importance of addressing these challenges to mitigate the
curse of multilinguality.

###  4.2 Potential Solutions

Recent studies have pointed at reduced monolingual and cross-lingual
capabilities of models due to curse of multilinguality, particularly for low-
resource languages not having a lot of pre-training data Wu and Dredze (2020);
Lauscher et al. (2020); Artetxe et al. (2020). Various approaches have been
proposed to address this problem and improve model performance.

####  4.2.1 Modular Multilingual Architecture

Pfeiffer et al. propose X-MOD, a modular multilingual architecture that
combines shared and language-specific parameters as a way of uplifting curse
of multilinguality Pfeiffer et al. (2022). X-MOD initializes modular models
during pre-training, facilitating inexpensive expansion to new languages
afterwards.

##### Architecture:

X-MOD is an extension of the transformer-based architecture from mBERT and
XLM-R, incorporating language-specific modules at every transformer layer.
Each language has its own module, while attention and feed-forward components
are shared, facilitating efficient training and inference without
significantly increasing computational costs.

##### Pre-training:

X-MOD is pre-trained using MLM on combined monolingual corpora in multiple
languages. Efficient utilization of language-specific modules ensures
effective handling of linguistic diversity during pre-training.

##### Extending to New Languages:

The modular design allows X-MOD to be extended to new languages after pre-
training, with minimal impact on performance in pre-trained languages. This
adaptability is achieved through the learning of new embeddings and adapter
modules for the target language via MLM.

##### Fine-tuning on Downstream Tasks:

X-MOD can be fine-tuned for cross-lingual downstream tasks by selectively
updating shared weights on source language data while keeping modular
components frozen, ensuring efficient adaptation to target languages.

####  4.2.2 Alternative Approaches

Several alternative approaches have been proposed to extend multilingual and
monolingual LLMs to other languages and improve their cross-lingual
capabilities as a mitigation of curse of dimensionality.

##### Training New Embedding Layer:

Artetxe et al. propose training a new embedding layer with a corresponding
target-language tokenizer to extend monolingual models to new languages
Artetxe et al. (2020). This approach helps with language extension while
maintaining model stability.

##### Transliteration and Subword Mappings:

Transliteration-based approaches and subword mappings offer potential
solutions for incorporating additional languages into multilingual models
Muller et al. (2021); Vernikos and Popescu-Belis (2021). These methods
contribute to the expansion of multilingual model capabilities.

##### Adapter-based Approaches:

Adapter-based approaches, proposed by Pfeiffer et al., offer efficient
solutions for adapting multilingual LLMs to specific languages or extending
multilingual them to unseen languages and overcoming curse of dimensionality
Pfeiffer et al. (2020). While achieving significant performance gains, these
approaches may build upon sub-optimal parameter initializations.

##  5 Conclusion

In summary, multilingual LLMs stand out as robust tools in NLP, showcasing
proficiency across multiple languages and tasks. This paper has provided
insights into key models such as mBERT, XLM-R, mBART, mT5, XGLM, PALM, BLOOM,
and GPT-3, highlighting their underlying architectures and technical
characteristics. These models represent a significant advancement in NLP,
having deep multilingual understanding and potential for cross-lingual
applications. Their development plays a great role in advancing language
technology and preserving linguistic diversity across the world.

Despite their capabilities, multilingual LLMs face challenges such as the
curse of multilinguality, which constrains their performance across different
languages. However, various approaches have been developed and are being
pursued to mitigate these challenges. These efforts aim to improve the
effectiveness and adaptability of multilingual LLMs.

## References

  * Abadji et al. (2022) Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022.  Towards a cleaner document-oriented multilingual crawled corpus.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4344–4355, Marseille, France. European Language Resources Association.
  * Aharoni et al. (2019) Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019.  Massively multilingual neural machine translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3874–3884, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4623–4637, Online. Association for Computational Linguistics.
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016.  Layer normalization.  _arXiv preprint arXiv:1607.06450_.
  * Bebis and Georgiopoulos (1994) G. Bebis and M. Georgiopoulos. 1994.  Feed-forward neural networks.  _IEEE Potentials_ , 13(4):27–31.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, page 1877–1901. Curran Associates, Inc.
  * Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2023.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ , 24(240):1–113.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Multilingual bert.  https://github.com/google-research/bert/blob/master/multilingual.md.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dreyfus (1990) Stuart E Dreyfus. 1990.  Artificial neural networks, back propagation, and the kelley-bryson gradient procedure.  _Journal of guidance, control, and dynamics_ , 13(5):926–928.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020.  The pile: An 800gb dataset of diverse text for language modeling.
  * Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017.  Convolutional sequence to sequence learning.  In _International conference on machine learning_ , pages 1243–1252. PMLR.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Laurençon et al. (2023) Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite. 2023.  The bigscience roots corpus: A 1.6tb composite multilingual dataset.
  * Lauscher et al. (2020) Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020.  From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4483–4499, Online. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin et al. (2022a) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022a.  Few-shot learning with multilingual generative language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Lin et al. (2022b) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022b.  Few-shot learning with multilingual language models.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Muller et al. (2021) Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021.  When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 448–462, Online. Association for Computational Linguistics.
  * Pfeiffer et al. (2022) Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022.  Lifting the curse of multilinguality by pre-training modular transformers.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3479–3495, Seattle, United States. Association for Computational Linguistics.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020.  MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7654–7673, Online. Association for Computational Linguistics.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Scao et al. (2023) BigScience Workshop: Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Daniel McDuff, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2023.  Bloom: A 176b-parameter open-access multilingual language model.
  * Schuster and Nakajima (2012) Mike Schuster and Kaisuke Nakajima. 2012.  Japanese and korean voice search.  In _2012 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 5149–5152. IEEE.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Taylor (1953) Wilson L Taylor. 1953.  “cloze procedure”: A new tool for measuring readability.  _Journalism quarterly_ , 30(4):415–433.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Vernikos and Popescu-Belis (2021) Giorgos Vernikos and Andrei Popescu-Belis. 2021.  Subword mapping and anchoring across languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2633–2647, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Communications of the ACM_ , 57(10):78–85.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4003–4012, Marseille, France. European Language Resources Association.
  * Wu and Dredze (2020) Shijie Wu and Mark Dredze. 2020.  Are all languages created equal in multilingual BERT? In _Proceedings of the 5th Workshop on Representation Learning for NLP_ , pages 120–130, Online. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.


# Multimodal Neurodegenerative Disease Subtyping Explained by ChatGPT

Diego Machado Reyes, Hanqing Chao, Juergen Hahn, Li Shen , Pingkun Yan, and
for the Alzheimer’s Disease Neuroimaging Initiative Manuscript received on
xxx, 2023.D. Machado Reyes, H. Chao, J. Hahn, and P. Yan are with the
Department of Biomedical Engineering and the Center for Biotechnology and
Interdisciplinary Studies, Rensselaer Polytechnic Institute, Troy, NY, USA
12180;L. Shen is with the Department of Biostatistics, Epidemiology and
Informatics, Perelman School of Medicine at the University of Pennsylvania,
Philadelphia, USAThis work was supported in part by the NSF CAREER award
2046708, NSF IIS 1837964, and NIH grants U01 AG066833, U01 AG068057, and RF1
AG063481.Data used in preparation of this article were obtained from the
Alzheimer’s Disease Neuroimaging Initiative (ADNI) database
(adni.loni.usc.edu). As such, the investigators within the ADNI contributed to
the design and implementation of ADNI and/or provided data but did not
participate in analysis or writing of this report. A complete listing of ADNI
investigators can be found at: http://adni.loni.usc.edu/wp-
content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf

###### Abstract

Alzheimer’s disease (AD) is the most prevalent neurodegenerative disease; yet
its currently available treatments are limited to stopping disease
progression. Moreover, effectiveness of these treatments is not guaranteed due
to the heterogenetiy of the disease. Therefore, it is essential to be able to
identify the disease subtypes at a very early stage. Current data driven
approaches are able to classify the subtypes at later stages of AD or related
disorders, but struggle when predicting at the asymptomatic or prodromal
stage. Moreover, most existing models either lack explainability behind the
classification or only use a single modality for the assessment, limiting
scope of its analysis. Thus, we propose a multimodal framework that uses
early-stage indicators such as imaging, genetics and clinical assessments to
classify AD patients into subtypes at early stages. Similarly, we build
prompts and use large language models, such as ChatGPT, to interpret the
findings of our model. In our framework, we propose a tri-modal co-attention
mechanism (Tri-COAT) to explicitly learn the cross-modal feature associations.
Our proposed model outperforms baseline models and provides insight into key
cross-modal feature associations supported by known biological mechanisms.

###### Index Terms:

Disease subtyping, multimodal attention, early-stage biomarker
characterization, ChatGPT, Alzheimer’s disease.

##  I Introduction

Alzheimer’s disease (AD) is the most prevalent neurodegenerative disorder,
which affects over 6.5 million people in the US alone and its rate is expected
to keep increasing [1]. Current therapies for AD are limited to symptom
management and promising drugs that may slow AD progression; thus, it is vital
to diagnose neurodegenerative disease early. However, early diagnosis of AD is
very challenging as the characteristic symptoms used in clinical assessments
appear only at intermediate or later stages and can vary among patients due to
the disease heterogeneity. Therefore, it is essential to develop methods that
can characterize the factors driving disease development and identify the
specific subtype for individual patients at an early stage.

AD is traditionally diagnosed based on characteristic cognitive decline and
behavioral deficits that do not become apparent until intermediate to late
stages of the disease. Data driven approaches focus on classifying patients
into subtypes based on the disease progression and mild cognitive impairment
(MCI, a prodromal stage of AD) to AD conversion [2, 3, 4, 5]. Current methods
for AD and related disorders subtyping have mainly focused on using the
longitudinal data from clinical assessments for unsupervised learning [6, 7,
8]. For clustering approaches such as in [4, 2, 6, 8], the authors subtype AD
patients using single modalities at baseline, such as blood markers [4],
genomic data [6], or imaging derived traits from longitudinal measurements
[8]. These single modality approaches at baseline data have shown the
potential of early stage indicators for AD subtyping. However, most of them
miss to show how they relate biologically to AD development or use multiple
time points, which hinders the ability to diagnose patients at early stages
after AD onset.

Deep learning models [5] have effectively identified subgroups using
multimodal imaging data and correlation-based approaches which allow for
increased explainability of the model feature relationships. Nevertheless,
correlation-based approaches treat the clustering goal indirectly. Works on
related disorders have shown the relevance of multimodal approaches [9] using
non-negative matrix factorization and Gaussian mixture models, or employed
autoencoders and long short-term memory networks (LSTM) to learn deep
embeddings of the disease progression [10]. However, this requires
longitudinal data involving several time points. While several clustering and
deep learning-based approaches have high accuracy when having multiple time
points, the performance decreases significantly given only baseline data. This
is driven by the subtle expression of the symptoms tracked in the clinical
assessments limiting the scope of the models. Early stage indicators such as
imaging traits or genomic risk factors are rarely used, and are simply
aggregated to clinical assessments as additional indicators. Therefore, it is
essential to target early stage indicators such as genetics, imaging and
cognitive assessments.

Multimodal deep learning approaches can combine different modalities and
provide a much more informed picture of the disease drivers and aid in the
disease subtyping [11]. However, it is not a trivial task to identify the
relevant features across modalities and how to fuse them. The rest of this
section first reviews the related works on multimodal fusion and then provides
an overview of our proposed method.

Figure 1: The three main multimodal fusion strategies, early, intermediate and
late fusion, for deep learning methods.

###  I-A Multimodal Fusion

Multimodal fusion, while very promising, poses new challenges. There are
multiple ways to fuse the data and stages of encoding where to fuse. The
strategy effectiveness varies depending on data modalities and tasks. One of
the key factors is the similarity between the modalities. Highly heterogeneous
data such as imaging, genetics and clinical data, might not be immediately
fused. Their difference in type, signal to noise ratio, and dimensionality
makes it be very challenging to combine them without first projecting them
into a similar space. The relationship between input and output is just as
important to consider when fusing data. For example, clinical assessments
reflect the direct impacts of the disease, while genetic data describes the
building blocks of cells. The phenotype-related information available in
clinical assessments requires considerably less processing than what the
genomic data might require to find the connection with the disease.

As illustrated in Fig. 1, the existing multimodal fusion strategies can be
grouped into three main categories [12], including early, intermediate (also
called joint) and late fusion. It is essential to choose the right approach
based on the task at hand and the data used. Early stage fusion has shown very
promising results in recent vision-language models [13, 14], while late-fusion
strategies have traditionally been very effective in aggregating machine
learning models decisions. Early and late stage fusion strategies, while
effective for certain tasks, are not ideal for AD subtyping using multimodal
data. Early stage fusion struggles with dealing with highly heterogeneous and
differently biologically-related data such as genetics, imaging and clinical.
These require further encoding to first learn highly informative
representations for every modality and condense them into a similar latent
space. While late fusion strategies can be very effective at aggregating the
decisions based on each modality, they cannot learn the feature relationships
across modalities severely limiting the usefulness of the model. The
intermediate fusion approach tackles both challenges by first learning the
crucial patterns associated with each modality. On the next stage, it uses the
condensed patterns from each modality to learn the cross-modal relationships.
This enables a more harmonious fusion of the heterogeneous modalities.

Figure 2: Illustration of the proposed framework for AD subtyping, consisting
of two main sections: (a) single modality encoding and (b) tri-modal attention
and joint encoding.

###  I-B Overview and Contributions

Despite the potential advantages of multimodal approaches, several technical
challenges exist for effectively leveraging the key data from each modality.
The high heterogeneity of the data modalities for AD subtyping, and the
explicit learning of the cross-modal interactions need to be addressed.
Previous approaches in related fields have proposed dual co-attention
mechanisms to explicitly learn the cross-modal feature interaction and joint
data representations [15]. While they have shown very promising results, these
have yet to be explored for neurodegenerative disease subtyping and its
corresponding modalities. Moreover, in AD and related disorders subtyping, we
need to fuse three critical but highly heterogeneous modalities, i.e.,
imaging, genetics and clinical data. Therefore, in this paper, we propose a
tri-modal co-attention (Tri-COAT) framework that can explicitly learn the
cross-modal feature interactions towards the subtype classification task.
While deep learning models for disease assessment promise improved accuracy,
they remain limited in the interpretability of their results. This is a major
entry barrier for the inclusion of deep learning models in the medical field.

The advent of large language models has offered new areas of opportunity to
bridge the gap between deep learning models and their reach to the general
public. It is vital to develop models that physicians can use as tools that
provide clear information of what the model predictions mean beyond a
probability score. ChatGPT [16] allows for very accessible written outputs
from complex prompts. By crafting a prompt using the output information from
our proposed network, we can use ChatGPT to interpret the model AD assessment.

Our contributions in this paper are two fold: Application: our framework
incorporates features from three early stage biomarker modalities and provides
a cutting-edge approach to early neurodegenerative disease subtyping. Through
ChatGPT, the model results are interpreted into a highly accessible text
output. Technique: our new tri-modal co-attention can efficiently and
explicitly learn the interactions between highly heterogeneous modalities,
encode the information into a joint representation, and provide explainability
to the cross-modal interactions. Our proposed method, Tri-COAT, achieved
state-of-the-art performances on the landmark dataset Alzheimer’s Disease
Neuroimaging Initiative (ADNI), and provided key insights into the biological
pathways leading to neurodegenerative disease development.

##  II Method

Our proposed framework can be divided into two main parts. As seen in Fig. 2,
single modality encoders are first built using transformer modules to learn
feature representations for each modality. Second, the Tri-COAT mechanism
explicitly learns the critical cross-modal feature relationships and uses that
to weigh the feature representation. The jointly learned representation goes
through a multilayer perceptron (MLP) for disease subtype classification.

###  II-A Single-modality encoding

Three branches encode each modality individually. Each branch is comprised of
a transformer encoder with several transformer layers. Each branch learns
representations of a modality to later combine them into a joint
representation through the Tri-COAT mechanism.

Imaging modality. The imaging feature encoder branch uses MRI derived
quantitative traits as input. These quantiative traits are derived from T1
weighted MRI scans. The scans are first segmented based on the FreeSurfer
atlas for cross-sectional processing. Next, for each reconstructed regions of
interest (ROI) the cortical region, cortical volume, thickness average,
thickness standard deviation, and surface area are calculated. Further details
are described in section III-A. The imaging traits are then used to build
tokens, where each token represents an ROI in the brain and is comprised of
four imaging derived traits (Cortical Thickness Average, Cortical Thickness
Standard Deviation, Surface Area, Volume from Cortical Parcellation). Let
$\bm{X_{I}}\in\mathbb{R}^{M\times 4}$ represent the imaging input to the
proposed model, where $M=72$ and is the number of ROIs. After, the token
dimensions are expanded through a fully-connected layer to match the model
dimensions $k$. The imaging tokenization allows to build an initial
representation for each ROI rather than each trait, leading to a smaller
number of input features, and a more biologically informative and
interpretable input.

Genetic modality. The genotype branch has as input single nucleotide
polymorphism (SNP) data. After quality control and preprocessing of the
genotype data as described in section III-A, tokens for each SNP are built.
Each token is comprised by the allele dosage from the patient, the
corresponding odds ratio and rare allele frequency obtained from the most
recent AD GWAS study, and whether the SNP is within an intergenic region
(regulatory region) as a binary label. Then, the token dimensions are expanded
through a fully-connected layer to $k/2$. Let
$\bm{X_{SNP}}\in\mathbb{R}^{N\times k/2}$ represent the genotype input to the
model, where $N=70$ and is the number of SNPs filtered out (see section 3.1
for details). Moreover, based on the chromosome for which each SNP is located
in an additional embedding for each SNP can be built. By including the
chromosome embedding, location knowledge for each SNP can be incorporated.
Using an embedding layer, an embedding for each chromosome can be obtained
$\bm{X_{Chr}}\in\mathbb{R}^{N\times k/2}$. Finally, $\bm{X_{SNP}}$ and
$\bm{X_{Chr}}$ are concatenated to obtain the final genotype embedding
$\bm{X_{G}}\in\mathbb{R}^{N\times k}$. Similarly to the imaging data encoding,
the genetic tokenization allows to build more informative input structures to
the genetic encoder. By providing additional attributes for each SNP beyond
the patient mutation status, the model can learn richer patterns of
characteristics that relate each SNP.

Clinical modality. The clinical data is already very closely related to the
outcome of interest and contains only few features; therefore, no further
extensive tokenization is performed. As there is just one value per clinical
assessment, the tokens are directly built with one dimension. Let
$\bm{X_{C}}\in\mathbb{R}^{B\times 1}$ represent the clinical input to our
model, where $B=7$ and is the number of clinical features. Next, the token
dimensions are expanded through a fully-connected layer to match the model
dimensions $k$.

Single modality encoders. After the tokenization of each modality, they are
fed into independent transformer encoders with $L$ layers. The full process of
the $L$-th layer in our transformer encoder is formulated as:

$F^{\prime}_{l}=\mathrm{MHA}(\mathrm{LN}(F_{l-1}))+F_{l-1},F_{l}=\mathrm{FF}(\mathrm{LN}(F^{\prime}_{l}))+F^{\prime}_{l},$
(1)

where $\mathrm{LN}(\cdot)$ is the normalization layer [17], and
$\mathrm{MHA}(\cdot)$ is multihead self-attention [18].

###  II-B Tri-modal co-attention

After each transformer encoder has learned a new representation for each
modality, these are then used to learn the cross-modal feature relationships
to guide the co-attention process on the clinical branch. In other words, the
imaging and genomic features are employed to modulate the clinical learning
process by highlighting the key hidden features that share relationships
across modalities. The intuition behind the proposed approach is that as the
clinical data is most closely related to the disease phenotype, this branch
will carry most of the necessary information to classify the patients.
Nevertheless, the imaging and genomic data provide also valuable information.
The idea is analogous to the clinical data being the subject and verb in a
sentence while the imaging and genomic data are the adjectives and adverbs.
These two elements enrich the representation of the patient health status,
analogous to enriching a sentence for a fuller meaning.

Let $\bm{X_{Emb}}\in\mathbb{R}^{\\{M,N,B\\}\times k}$ represent the learned
representation of a given single-modality encoder. These become then query
matrices for the genetics $\bm{Q_{G}}$ and imaging $\bm{Q_{I}}$ data, and key
$\bm{K_{C}}$, value $\bm{V_{C}}$ matrices for the clinical data. Following an
attention mechanism structure, the co-attention between two modalities is
computed as:

$\mathrm{CoAttn}(\\{G,I\\},C)=\mathrm{softmax}\left(\frac{{Q_{\\{G,I\\}}}K_{C}^{T}}{\sqrt{d_{k}}}\right)V_{C}^{T}.$
(2)

Next, the resulting co-attention filtered value matrices are then concatenated
to obtain a final joint representation. This joint representation is then
flattened and used to classify the patients into the clusters through an MLP.

TABLE I: Subject data distribution reported as mean $\pm$ standard deviation for MMSE and age, and counts per category for number of participants and gender. | Slow | Intermediate | Fast
---|---|---|---
Participants | 177 | 302 | 15
MMSE (BL) | 27.35 $\pm$ 2.51 | 27.66 $\pm$ 1.86 | 24.93 $\pm$ 3.55
MMSE (m24) | 28.15$\pm$ 2.15 | 23.86 $\pm$ 3.68 | 15.9 $\pm$ 4.84
Age | 73.26 $\pm$ 7.82 | 72.44 $\pm$ 7.55 | 71.22 $\pm$ 3.922
Gender | M: 102 F: 75 | M: 185 F: 117 | M: 9 F: 6

(a)

(b)

Figure 3: AD subtype clusters based on the decrease of MMSE at each visit. (a)
each line represents the average score across patients for each cluster and
the shadow represents one standard deviation. (b) individual lines per patient
are plotted.

##  III Materials and Experiments

###  III-A Dataset

The Alzheimer’s Disease Neuroimaging Initiative (ADNI) database is a landmark
dataset for the advancement of our understanding of Alzheimer’s disease. ADNI
is comprised by a wide range of data modalities including MRI and PET images,
genetics, cognitive tests, CSF and blood biomarkers (for up-to-date
information, see www.adni-info.org). Longitudinal data for all subjects was
selected as up to two years of progression after disease onset due to very
high missingness rate (percentage of data points missing across patients for a
given time point) present for time points posterior to the two years. Subjects
were clustered using k-means in 3 main groups based on their Mini-Mental State
Examination (MMSE) scores as can be seen in Fig 3. These clusters match the
cognitive decline rate of patients over time. The MMSE score at each visit
(basline, 6 months, 12 months, and 24 months) was used to determine the
cognitive decline for each patient. As each patient may have a different
starting level at baseline, the baseline measurement is subtracted from each
of the following time points; thus all patients start at 0. Then, using
k-means clustering, using $k=3$, slow, intermediate and fast cognitive decline
groups are defined. As seen in Table I the raw average MMSE score at baseline
is comparable across all groups with a steep decrease for the fast and
intermediate groups at 24 months. The slow or otherwise stable group MMSE
score at 24 months is comparable to the one at the initial stage. On the other
hand, all 3 groups are age matched. Similarly, gender distributions across
groups is maintained with male subjects representing approximately 60% of the
subjects for each group.

Data preprocessing: The data for the imaging and genotype modalities was
processed previously to the tokenization process following best practices for
the corresponding modality as described below.

  * •

Imaging: The FreeSurfer image analysis suite
(http://surfer.nmr.mgh.harvard.edu/) is used to conduct cortical
reconstruction and volumetric segmentation. T1 weighted MRI scans are
segmented based on the FreeSurfer atlas for cross-sectional processing,
enabling group comparison at a specific time point [19]. For each
reconstructed cortical region, cortical volume, thickness average, thickness
standard deviation, and surface area measurements are labeled by the 2010
Desikan-Killany atlas. The UCSF ADNI team conducted this process [20], and
more information can be found at http://adni.loni.usc.edu.

  * •

Genotype: The genotype variants were filtered using the intersection between
the List of AD Loci and Genes with Genetic Evidence Compiled by ADSP Gene
Verification Committee and the most recent genome wide association study
(GWAS) on AD [21]. The odds Ratios, rare allele frequency and intergenic
region binary trait were obtained from the most recent GWAS study with
accession number (GCST90027158), accessed through the GWAS catalog [22].
Furthermore, the genotype variants were processed for sample and variant
quality controls using PLINK1.9 [23].

  * •

Clinical: The clinical assessment features correspond to 7 different cognitive
metrics available through ADNI. These are Logical Memory - Delayed Recall
(LDELTOTAL), Digit Symbol Substitution (DIGITSCOR), Trail Making Test B
(TRABSCOR) and Rey Auditory Verbal Learning Test (RAVLT) scores: immediate,
learning, forgetting and percent forgetting. Values for imaging and clinical
modalities were normalized in each train set before they were used as inputs
to the network.

###  III-B Experimental design

All models were trained and evaluated in 10-fold stratified cross-testing,
5-fold stratified cross-validation for hyperparameter tuning. First the full
dataset was split into 10 folds where 1 fold is left out for testing and the
rest for training, this training set is then split into 5 folds to perform
hyperparameter tuning. Through this framework, all data leakage is prevented.
The best hyperparameters were determined on the validation set at each
experimental run by selecting the best performing model. Predictions were
evaluated using the area under the receiver operating curve (AUROC). A one-vs-
one strategy is employed where the average AUROC of all possible pairwise
combinations of classes is computed for a balanced metric. This is implemented
using Sci-Kit Learn API [24], which implements the method described in [25].
The model was compared against the stage-wise deep learning intermediate
fusion model introduced in [26] and several well-established traditional ML
models - random forest (RF), support vector machine (SVM) with radial-basis
function (RBF) kernel. Similarly, each of the branches of the model was used
as comparison using a series of transformer encoder layers and MLP head for
classification.

Tri-COAT consist of four transformer layers for each of the single modality
encoders, with four attention heads per transformer layer. The tri-model co-
attention process is done in a single-head attention mechanism. The
classifying MLP has one hidden layer with 256 units. Embedding dimension of
$k=256$ was used for all modalities. The model dimensions for the single-
modality encoders were kept at 256 and all throughout as this combination
achieved the best results on the validation set. The final mlp had the
concatenated class tokens resulting from the tri-modal co-attention module and
computed the output logits for each one of the three possible classes. Adam
was used to optimize both Tri-COAT and the stage-wise mlp model using learning
rates of 0.0001 for Tri-COAT and 0.0001 for the stage-wise fusion benchmarking
model. All deep learning models were trained using cross-entropy loss. All
deep learning models were implemented using PyTorch, while the RF and SVM
models were implemented using scikit-learn. All deep learning models were
trained for 100 epochs and the best checkpoint, meaning epoch with the highest
AUROC on the validation set, was selected for model evaluation. The stage-wise
deep learning fusion model had dimensions of 64 units for the single-modality
layers, 32 for the second-stage and 16 for the final stage. The model
dimensions were selected following the described hyperparamethers in [26]. The
SVM used an rbf kernel, regularization parameter C of 1. The random forest
used gini as its criterion for leaf splitting, 100 trees and no maximum depth.

(a) Clinical and Imaging (b) Clinical and Genotype (c) Imaging and Genotype

Figure 4: AD key biomarker associations form learned co-attention. TABLE II: Mean AUROC $\pm$ SD of 10-fold cross-testing results. The proposed model significantly outperformed all the baseline models. The statistical significance was evaluated by paired $t$-test with $\alpha=0.005$, except for the entry where $\alpha=0.05$.  Method |  Full |  Imaging |  Genetics |  Clinical
---|---|---|---|---
SVM |  0.705 $\pm$ 0.036 |  0.669 $\pm$ 0.060 |  0.525 $\pm$ 0.034 |  0.639 $\pm$ 0.078
RF |  0.684 $\pm$ 0.048 |  0.677 $\pm$ 0.052 |  0.505 $\pm$ 0.031 |  0.659 $\pm$ 0.087
Stage-wise fusion[26] |  0.641 $\pm$ 0.017 |  0.557 $\pm$ 0.096 |  0.562 $\pm$ 0.078 |  0.655 $\pm$ 0.057
Tri-COAT |  0.734 $\pm$ 0.076 |  0.648 $\pm$ 0.056 |  0.539 $\pm$ 0.084 |  0.697 $\pm$ 0.063
TABLE III: Mean AUROC $\pm$ SD of 10-fold cross-testing results. The proposed model significantly outperformed all the baseline models. The statistical significance was evaluated by paired $t$-test with $\alpha=0.005$, except for the entry where $\alpha=0.05$.  Method |  AUROC
---|---
Early |  0.571 $\pm$ 0.053
Late |  0.604 $\pm$ 0.048
Tri-COAT |  0.734 $\pm$ 0.076

##  IV Results and Discussion

###  IV-A Clustering, label definition

Following the literature, the number of clusters was set to three main groups.
The MMSE score was used as an indicator of mental decline. Based on the speed
of the progression of the mental decline over a period of two years, three
groups were defined, i.e., fast, intermediate and slow progressing subtypes.
K-means clustering was used to assign each subject to one of the groups.
Through this process labels were defined for all subjects. Only baseline data
was used as input to Tri-COAT and all competing models. Based on the baseline
data Tri-COAT was able to effectively classify the subjects into their
corresponding subtypes.

###  IV-B AD subtype classification

As seen in Table II, Tri-COAT outperformed the single modality ablations and
baseline models, achieving an average AUROC of 0.733 $\pm$ 0.070 across all
test sets in the 10-fold cross-testing framework. For the single-modaltiy
ablation studies, each modality was used independently to classify the AD
subtype. For Tri-COAT, a single-modality transformer encoder backbone and MLP
head were used. For the stage-wise fusion model, it was adapted to MLPs using
the same number of hidden layers as the first plus last stage of the
multimodal version. For SVM and RF, no variations were required. Each modality
was evaluated using the same 10-fold cross testing - 5 fold-cross validation
hyperparameter tuning framework. Moreover, the single modality ablation models
outperformed their baseline counterparts for the clinical and genetics
modalities. In contrast to this the baselines achieved better performances for
the imaging derived traits. For the three modalities, clinical achieves the
best classification AUROC, followed by imaging and genetics. This is expected
as biologically, the same order follows for the closest relation between the
observed phenotype and the mechanisms behind it. Clinical (cognitive)
assessments are the closest to the MMSE metric, followed by imaging (changes
in the brain morphology) which is directly related to the observed phenotype
and genetics being the farthest apart from the expressed symptoms. Both
comparative models and Tri-COAT achieve higher performances in their
multimodal configuration compared to single modalities, agreeing with previous
literature regarding multimodal approaches for classification of AD and
related disorders.

Furthermore, as seen in Table III, Tri-COAT outperformed variations of itself
using alternative fusion strategies. The early fusion model considerably
underperforms achieving an AUROC of 0.571 $\pm$ 0.053 because of its limited
capabilities to simultaneous encode highly heterogeneous data with distinct
biological level relationships to the outcome. Similarly, the late fusion
model underperforms as it is limited to joining the predictions from each
branch and cannot learn the relationships between the different modalities in
the latent space.

Figure 5: Tri-COAT interpretability through ChatGPT. Prompts were built based
on Tri-COAT’s prediction and integrated gradients feature attributions for
each patient. Example portions of the prompts are showing in the blue text
bubbles and ChatGPT answers are shown in the green text bubbles.

###  IV-C Biomarker Associations Learned by Co-attention

One of the key advantages of Tri-COAT compared to the baseline models and
traditional deep learning approaches is the ability to learn insights into the
cross-modal feature associations. In order to explore the learned
relationships, the model with highest test AUROC from the evaluation framework
was selected for attention visualization in which the learned attention scores
were averaged across all test subjects. Chord plots were drawn using the
circlize R library [27] to visualize the cross-modal attention. As seen in
Figure 4, Tri-COAT identified key associations between the Trails making test
B score (TRABSCORE) in clinical-imaging and clinical-genetics. This score
tests for cognitive ability of the patient for working memory and secondarily
task-switching ability [28]. Clinical literature shows strong correlation
between gyri structures - temporal gyrus and Parahippocampal gyrus
(LTransTemp, LPH) and TRABSCORE [29]. Similarly, for the clinical-genotype
association, TRABSCORE was found to be associated with the CD2AP gene which
has been clinically identified as a driver for the AD hallmark -
neurofibrillary tangles (NFT) in the temporal gyrus region [30]. This is a
very exciting finding for our network as it establishes a putative
relationship between genetics (CD2AP gene), brain ROIs (temporal gyrus) and
clinical symptoms (TRABSCORE).

###  IV-D Tri-COAT assessments explained through ChatGPT and Bard

In this section, we investigate using large language models, namely ChatGPT
(GPT-3.5 March 23, 2023 version) [16] and Bard(Accessed Jan 30th 2024) [31],
to interpret the results of multimodal deep learning models. In order to
enhance the interpretability of Tri-COAT, descriptions of each patient case
(“prompts”) are built using the top-10 features with their corresponding
standard deviation and ChatGPT is employed to interpret said prompts. First,
the attribution for each input feature were calculated using integrated
gradients [32]. Next, based on the absolute contribution, the most salient
features for each patient were determined. Finally, a text prompt for each
patient was built using the most salient features. The prompt describes the
quantitative attribution of the top-10 salient features and how many standard
deviations the patient’s measurements are from the whole sample distribution.
ChatGPT provided very informative answers explaining the biological relevance
of each salient feature towards AD development. A short example can be seen in
5.

To provide more detail, we built a prompt for each patient based on the top-
weighted features obtained from integrated gradients. The prompts were
structured as a question and a case describing the patient biomarker
measurements. The question asks to explain the model prediction and the case
contains the selected features’ attribution and corresponding standard
deviation. The attribution score obtained for each feature through integrated
gradients describes how relevant each feature is toward classifying each
patient. The input features are ranked using the attribution scores and the
top 10 features are selected for each subject. In order to describe how each
subject differs from the rest of the group based on the analyzed biomarkers,
the standard deviation for every feature is calculated. A template was built
to be filled out with the patient specific data. The template was iteratively
refined to improve the LLM output and its readability. The template followed a
11 short descriptions structure, the first being the patient risk score
assessment plus one description per salient feature. For the patient risk
assessment model, it started by introducing the patient risk and the
predictive model as seen in Fig. 5. Next, 10 short descriptions of the top-10
attributions, including slots for the salient feature name, the corresponding
standard deviation of the patient measurement and if this represents a higher
or lower value compared to the population.

The reports generated by ChatGPT and Bard includes clear descriptions for each
feature of interest and their relationship toward AD development and how
jointly they impact the deep learning model’s assessment. This approach
provides not only high explainability to the model assessment but also
provides key insights into cross-modal feature associations supported by known
biological mechanisms. For example, ChatGPT describes the biological functions
of the right insula and its connection with AD. Next, it explains that the
significantly reduced volume in this brain structure suggests a loss of
function and a rapidly progressing AD [33]. Similarly, it is exciting to see
that even when the measured trait deviates from the group in the opposite
direction as expected for an AD patient, ChatGPT identifies the value as
unexpected and provides an explanation for it. For example, for the abnormally
higher cortical thickness of the right cuneus, it explains that this could
indicate a compensatory mechanism [34]. We saw similar responses from both
ChatGPT and Bard, which results matched findings known in clinical literature.
It is however important to note that LLMs have been found to hallucinate [35].
While there are promising approaches to mitigate hallucinations; this is a key
limitation of current LLMs towards implementation in clinical settings.

##  V Conclusions

AD is the most prevalent neurodegenerative disease, and all current treatments
are limited to slowing disease progression; thus making early diagnosis
essential. Furthermore, there are multiple subtypes with different rates of
cognitive decline. In order to move closer to personalized medicine, it is
essential to have a better understanding of the heterogeneity surrounding the
disease development. However, early subtyping is a very challenging task. Our
proposed model was able to effectively classify AD patients into three main
subtypes using prodromal factors measured at baseline. Moreover, the model was
able to identify multiple putative cross-modal biomarker networks.
Nevertheless, the generalizability of the learned features on other datasets
remains to be tested. We demonstrated that multimodal deep learning models can
achieve increased interpretability through large language models and
attribution methods; which, in turn, can lead in future work to personalized
medicine applications of deep learning models in clinical settings. Similarly,
in future work we propose to extend this framework to other heterogeneous
neurodegenerative diseases such as Parkinson’s disease.

##  VI Acknowledgements

Data collection and sharing for this project was funded by the Alzheimer’s
Disease Neuroimaging Initiative (ADNI) (National Institutes of Health Grant
U01 AG024904) and DOD ADNI (Department of Defense award number
W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the
National Institute of Biomedical Imaging and Bioengineering, and through
generous contributions from the following: AbbVie, Alzheimer’s Association;
Alzheimer’s Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.;
Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.;
Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La
Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE
Healthcare; IXICO Ltd.;Janssen Alzheimer Immunotherapy Research & Development,
LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity;
Lundbeck; Merck & Co., Inc.;Meso Scale Diagnostics, LLC.; NeuroRx Research;
Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.;
Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition
Therapeutics. The Canadian Institutes of Health Research is providing funds to
support ADNI clinical sites in Canada. Private sector contributions are
facilitated by the Foundation for the National Institutes of Health
(www.fnih.org). The grantee organization is the Northern California Institute
for Research and Education, and the study is coordinated by the Alzheimer’s
Therapeutic Research Institute at the University of Southern California. ADNI
data are disseminated by the Laboratory for Neuro Imaging at the University of
Southern California

## References

  * [1] A. Association, “2022 Alzheimer’s disease facts and figures,” _Alzheimer’s & Dementia_, vol. 18, no. 4, pp. 700–789, 2022.
  * [2] A. Mitelpunkt _et al._ , “Novel Alzheimer’s disease subtypes identified using a data and knowledge driven strategy,” _Scientific Reports_ , vol. 10, no. 1, p. 1327, Jan. 2020, number: 1 Publisher: Nature Publishing Group.
  * [3] A. Badhwar _et al._ , “A multiomics approach to heterogeneity in Alzheimer’s disease: focused review and roadmap,” _Brain_ , vol. 143, no. 5, pp. 1315–1331, May 2020.
  * [4] G. Martí-Juan _et al._ , “Revealing heterogeneity of brain imaging phenotypes in Alzheimer’s disease based on unsupervised clustering of blood marker profiles,” _PLOS ONE_ , vol. 14, no. 3, p. e0211121, Mar. 2019, publisher: Public Library of Science.
  * [5] Y. Feng, _et al._ , “Deep multiview learning to identify imaging-driven subtypes in mild cognitive impairment,” _BMC Bioinformatics_ , vol. 23, no. 3, p. 402, Sep. 2022.
  * [6] M. A. Emon _et al._ , “Clustering of Alzheimer’s and Parkinson’s disease based on genetic burden of shared molecular mechanisms,” _Scientific Reports_ , vol. 10, no. 1, p. 19097, Nov. 2020.
  * [7] J. Wen, _et al._ , “Multi-scale semi-supervised clustering of brain images: Deriving disease subtypes,” _Medical Image Analysis_ , vol. 75, p. 102304, Jan. 2022.
  * [8] K. Poulakis _et al._ , “Multi-cohort and longitudinal Bayesian clustering study of stage and subtype in Alzheimer’s disease,” _Nature Communications_ , vol. 13, no. 1, p. 4566, aug 2022, number: 1 Publisher: Nature Publishing Group.
  * [9] A. Dadu _et al._ , “Identification and prediction of Parkinson’s disease subtypes and progression using machine learning in two cohorts,” _npj Parkinson’s Disease_ , vol. 8, no. 1, pp. 1–12, Dec. 2022, number: 1 Publisher: Nature Publishing Group.
  * [10] C. Su, _et al._ , “Integrative analyses of multimodal clinical, neuroimaging, genetic, and transcriptomic data identify subtypes and potential treatments for heterogeneous Parkinson’s disease progression,” Oct. 2022, pages: 2021.07.18.21260731.
  * [11] N. D. Nguyen and D. Wang, “Multiview learning for understanding functional multiomics,” _PLOS Computational Biology_ , vol. 16, no. 4, p. e1007677, Apr. 2020, publisher: Public Library of Science.
  * [12] S.-C. Huang, A. Pareek, S. Seyyedi, I. Banerjee, and M. P. Lungren, “Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines,” _NPJ Digital Medicine_ , vol. 3, p. 136, Oct. 2020. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7567861/
  * [13] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan, “Flamingo: a Visual Language Model for Few-Shot Learning,” Nov. 2022, arXiv:2204.14198 [cs]. [Online]. Available: http://arxiv.org/abs/2204.14198
  * [14] H. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F. Chang, Y. Cui, and B. Gong, “VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text,” Dec. 2021, arXiv:2104.11178 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2104.11178
  * [15] R. J. Chen, M. Y. Lu, W.-H. Weng, T. Y. Chen, D. F. K. Williamson, T. Manz, M. Shady, and F. Mahmood, “Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 4015–4025.
  * [16] OpenAI, “GPT-4 Technical Report,” Mar. 2023, arXiv:2303.08774 [cs]. [Online]. Available: http://arxiv.org/abs/2303.08774
  * [17] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer Normalization,” _arXiv:1607.06450 [cs, stat]_ , Jul. 2016, arXiv: 1607.06450.
  * [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is All you Need,” in _Advances in Neural Information Processing Systems_ , vol. 30. Curran Associates, Inc., 2017\.
  * [19] B. Fischl and A. M. Dale, “Measuring the thickness of the human cerebral cortex from magnetic resonance images,” _Proceedings of the National Academy of Sciences_ , vol. 97, no. 20, pp. 11 050–11 055, 2000.
  * [20] M. Hartig, D. Truran-Sacrey, S. Raptentsetsang, A. Simonson, A. Mezher, N. Schuff, M. Weiner _et al._ , “Ucsf freesurfer methods,” _ADNI Alzheimers Disease Neuroimaging Initiative: San Francisco, CA, USA_ , 2014.
  * [21] C. Bellenguez _et al._ , “New insights into the genetic etiology of Alzheimer’s disease and related dementias,” _Nature Genetics_ , vol. 54, no. 4, pp. 412–436, Apr. 2022, number: 4 Publisher: Nature Publishing Group.
  * [22] A. Buniello and J. MacArthur, “GWAS Catalog,” 2019, vol. 47 (Database issue): D1005-D1012.
  * [23] S. Purcell _et al._ , “PLINK: a tool set for whole-genome association and population-based linkage analyses,” _American Journal of Human Genetics_ , vol. 81, no. 3, pp. 559–575, Sep. 2007.
  * [24] F. Pedregosa and others, “Scikit-learn: Machine Learning in Python,” _Journal of Machine Learning Research_ , vol. 12, pp. 2825–2830, 2011.
  * [25] D. J. Hand and R. J. Till, “A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems,” _Machine Learning_ , vol. 45, no. 2, pp. 171–186, Nov. 2001.
  * [26] T. Zhou, K.-H. Thung, X. Zhu, and D. Shen, “Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis,” _Human Brain Mapping_ , vol. 40, no. 3, pp. 1001–1016, Feb. 2019.
  * [27] Z. Gu and others, “circlize implements and enhances circular visualization in R,” _Bioinformatics_ , vol. 30, no. 19, pp. 2811–2812, Oct. 2014.
  * [28] S. Terada _et al._ , “Trail Making Test B and brain perfusion imaging in mild cognitive impairment and mild Alzheimer’s disease,” _Psychiatry Research: Neuroimaging_ , vol. 213, no. 3, pp. 249–255, Sep. 2013.
  * [29] J. A. Matías-Guiu _et al._ , “Neural Basis of Cognitive Assessment in Alzheimer Disease, Amnestic Mild Cognitive Impairment, and Subjective Memory Complaints,” _The American Journal of Geriatric Psychiatry_ , vol. 25, no. 7, pp. 730–740, Jul. 2017.
  * [30] J. Camacho _et al._ , “Association of CD2AP neuronal deposits with Braak neurofibrillary stage in Alzheimer’s disease,” _Brain Pathology_ , vol. 32, no. 1, p. e13016, Sep. 2021.
  * [31] Google Bard. Alzheimer’s disease risk factors and their biological implications. [Online]. Available: https://bard.google.com
  * [32] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic Attribution for Deep Networks,” Jun. 2017, arXiv:1703.01365 [cs]. [Online]. Available: http://arxiv.org/abs/1703.01365
  * [33] L.-Y. Zhu, L. Shi, Y. Luo, J. Leung, and T. Kwok, “Brain MRI biomarkers to predict cognitive decline in older people with alzheimer’s disease,” vol. 88, no. 2, pp. 763–769, publisher: IOS Press. [Online]. Available: https://content.iospress.com/articles/journal-of-alzheimers-disease/jad215189
  * [34] C. Pettigrew, A. Soldan, Y. Zhu, M.-C. Wang, T. Brown, M. Miller, and M. Albert, “Cognitive reserve and cortical thickness in preclinical alzheimer’s disease,” vol. 11, no. 2, pp. 357–367. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5743433/
  * [35] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.” [Online]. Available: http://arxiv.org/abs/2311.05232