# Histo-Genomic Knowledge Distillation For Cancer Prognosis From
Histopathology Whole Slide Images

Zhikang Wang1, Yumeng Zhang2 , Yingxue Xu3, Seiya Imoto4, Hao Chen3, Jiangning
Song1
1Monash University, 2Shanghai Jiao Tong University, 3HKUST, 4University of
Tokyo
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author.

###### Abstract

Histo-genomic multi-modal methods have recently emerged as a powerful
paradigm, demonstrating significant potential for improving cancer prognosis.
However, genome sequencing, unlike histopathology imaging, is still not widely
accessible in underdeveloped regions, limiting the application of these multi-
modal approaches in clinical settings. To address this, we propose a novel
Genome-informed Hyper-Attention Network, termed G-HANet, which is capable of
effectively distilling the histo-genomic knowledge during training to elevate
uni-modal whole slide image (WSI)-based inference for the first time. Compared
with traditional knowledge distillation methods (i.e., teacher-student
architecture) in other tasks, our end-to-end model is superior in terms of
training efficiency and learning cross-modal interactions. Specifically, the
network comprises the cross-modal associating branch (CAB) and hyper-attention
survival branch (HSB). Through the genomic data reconstruction from WSIs, CAB
effectively distills the associations between functional genotypes and
morphological phenotypes and offers insights into the gene expression profiles
in the feature space. Subsequently, HSB leverages the distilled histo-genomic
associations as well as the generated morphology-based weights to achieve the
hyper-attention modeling of the patients from both histopathology and genomic
perspectives to improve cancer prognosis. Extensive experiments are conducted
on five TCGA benchmarking datasets and the results demonstrate that G-HANet
significantly outperforms the state-of-the-art WSI-based methods and achieves
competitive performance with genome-based and multi-modal methods. G-HANet is
expected to be explored as a useful tool by the research community to address
the current bottleneck of insufficient histo-genomic data pairing in the
context of cancer prognosis and precision oncology. The Pytorch implemented
code is available at https://github.com/ZacharyWang-007/G-HANet.

##  1 Introduction

Survival analysis delves into the duration until a specific event happens and
has been extensively applied in biology, engineering, economics, and medicine
[23]. Particularly in the field of cancer research [10, 9], experts can
estimate survival status, compare survival rates among different patient
groups, and identify critical prognostic factors. Given that cancer stands as
a leading cause of human mortality worldwide [13], cancer survival prediction
has immense clinical value, thereby drawing increasing research attention.

In current clinical practice, histopathology whole slide image (WSI) and
genomic data are widely utilized for cancer prognosis. WSIs provide
qualitative insights into the morphology and spatial structure of tumor
tissues, as well as their interactions with the adjacent microenvironment;
whereas genomic data delivers a quantitative molecular overview from the
microcosmic perspective. Both two varieties of data have their unique
strengths and constraints. In recent years, a large number of WSI-based [27,
5, 36] and genome-based [2, 7, 35, 39] uni-modal algorithms have been proposed
for cancer prognosis. Moreover, many histo-genomic multi-modal algorithms [43,
4, 26, 6, 41] have demonstrated superior capabilities in precision medicine,
creating a more comprehensive and personalized representation that aligns
closely with the patient’s health status. Nevertheless, genome sequencing
still remains costly for most cancer patients, especially in underdeveloped
areas. Hence, the broader implementation of multi-modal algorithms in clinical
practice and research could be stalled, preventing their potential benefits
from being realized on a larger scale and across diverse demographic groups.

Maximizing the utility of existing histo-genomic pairs to enhance WSI-based
network performance is a promising strategy to address the discussed data
bottleneck. However, this issue remains unexplored in current computational
pathology. Given the success of knowledge distillation (KD) [16, 18, 21, 15,
24, 32] in many tasks, a potential solution to address the above issue is to
transfer the knowledge from multi-modal models to uni-modal ones.
Specifically, we need a powerful multi-modal network as a teacher and a WSI-
based uni-modal network as the student. However, most response-based knowledge
distillation (KD) methods [16, 18, 21] are primarily designed for conventional
classification tasks and may not be effective in survival prognosis due to the
distinct optimization target. In terms of representation-based methods [15,
24, 32], aligning the feature distributions between the teacher and student
networks may fall short in comprehensively transferring the multi-modal
knowledge in terms of interactions and functions. Additionally, training two
networks, i.e., multi-modal and uni-modal networks, can be laborious and
computationally intensive. Consequently, the current KD strategies designed
for most computer vision tasks are not the optimal solution to this issue.

To address the challenge highlighted, we propose the genome-informed hyper-
attention network (G-HANet). G-HANet notably elevates the performance in uni-
modal WSI-based inference by distilling histo-genomic knowledge from the
paired data in the training phase. Specifically, the network comprises the
cross-modal associating branch (CAB) and hyper-attention survival branch
(HSB). Through the functional genomic data reconstruction from WSIs, CAB can
effectively distill the associations between functional genotypes and
morphological phenotypes and provide insights into the gene expression
profiles in the feature space. To reveal this association, the multi-head
cross-attention (MHCA) [34] module is incorporated into the reconstruction
phase to offer the pairwise histo-genomic score. Recognizing the complementary
characteristics of genome data and histopathology slides during survival
analysis, HSB subsequently utilizes both the derived histo-genomic
associations and generated morphology-based weights to achieve the hyper-
attention modeling of the patients using WSIs from both histopathology and
genomic perspectives. The extracted features from HSB will be utilized for the
final survival prognosis. Given that genomic data is exclusively utilized to
optimize CAB during training, merely WSIs are required for model inference,
hence, our approach offers an efficient solution to the challenge of lacking
genomic data.

To evaluate the proposed G-HANet, extensive experiments are conducted on five
benchmarking TCGA datasets. Our proposed G-HANet significantly outperforms
current WSI-based methods and achieves competitive performance with other
genome-based and multi-modal methods. Moreover, our study for the first time
considers the unavailability of genome data for cancer prognosis in clinical
practice and aims to enhance the WSI-based prognosis by leveraging knowledge
distilled from training data pairs, thereby holding substantial importance for
the computational pathology community.

Figure 1: Overview of the proposed genome-informed hyper-attention network
(G-HANet). The WSIs and genomic data are formulated as bags of features
$F^{p}$ and categorized according to the functions as $X^{g}$. In G-HANet, the
Cross-modal Associating Branch distills the histo-genomic knowledge through
genome reconstruction from WSIs. Afterward, with the assistance of the
distilled knowledge, HSB achieves the hyper-attention modeling of the WSIs
from both histopathology and genomic perspectives and conducts survival
prediction. The genome data processing and reconstruction (framed by the red
boxes) are only required in the training phase, thereby enabling the WSI-based
inference.

##  2 Related Work

###  2.1 Uni-modal for Cancer Survival Prognosis

Genome-based Methods. Genomic data quantifies the molecular profiles of
patients from the microcosmic perspective in an explicit manner. Xu et al.
[40] utilized the support vector machine-based recursive feature elimination
(SVM-RFE) approach to discover the critical gene signatures and conduct
prognosis prediction for breast cancer. Chaudhary et al. [2] specifically
investigated the deep learning-based model to differentiate survival
subpopulations of liver cancer patients in six cohorts. This is the first
deep-learning study to identify multi-omics features linked to the
differential survival of patients with hepatocellular carcinoma. Ching et al.
[7] developed the Cox-nnet to predict patient prognosis from transcriptomics
data and found functional biological insights. Wang et al. [35] and Xing et
al. [39] employed the graph neural network to model the genomic data for
cancer survival prediction based on sample similarity and co-expression gene
matrix, respectively.

WSI-based Methods. WSIs qualitatively offer insights into the tumor
characteristics and its adjacent microenvironment. Given their gigapixel
resolutions, existing methods [27, 5, 30, 33, 45, 36] generally model them
through the weakly supervised multiple instance learning (MIL) strategy [31].
Li et al. [27] explored the significant role of topological features of WSIs
in survival analysis by modeling each WSI as a graph. Afterward, the graph
convolutional neural network with attention learning was adopted to proceed
with the survival prediction. Chen et al. [5] proposed the context-aware and
spatially-resolved patch-based graph convolutional network, termed Patch-GCN,
to hierarchically aggregate instance-level histology features in the tumor
microenvironment. Wang et al. [36] proposed a Surformer that can quantify
specific histological patterns through bag-level labels without any
patch/cell-level auxiliary information.

Both WSIs and genomic data have their unique strengths and can be
synergistically complementary to each other. Merely relying on one data
modality will inevitably lose some critical information of patients, leading
to suboptimal prognosis.

###  2.2 Multi-modal for Cancer Prognosis

Recent histo-genomic multi-modal methods have demonstrated significant success
in the field of precision medicine [43, 38, 3, 4, 20, 26, 6, 28, 46]. Yao et
al. [43] proposed the Deep Correlational Survival Model (DeepCorrSurv), which
comprises view-specific and common sub-networks, for integrating multi-view
data. They explicitly maximize the correlation among the views and transfer
the learned knowledge for the survival regression task. Chen et al. [4]
proposed the Multimodal Co-Attention Transformer (MCAT) framework to learn an
interpretable, dense co-attention mapping between WSIs and genomic data. Li et
al. [26] proposed a novel hierarchical multimodal fusion approach termed
HFBSurv. HFBSurv employs a factorized bilinear model to fuse genomic and image
features and connect multi-modal information from low-level to high-level.
Chen et al. [6] utilized multimodal deep learning to jointly examine WSIs and
molecular profile data from 14 cancer types and discovered prognostic features
correlated to the outcomes. Lipkova et al. [28] specifically discussed three
different fusion strategies in multi-modal learning and emphasized the
enormous potential of personalized precision medicine.

Despite the promising performance, the clinical application of these methods
can be hindered by the shortage and associated sequencing costs and
examinations. Therefore, in this study, we aim to leverage the available
histo-genomic data pairs to develop a new and powerful deep network with
enhanced performance to facilitate WSI-based inference and improve cancer
prognosis.

##  3 Method

###  3.1 Overview and Problem Formulation

As illustrated in Fig. 1, the framework comprises the data processing step and
G-HANet survival prognosis. Initially, we convert the WSIs into bags of
histopathology features and categorize gene profiles according to their
functions. Afterward, the processed bags are fed into the G-HANet for
functional gene reconstruction and survival prediction. In this process, the
cross-modal associating branch distills the histo-genomic associations and
generates insightful features reflecting gene expression profiles.
Subsequently, the hyper-attention survival branch utilizes the distilled
associations and generated morphology-based weights to model the patients
using WSIs from both histopathology and genomic perspectives. It is worth
noting that genome-correlated operations in the red boxes (i.e., genome
sequencing, function categorization, and genome reconstruction) will be
discarded during the inference stage, which enables the WSI-based uni-modal
inference.

WSI bags formulation. WSIs inherently have gigapixel resolutions, which pose
considerable challenges for end-to-end processing using current hardware.
Building on previous studies [37, 4, 6], we employ the weakly supervised
multiple-instance learning (MIL) strategy [31], through which each WSI can be
formulated as multiple instances assigned with one bag-level label. We
initially utilize the automated segmentation algorithm to distinguish the
tissue regions from the background and then crop the informative tissues into
a series of patches $X^{p}=\\{x_{i}^{p}\\}_{i=1}^{N_{p}}$. Eventually, the
$N_{p}$ cropped patches are converted into $d$-dimensional feature vectors
through a pre-trained encoder $E$ as:

$F^{p}=\\{E(x_{i}^{p}):x_{i}^{p}\in
X^{p}\\}=\\{f_{i}\\}_{i=1}^{N_{p}}\in\mathbb{R}^{N_{p}\times d}.$ (1)

Genome Pre-processing. Genome profiles, e.g., gene mutation status, copy
number variation (CNV), and bulk RNA-Seq abundance, are quantified as 1x1
measurements. As such, they cannot directly convey semantic characteristics in
terms of functions or signatures in the context of genome analysis. In this
study, we build upon the previous works [4, 41] and categorize the genes by
referring to the six genomic functions: 1) tumor suppression, 2) oncogenesis,
3) protein kinases, 4) cellular differentiation, 5) transcription, and 6)
cytokines and growth. In G-HANet, we aim to distill the histo-genomic
associations by reconstructing functional genes from WSIs. Given that only a
few genes are highly related to the cancer prognosis, reconstructing all the
genome data could potentially increase the unnecessary training burden of the
neural networks, leading to inferior performance. Consequently, we first
conducted the differential analysis using the DESeq2 [29] to identify the
differential genes between the high-risk and low-risk patients in the training
data. Here, the two patient groups were categorized according to their
survival status at the middle time point. The detailed sequence length
variations across functions and cancer types can be found in Table 1. Hereby,
the final functional genes can be represented as
$X^{g}=\\{x^{g}_{i}\\}_{i=1}^{N_{g}}$.

###  3.2 Cross-modal Reconstruction for Knowledge Distillation

The cross-modal associating branch (CAB) is designed to distill the histo-
genomic association matrix and generate functional features. Here, the matrix
can associate the genotypes and phenotypes, and the functional features can
reflect the gene expression profiles in the feature space, both of which will
be fed into HSB for survival prognosis. To achieve this target, we reconstruct
the functional genes $X^{g}$ from WSIs, in which way the deep correlations
between the two modalities can be automatically established. As illustrated in
Fig. 1, CAB proceeds with feature extraction through two multi-head cross-
attention (MHCA) modules with each accompanied by a feed-forward network
(FFN), and genome reconstruction through $N_{g}$ Self-Normalizing Networks
(SNNs) [22].

Regarding the first MHCA, it takes learnable tokens $T$ and the bag of feature
$F^{p}$ as input. In this process, $T$ targets collecting genomic function-
related morphological features and will be optimized throughout the training
process. The query ($Q$) is derived from $T$, and the key ($K$) and value
($V$) are derived from $F^{p}$, all of which are achieved through linear
transformations. The multi-head mechanism is employed to segment the features
into $n$ portions along the feature dimension, generating
$Q\in\mathbb{R}^{N_{g}\times n\times(d^{\prime}/n)}$ and
$K,V\in\mathbb{R}^{N_{p}\times n\times(d^{\prime}/n)}$. Subsequently, a cross-
product operation between $Q$ and $K$ is executed to derive the attention
matrix and a $softmax$ operation is applied to scale the attention values. For
this explanation, we assume $n=1$ and formulate this process as follows:

$m^{1^{\prime}}_{i,j}=\frac{exp(m_{i,j})}{\sum_{j=1}^{N_{p}}exp(m_{i,j})},\quad
m_{i,j}^{1}=\frac{Q_{i}\times K_{j}}{\sqrt{d_{k}}},$ (2)

where $\sqrt{d_{k}}$ is a scaling factor and, $i$ and $j$ present the indices
of $Q$ and $K$, respectively. Each element of the matrix, e.g., $m_{i,j}$,
serves as an indirect marker of the relative likeness between $Q_{i}$ (derived
from $T$) and $K_{j}$ (derived from $F^{p}$) in the feature space, which
essentially reflects the pair-wise association between the $i_{th}$ genomic
function and $j_{th}$ patch. Afterward, we aggregate the detected features
through the cross-product operation between $V$ and $m^{\prime}$ and transform
the features with a linear layer.

Upon aggregating the features, we employ the FFN for the feature nonlinear
transformation, getting functional features $f_{F1}$. Although the tokens $T$
can be optimized towards different functional genes, collecting critical
features from large amounts of patches could be challenging ($N_{p}\gg
N_{g}$). Here, we employ the summation of $T$ and aggregated features $f_{F1}$
as the query in the second MHCA for another round of detection and
aggregation. In this way, the query consists of knowledge from both optimized
$T$ and the last computation round, thereby ensuring more sensitivity towards
correlative features. We regard the attention matrix before the $softmax$
operation as the final association matrix ($M^{am}$). After the second
detection round, the features will be fed into another FFN for transformation
and produce the final functional features $f_{F2}$, which will be utilized for
subsequent genome reconstruction. The whole process can be represented as
follows:

$\displaystyle f_{F1}=FFN_{0}(MHCA_{0}(T,F^{p}));$ (3) $\displaystyle
f_{F2}=FFN_{1}(MHCA_{1}(T+f_{F1},F^{p})).$

Notably, the two MHCAs share the parameters to facilitate and stabilize the
model training.

Given the varying lengths of functional genes, a specific self-normalizing
network (SNN [22])-based prediction sub-branch is designated for each
category. SNN is a powerful network in modeling high-dimensional low-sample
size (HDLSS) scenarios. Here, each SNN sub-branch is constructed by two linear
layers, with the first followed by a layer normalization ($LN$) [1] and $ELU$
activation function [8]. We denote the reconstructed functional genes as
$P^{g}=\\{p_{i}^{g}\\}_{i=1}^{N_{g}}$.

Figure 2: Overview of the hyper-attention module. It takes the transformed bag
of features $F^{p}$ and association matrix $m$ as input and outputs the
aggregated genome-informed features $f_{GI}$.

###  3.3 Histo-genomic Hyper-attention for Prognosis

The hyper-attention survival branch (HSB) aims to generate comprehensive
prognosis representations for patients through hyper-attention WSI modeling
and feature long-range dependency establishment. Specifically, the proposed
hyper-attention module (HM) considers the complementary characteristics of
different data modalities and aggregates patch features from both
histopathology and genomic perspectives. The aggregated features concatenated
with $f_{F2}$ will be fed into the multi-head self-attention (MHSA) module and
FFN for dependency establishment across features, and into a classification
layer for survival prediction. In the following parts, we will introduce the
HM and survival prediction process in detail.

Hyper-attention Module. The proposed hyper-attention module is shown in Fig.
2. Specifically, it takes both $F^{p}$ and $M^{am}\in R^{N_{p}\times N_{g}}$
(from the second MHCA in CAB) as input. For the attention from the
histopathology perspective, we utilize the gated attention network (GAN) [19]
to generate the morphology-based weights ($w^{mor}$) for patches. Here, we
first feed $F^{p}$ into two parallel linear layers for feature transformation,
each of which is followed by the $sigmoid$ or $tanh$ activation functions,
respectively. Afterward, the dot product of the two transformed features is
fed into another linear layer to generate the patch-level attention weights.
We regard the attention weights after $softmax$ operation along the instance
dimension as the final morphology-based weights
$w^{mor}\in\mathbb{R}^{N_{p}\times 1}$. In terms of the attention from the
genomic perspective, we particularly operate $M^{am}$. We apply the $softmax$
operation on the top $k\%$ high-score patches that correspond to each category
of gene function and mask the rest as zeros, thereby getting normalized
$M^{am^{\prime}}$ ($\sum_{j=1}^{N_{p}}M_{i,j}^{am^{\prime}}=1$). The gradient
of $M^{am^{\prime}}$ was detached to maintain the learning autonomy of CAB.
The final hyper matrix $M_{hyper}$ is the mean of $w^{mor}$ and processed
$M^{am}$ and the whole process can be formulated as:

$M_{hyper}=\frac{w^{mor}+softmax^{k\%}(M^{am})}{2}.$ (4)

We take the cross product of the $M_{hyper}$ and $F^{p}$ after a linear
transformation as the genome-informed features $f_{GI}$.

Survival Prediction. We concatenate the genome-informed features $f_{GI}$ and
$f_{F2}$ as the feature matrix $f_{m}\in R^{N^{g}\times d^{\prime\prime}}$ to
represent the WSI. This bridging operation not only enriches the features for
survival analysis but also establishes the synergy between the two branches
for a joint gradient backpropagation. Then, the feature matrix will be fed
into the MHSA and FFN for further computation. Compared with MHCA, the query
($Q$), key ($K$), and value ($V$) of MHSA are all derived from the identical
input and there is a residual operation between the input and output features.
Afterward, we employ a linear layer followed by a $LN$ [1] and a $ReLU$
activation function for feature dimension compression as $f_{m}^{\prime}\in
R^{N_{g}\times d^{\prime\prime\prime}}$. Rather than employing a readout
function (mean or max) for aggregating the first dimension of
$f_{m}^{\prime}$, we flatten the features into a single features vector. In
this way, the features corresponding to different functional genes can be
presented in a series connection and the semantic information can be better
spatially preserved.

###  3.4 Loss Function

In this study, we aim to proceed with genome reconstruction and survival
prediction simultaneously. The genome profiles corresponding to certain
functions have very high feature dimensions with some exceeding 1,500,
indicating that precisely predicting all the functional genes is extremely
challenging. As a result of this, our CAB targets both the gene expression
levels and the functional genomic distribution using the mean square error
loss $\mathcal{L}_{MSE}$ and scaled cosine error loss $\mathcal{L}_{SCE}$
[17], respectively. $\mathcal{L}_{SCE}$ [17] has demonstrated its superiority
in reconstructing high-dimensional features in the self-supervised learning
setting. Specifically, it can be formulated as follows:

$\mathcal{L}_{SCE}=\frac{1}{|P^{g}|}\sum_{i=1}^{N^{g}}(1-\frac{(p_{i}^{g})^{T}x_{i}^{g}}{||p_{i}^{g}||\cdot||x_{i}^{g}||})^{\gamma},$
(5)

where $\gamma>=1$ controls the feature constraints, $x_{i}^{g}$ and
$p_{i}^{g}$ indicates the ground truth and predicted genes. Regarding the
hazard loss, we follow the previous works [4] and utilize the negative
likelihood loss $\mathcal{L}_{NLL}$ [44] for optimizing the survival branch.
$\mathcal{L}_{NLL}$ can be formulated as follows:

$\displaystyle\mathcal{L}_{NLL}=$ $\displaystyle-c_{i}\cdot
log(f_{surv}(y_{i},F_{i}^{p}))$ (6) $\displaystyle-(1-c_{i})\cdot
log(f_{surv}(y_{i}-1,F_{j}^{p}))$ $\displaystyle-(1-c_{i})\cdot
log(f_{hazard}(y_{i},F_{j}^{p})),$

where $c$ and $y$ indicate the censorship and overall survival of the patient,
respectively. $f_{surv}$ represents the accumulated hazard rates of the
patient:

$f_{surv}(y_{i},F_{i}^{p})=\prod_{j=0}^{y_{i}}(1-f_{hazard}(F_{i}^{p})[j]).$
(7)

The overall loss function is:

$\mathcal{L}=\mathcal{L}_{NLL}+\alpha(\mathcal{L}_{MSE}+\mathcal{L}_{SCE}),$
(8)

where $\alpha$=0.3 is a hyperparameter to balance the reconstruction and
survival prediction.

Table 1: Details of the five datasets in terms of genomic sequence lengths
before and after differential analysis.

Original sequence lengths
---
Function category | BLCA | BRCA | GBMLGG | LUAD | UCEC
tumor suppression | 94 | 91 | 84 | 89 | 3
oncogenesis | 334 | 353 | 314 | 334 | 24
protein kinases | 521 | 553 | 498 | 534 | 21
cellular differentiation | 468 | 480 | 415 | 471 | 22
transcription | 1,495 | 1,565 | 1,395 | 1,509 | 64
cytokines and growth | 479 | 482 | 428 | 482 | 15
Differential analyzed sequence lengths
Function category | BLCA | BRCA | GBMLGG | LUAD | UCEC
tumor suppression | 20.2 | 30.2 | 34.6 | 20.0 | 1.2
oncogenesis | 73.8 | 123.2 | 129.8 | 87.8 | 8.4
protein kinases | 102.2 | 109.6 | 177.4 | 146.4 | 7.2
cellular differentiation | 94.8 | 171.4 | 139.8 | 171.4 | 8.0
transcription | 351.6 | 538.6 | 535.2 | 386.2 | 24.0
cytokines and growth | 78.2 | 165.8 | 126.4 | 128.2 | 5.4

##  4 Experiments

###  4.1 Datasets and Evaluation Metrics

Following [4, 41], extensive experiments of this study were conducted on five
different cancer types from The Cancer Genome Atlas (TCGA) including Bladder
Urothelial Carcinoma (BLCA), Breast Invasive Carcinoma (BRCA), Glioblastoma &
Lower Grade Glioma (GBMLGG), Lung Adenocarcinoma (LUAD), and Uterine Corpus
Endometrial Carcinoma (UCEC) datasets. 5-fold cross-validation with the 4:1
ratio of training and validation sets is performed on each dataset. The mean
concordance index (c-index) with its standard deviation (std) is reported for
a comprehensive performance comparison.

Table 1 specifically details the sequence lengths of initial and after
differential analysis of the five datasets across six functional categories
(tumor suppressor genes, oncogenes, protein kinases, cell differentiation
markers, transcription, and cytokines and growth). It is important to note
that the differential analysis was conducted on the training samples of each
fold. As a result, the number of significant genes identified can vary, and
only the mean sequence lengths are reported. Owing to the markedly reduced
number of genes in the UCEC dataset relative to the other four datasets,
differential analysis was not performed while training the UCEC-based model.

Table 2: Performance comparison with state-of-the-art genome-based/WSI-based
uni-modal algorithms and histo-genomic multi-modal algorithms on TCGA datasets
in terms of c-index. The ’Patho.’ (WSI) and ’Geno.’ (genomic data) indicate
the data utilized during inference. ’KD’ represents the knowledge distillation
strategy. We label the best WSI-based inference model with red color and
highlight the best performance in bold format.

Methods | Patho. | Geno. | KD | BLCA | BRCA | GBMLGG | LUAD | UCEC | Overall
---|---|---|---|---|---|---|---|---|---
SNN [22] |  | ✓ |  | 0.618 ± 0.022 | 0.624 ± 0.060 | 0.834 ± 0.012 | 0.611 ± 0.047 | 0.679 ± 0.040 | 0.673
SNNTrans [22, 33] |  | ✓ |  | 0.659 ± 0.032 | 0.647 ± 0.063 | 0.839 ± 0.014 | 0.638 ± 0.022 | 0.656 ± 0.038 | 0.688
AttnMIL [19] | ✓ |  |  | 0.551 ± 0.049 | 0.577 ± 0.043 | 0.786 ± 0.026 | 0.561 ± 0.078 | 0.639 ± 0.057 | 0.623
DeepAttnMISL [42] | ✓ |  |  | 0.504 ± 0.042 | 0.524 ± 0.043 | 0.734 ± 0.029 | 0.548 ± 0.050 | 0.597 ± 0.059 | 0.581
CLAM-SB [30] | ✓ |  |  | 0.559 ± 0.034 | 0.573 ± 0.044 | 0.779 ± 0.031 | 0.594 ± 0.063 | 0.644 ± 0.061 | 0.630
CLAM-MB [30] | ✓ |  |  | 0.565 ± 0.027 | 0.578 ± 0.032 | 0.776 ± 0.034 | 0.582 ± 0.072 | 0.609 ± 0.082 | 0.622
TransMIL [33] | ✓ |  |  | 0.608 ± 0.139 | 0.626 ± 0.042 | 0.798 ± 0.033 | 0.641 ± 0.033 | 0.657 ± 0.044 | 0.666
DTFD-MIL [45] | ✓ |  |  | 0.546 ± 0.021 | 0.609 ± 0.059 | 0.792 ± 0.023 | 0.585 ± 0.066 | 0.656 ± 0.045 | 0.638
Surformer [36] | ✓ |  |  | 0.594 ± 0.027 | 0.628 ± 0.037 | 0.809 ± 0.026 | 0.591 ± 0.064 | 0.681 ± 0.028 | 0.661
Porpoise [3] | ✓ | ✓ |  | 0.636 ± 0.024 | 0.652 ± 0.042 | 0.834 ± 0.017 | 0.647 ± 0.031 | 0.695 ± 0.032 | 0.693
MCAT [4] | ✓ | ✓ |  | 0.672 ± 0.032 | 0.659 ± 0.031 | 0.835 ± 0.024 | 0.659 ± 0.027 | 0.649 ± 0.043 | 0.695
MOTCat [41] | ✓ | ✓ |  | 0.683 ± 0.026 | 0.673 ± 0.006 | 0.849 ± 0.028 | 0.670 ± 0.038 | 0.675 ± 0.040 | 0.710
CLAM-SB [30, 16] | ✓ |  | ✓ | 0.567 ± 0.026 | 0.601 ± 0.039 | 0.801 ± 0.030 | 0.570 ± 0.021 | 0.668 ± 0.057 | 0.641
CLAM-SB [30, 24] | ✓ |  | ✓ | 0.563 ± 0.066 | 0.589 ± 0.030 | 0.794 ± 0.029 | 0.577 ± 0.095 | 0.640 ± 0.068 | 0.633
CLAM-MB [30, 16] | ✓ |  | ✓ | 0.560 ± 0.047 | 0.585 ± 0.041 | 0.796 ± 0.030 | 0.579 ± 0.053 | 0.608 ± 0.058 | 0.626
CLAM-MB [30, 24] | ✓ |  | ✓ | 0.618 ± 0.029 | 0.533 ± 0.050 | 0.779 ± 0.026 | 0.578 ± 0.053 | 0.635 ± 0.066 | 0.629
Ours | ✓ |  | ✓ | 0.630 ± 0.032 | 0.664 ± 0.065 | 0.817 ± 0.022 | 0.612 ± 0.028 | 0.729 ± 0.050 | 0.690

Table 3: Analysis $k\%$ in CAB on five TCGA datasets.

$k$ | BLCA | BRCA | GBMLGG
---|---|---|---
10 | 0.619 ± 0.034 | 0.650 ± 0.054 | 0.815 ± 0.021
15 | 0.625 ± 0.025 | 0.664 ± 0.065 | 0.817 ± 0.022
20 | 0.630 ± 0.032 | 0.651 ± 0.063 | 0.814 ± 0.020
25 | 0.626 ± 0.033 | 0.654 ± 0.064 | 0.811 ± 0.019
30 | 0.624 ± 0.036 | 0.650 ± 0.057 | 0.812 ± 0.021
35 | 0.620 ± 0.026 | 0.647 ± 0.064 | 0.814 ± 0.022
$k$ | LUAD | UCEC | Overall
10 | 0.604 ± 0.041 | 0.706 ± 0.037 | 0.679
15 | 0.610 ± 0.039 | 0.704 ± 0.030 | 0.684
20 | 0.612 ± 0.028 | 0.717 ± 0.021 | 0.685
25 | 0.604 ± 0.045 | 0.721 ± 0.057 | 0.683
30 | 0.612 ± 0.039 | 0.727 ± 0.056 | 0.685
35 | 0.609 ± 0.041 | 0.729 ± 0.050 | 0.684

###  4.2 Implementation Details

All the WSIs undergo processing at 20 times (20 $\times$) magnification level.
During the pre-processing step, an automated segmentation algorithm was
utilized to eliminate the background patches with a low saturation value ($<$
15). Subsequently, the tissue regions were cropped into a series of non-
overlapping patches with a unified 256 $\times$ 256 resolution. Then, a
modified ImageNet [12] pre-trained ResNet50 model [14] (constructed by one
convolution block and three residual blocks) embedded the original patches
into a collection of feature vectors. In the feed-forward computation process,
we trained the neural network with the Adam optimizer for 20 epochs in an end-
to-end fashion. In each iteration, the batch size was set as 1 and the
gradients were accumulated over 32 steps for back-propagation. The learning
rate was initialized as 0.0002 for training the BLCA, BRCA, GBMLGG, and LUAD
datasets. Regarding the UCEC datasets, the learning rate was set as 0.0001.
All experiments were carried out using one NVIDIA GeForce RTX 3090 Card.

Figure 3: Kaplan-Meier survival curves of TransMIL and our proposed G-HANet on
five cancer datasets. High-risk and low-risk patients are indicated with red
and blue, respectively. The shaded areas indicate the confidence intervals of
each cohort. P-value $<$ 0.05 represents a significant statistical difference
between the two groups.  Table 4: Ablation study of the proposed G-HANet on
TCGA datasets in terms of c-index. CAB, CSB, and HSB refer to the cross-modal
associating branch, cross-attention survival branch, and hyper-attention
survival branch, respectively.

Methods | CAB1 | CAB2 | CSB | HSB | BLCA | BRCA | GBMLGG | LUAD | UCEC
---|---|---|---|---|---|---|---|---|---
$Model_{0}$ |  |  |  |  | 0.551 ± 0.049 | 0.577 ± 0.043 | 0.786 ± 0.026 | 0.561 ± 0.078 | 0.639 ± 0.057
$Model_{1}$ | ✓ |  | ✓ |  | 0.587 ± 0.024 | 0.630 ± 0.046 | 0.807 ± 0.026 | 0.584 ± 0.051 | 0.676 ± 0.032
$Model_{2}$ |  | ✓ | ✓ |  | 0.614 ± 0.064 | 0.647 ± 0.055 | 0.816 ± 0.027 | 0.612 ± 0.050 | 0.692 ± 0.021
$Model_{3}$ | ✓ |  |  | ✓ | 0.590 ± 0.024 | 0.640 ± 0.051 | 0.808 ± 0.021 | 0.590 ± 0.029 | 0.695 ± 0.030
$Model_{4}^{*}$ |  | ✓ |  | ✓ | 0.587 ± 0.048 | 0.646 ± 0.021 | 0.808 ± 0.017 | 0.594 ± 0.038 | 0.698 ± 0.028
$Model_{5}^{**}$ |  | ✓ |  | ✓ | 0.622 ± 0.026 | 0.647 ± 0.050 | 0.816 ± 0.015 | 0.604 ± 0.040 | 0.689 ± 0.045
$Model_{6}^{{\dagger}}$ |  | ✓ |  | ✓ | 0.617 ± 0.048 | 0.643 ± 0.063 | 0.808 ± 0.022 | 0.583 ± 0.041 | -
$Model_{7}$ |  | ✓ |  | ✓ | 0.630 ± 0.032 | 0.651 ± 0.063 | 0.814 ± 0.020 | 0.612 ± 0.028 | 0.717 ± 0.021

###  4.3 Comparison with State-of-the-art Methods

In this subsection, we compare our proposed G-HANet with other state-of-the-
art uni-modal (WSI-based and genome-based) methods and multi-modal methods in
Table 2. In terms of genome-based methods, we specifically implement SNN [22]
and SNNTrans [22, 33]. SNN [22] takes the concatenated genomic profiles as a
feature vector and predicts the survival. SNNTrans [22, 33] initially
categorizes the genomic data according to the functions and then utilizes
TransMIL [33] to predict the overall survival. As for WSI-based methods,
DeepAttnMISL [42], CLAM-SB [30], CLAM-MB [30] are all based on the attention
mechanism for representation learning. TransMIL [33] and Surformer [36] are
two Transformer-based methods with distinct learning strategies. TransMIL [33]
explores the long-range dependencies among patches; whereas Surformer [36]
utilizes the prototypes for pattern-specific feature aggregation. DTFD-MIL
[45] introduces the pseudo bags and employs knowledge distillation for
empowering the feature learning capability of the neural networks. Regarding
multi-modal methods, the attention-based Porpoise [3], Transformer-based MCAT
[4], and optimal-transport-based MOTCat [41] are introduced for a more
challenging comparison. We also implement KD-based [16, 24] CLAM-SB [30] and
CLAM-MB [30] with MCAT [4] as the teacher network.

G-HANet vs genome-based methods. When comparing G-HANet with SNN [22], G-HANet
surpasses SNN in performance on the BLCA, BRCA, LUAD, and UCEC datasets by
1.2%, 4.0%, 0.1%, and 5.0%, respectively. Against SNNTrans [22, 33], G-HANet
achieves better performance on the BRCA and UCEC dataset. Based on this
comparison, we can conclude that with the assistance of genomic data during
training, G-HANet can derive critical features from WSIs and achieve
competitive and even superior performance than genome-based methods.

G-HANet vs WSI-based methods. In this comparison, we witness a significant
performance improvement of G-HANet compared with other WSI-based methods. On
the BLCA, BRCA, GBMLGG, and UCEC datasets, G-HANet outperforms the second-best
methods by 2.2%, 3.6%, 0.8%, and 4.8%, respectively. On the LUAD dataset,
TransMIL [33] achieves a c-index of 0.641, surpassing G-HANet in performance
by 2.9%. However, TransMIL [33] has been trapped for its substantial increase
in computational demands when applied to large-scale Whole Slide Images
(WSIs). Meanwhile, the overall performance of TransMIL [33] on the five
datasets is 2.4% lower than G-HANet. We also implement response-based [16] and
representation-based [24] KD on CLAM-MB [30] and CLAM-SB [30] networks with
MCAT [4] as the teacher. By comparing KD-based methods with other WSI-based
methods, it is evident that KD [16, 24] does not yield benefits across every
dataset, and the overall improvement it offers is limited. Furthermore, the
representation-based KD [24] is designed to transfer the knowledge of a big
model to a small one with a similar network architecture, therefore, it is not
suitable for transferring knowledge from a multi-modal network to a uni-modal
one. Overall, we can conclude that our proposed G-HANet is superior to current
WSI-based methods and our distilling strategy is more suitable for distilling
multi-modal knowledge to elevate uni-modal inference.

G-HANet vs multi-modal methods.  In the comparison of G-HANet with multi-modal
methods, it achieves competitive performance on the BRCA dataset and the best
performance on the UCEC dataset. On the UCEC datasets, G-HANet surpasses the
MOTCat [41] and Porpoise [3] by 5.4% and 3.4%, respectively. We attribute the
moderate performance of the multi-modal methods to the limited number of
detected genes. As shown in Table 1 last column, the genes corresponding to
each function are mostly below 20. Moreover, not all the genes are correlative
to cancer prognosis. Merely relying on the communication between the two
modalities to model the patients will lose many critical features, thereby
resulting in inferior model performance. Consequently, it is evidenced that
G-HANet exhibits strong performance in comparison with the multi-modal methods
and demonstrates resilience to variations in genomic lengths.

Figure 4: Violin plots of the Spearman Correlation coefficients of six genomic
functions on five TCGA datasets. The mean values and corresponding standard
variations are placed next to each plot.

###  4.4 Ablation Study

Analysis of top $k\%$ in CAB: In CAB, we select the top $k\%$ patches for
generating the genome-informed features. Here, we specifically discuss the
impact of $k$ across the five TCGA datasets. As shown in Table 3, for
different cancer types, the model performs best with different $k$ values. On
BLCA and LUAD datasets, G-HANet achieves the best performance when $k=20$. On
BRCA and GBMLGG datasets, the optimal threshold is $k=15$. Regarding the UCEC
dataset, the optimal $k$ is 30. When $k=20$ and $30$, we can achieve the best
overall performance on the five datasets reaching 0.685. Given the cross-tumor
heterogeneity, which leads to variations in characteristics of progression,
cell types, and immune responses, it is reasonable to expect that different
datasets may have distinct optimal values for $k$.

Analysis of the proposed CAB and HSB: In this subsection, we discuss the
effectiveness of the proposed CAB and HSB. We set $k=20$ for a fair
comparison. The experiment results are shown in Table 4. $Model_{0}$ is the
baseline that utilizes a gated attention network (GAN) [19] to model the WSIs
from the histopathology view. Regarding the CAB, our discussion focuses on two
main branches: the self-derived-based and the cross-attention-based.
Specifically, CAB1 and CAB2 generate the features for the six functional
categories through GAN [19] and MHCA (Section III. B), respectively. As for
the survival branch, we replace the $M_{hyper}$ in HSB with the MHCA using
$f_{F2}$ as the query and term the modified one as the cross-attention
survival branch (CSB). Moreover, we indicate the HSB without the feature
bridging operation and merely relying on $M^{am}$ for patch aggregation as
$Model_{4}^{*}$ and $Model_{5}^{**}$, respectively. $Model_{6}^{{\dagger}}$
did not implement differential analysis on the genomic data. $Model_{7}$
represents our proposed G-HANet.

Here, we take the model performance on the BLCA as an example. By comparing
$Model_{0}$ with others, we can witness a significant performance gap, which
shows that introducing genome data into model training is quite effective. By
comparing $Model_{1}$, $Model_{2}$, $Model_{3}$, and $Model_{7}$, we can find
that our CAB2 is much more powerful than CAB1 with $Model_{2}$ outperforming
$Model_{1}$ by 1.7% and $Model_{7}$ outperforming $Model_{3}$ by 2.0%. By
comparing $Model_{7}$ with $Model_{2}$, there is also a 2.1% improvement,
indicating that utilizing the $M^{am}$ and $w^{mor}$ jointly for aggregating
patches can emphasize more survival-related features than the cross-attention-
based method. The comparison of $Model_{4}^{*}$, $Model_{5}^{**}$, and
$Model_{7}$ highlights the effectiveness of feature bridging operation and
$M^{am}$ in generating comprehensive representations for patients.
Additionally, without genomic data differential analysis, the trained models
would have performance decrease to some extent. In all, each proposed module
is effective in the representation learning process and can work independently
and cooperatively.

###  4.5 Genome Prediction Analysis

In the proposed G-HANet, we explored the histo-genomic associations by genome
reconstruction using CAB. Given the fact that predicting the expression values
of all the genes (more than 500 genes after differential analysis) from the
WSIs is technically quite challenging. Here, we evaluated the prediction
performance using the Spearman Correlation, which is a non-parametric measure
of the strength and direction of association that exists between two variables
measured on an ordinal scale. The coefficient ranges from -1 to +1, where -1,
0, and +1 indicate perfect negative, no, and perfect positive associations of
the ranks, respectively. As shown in Figure 4, the mean SC coefficients of the
six functional categories on the five datasets generally range from 0.2 to
0.3, which suggests a weak positive linear relationship between the ranks of
the two variables being compared. Although the trained CAB cannot accurately
predict the expression values for all the genes, the learned hidden
correlations could truly help understand the potential functions of patches,
thereby improving the characterization of patients from WSIs and enhancing the
cancer prognosis accuracy.

###  4.6 Kaplan-Meier Survival Curve

The Kaplan-Meier (KM) survival curve serves as a non-parametric approach to
display the survival probability over time for one or more groups. In this
subsection, we employ it to visualize the differences in survival status
between the predicted high-risk and low-risk cohorts. Specifically, the median
of the predicted survival rate at the middle time point was utilized as a
criterion to categorize the patients into the high-risk and low-risk groups
with an equal number of patients. Moreover, the log-rank test is conducted to
evaluate the statistically significant difference between the two groups of
patients. Specifically, we visualize the KM curves of both TransMIL [33] and
our proposed G-HANet on the five datasets in Fig. 3.

In all cases, G-HANet exhibits P-values smaller than 5e-2, whereas the P-value
for TransMIL on the BLCA dataset exceeds 5e-2. This means that TransMIL
trained on the BLCA dataset could not properly estimate the survival status of
patients, having bad generalization capability. G-HANet outperforms TransMIL
in terms of achieving smaller P-values on four datasets, except for an
identical result on the GBMLGG dataset. Moreover, G-HANet could better
distinguish the two patient groups on the five datasets qualitatively. For
example, patients from the BLCA and BRCA datasets could be better identified
by G-HANet before 8 and 10 months, respectively. Although TransMIL [33]
outperforms G-HANet on the LUAD dataset in terms of c-index, the KM-curve
indicates that patients living over 10 months are not significantly
distinguishable. Based on the above observations, it is evident that G-HANet
excels in distinguishing high-risk and low-risk cohorts and it can emerge as a
powerful model for cancer prognosis.

Figure 5: Histo-genomic association visualization from the G-HANet. Here, we
particularly visualized the histo-genomic associations of the six different
functional genes and presented the top four highlighted patches. Two WSIs from
the GBMLGG and UCEC datasets were taken as examples.

###  4.7 Histo-genomic Association Visualization

Visualizing the histo-genomic associations derived from the CAB is crucial for
model interpretation. In Figure 5, we employed two WSIs from the UCEC and
GBMLGG datasets to depict how specific histopathological patches relate to
various functional gene categories, i.e., tumor suppressors, oncogenes,
protein kinases, cellular differentiation, transcription, and cytokines.
Highlighted top four patches were also retrieved from the original WSI.

In essence, tumor suppressor genes serve to regulate cell growth and division,
whereas oncogenes are mutated or overexpressed versions of normal genes. The
two varieties of genes can be seen as opposing in the context of cancer
development and progression [25]. The illustration provided demonstrates a
clear distinction between the morphological patterns associated with tumor
suppressor genes and oncogenesis, with minimal overlap observed. Additionally,
it was noted that genes related to oncogenesis and cytokines tend to express
in similar regions. This observation is consistent with findings that oncogene
mutations can have a direct impact on cytokine gene expression [11]. In all,
this visualization could aid pathologists in comprehending the model’s
predictive mechanism, thereby enhancing the credibility and deployment of
trustworthy AI systems.

##  5 Conclusion

In this study, we have proposed a novel multi-modal deep neural network,
termed Genome-informed Hyper-Attention Network (G-HANet), to improve cancer
prognosis. It significantly elevates the network performance on WSI-based
inference through multi-modal knowledge distillation of the histo-genomic
associations in the training phase. Extensive experiments are conducted on
five different TCGA benchmarking datasets and illustrate that G-HANet
outperforms current state-of-the-art WSI-based methods and is on par with many
genome-based and multi-modal methods. Our approach is tailored to address the
current bottleneck of acquiring histo-genomic data pairing during model
development and deployment. Future work includes the exploration of a more
adaptable algorithm that can accept both uni-modal and multi-modal data to
achieve an accurate prognosis.

## References

  * [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [2] Kumardeep Chaudhary, Olivier B Poirion, Liangqun Lu, and Lana X Garmire.  Deep learning–based multi-omics integration robustly predicts survival in liver cancer.  Clinical Cancer Research, 24(6):1248–1259, 2018.
  * [3] Richard J Chen and et al.  Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis.  IEEE Transactions on Medical Imaging, 41(4):757–770, 2020.
  * [4] Richard J Chen and et al.  Multimodal co-attention transformer for survival prediction in gigapixel whole slide images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4025, 2021.
  * [5] Richard J Chen and et al.  Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24, pages 339–349. Springer, 2021.
  * [6] Richard J Chen and et al.  Pan-cancer integrative histology-genomic analysis via multimodal deep learning.  Cancer Cell, 40(8):865–878, 2022.
  * [7] Travers Ching, Xun Zhu, and Lana X Garmire.  Cox-nnet: an artificial neural network method for prognosis prediction of high-throughput omics data.  PLoS computational biology, 14(4):e1006076, 2018.
  * [8] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus).  arXiv preprint arXiv:1511.07289, 2015.
  * [9] David Collett.  Modelling survival data in medical research.  CRC press, 2023.
  * [10] David R Cox.  Regression models and life-tables.  Journal of the Royal Statistical Society: Series B (Methodological), 34(2):187–202, 1972.
  * [11] George D Demetri and et al.  Expression of ras oncogenes in cultured human cells alters the transcriptional and posttranscriptional regulation of cytokine genes.  The Journal of clinical investigation, 86(4):1261–1269, 1990.
  * [12] Jia Deng and et al.  Imagenet: A large-scale hierarchical image database.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 248–255. Ieee, 2009.
  * [13] Jacques Ferlay and et al.  Estimating the global cancer incidence and mortality in 2018: Globocan sources and methods.  International journal of cancer, 144(8):1941–1953, 2019.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [15] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi.  Knowledge transfer via distillation of activation boundaries formed by hidden neurons.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3779–3787, 2019.
  * [16] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [17] Zhenyu Hou and et al.  Graphmae: Self-supervised masked graph autoencoders.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 594–604, 2022.
  * [18] Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu.  Knowledge distillation from a stronger teacher.  Advances in Neural Information Processing Systems, 35:33716–33727, 2022.
  * [19] Maximilian Ilse, Jakub Tomczak, and Max Welling.  Attention-based deep multiple instance learning.  In International conference on machine learning, pages 2127–2136. PMLR, 2018.
  * [20] Guillaume Jaume, Anurag Vaidya, Richard Chen, Drew Williamson, Paul Liang, and Faisal Mahmood.  Modeling dense multimodal interactions between biological pathways and histology for survival prediction.  arXiv preprint arXiv:2304.06819, 2023.
  * [21] Taehyeon Kim, Jaehoon Oh, NakYil Kim, Sangwook Cho, and Se-Young Yun.  Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation.  arXiv preprint arXiv:2105.08919, 2021.
  * [22] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter.  Self-normalizing neural networks.  Advances in neural information processing systems, 30, 2017.
  * [23] David G Kleinbaum and Mitchel Klein.  Survival analysis a self-learning text.  Springer, 1996.
  * [24] Nikos Komodakis and Sergey Zagoruyko.  Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer.  In ICLR, 2017.
  * [25] Emmanuel N Kontomanolis and et al.  Role of oncogenes and tumor-suppressor genes in carcinogenesis: a review.  Anticancer research, 40(11):6009–6015, 2020.
  * [26] Ruiqing Li, Xingqi Wu, Ao Li, and Minghui Wang.  Hfbsurv: hierarchical multimodal fusion with factorized bilinear models for cancer survival prediction.  Bioinformatics, 38(9):2587–2594, 2022.
  * [27] Ruoyu Li, Jiawen Yao, Xinliang Zhu, Yeqing Li, and Junzhou Huang.  Graph cnn for survival analysis on whole slide pathological images.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II, pages 174–182. Springer, 2018.
  * [28] Jana Lipkova and et al.  Artificial intelligence for multimodal data integration in oncology.  Cancer cell, 40(10):1095–1110, 2022.
  * [29] Michael I Love, Wolfgang Huber, and Simon Anders.  Moderated estimation of fold change and dispersion for rna-seq data with deseq2.  Genome biology, 15(12):1–21, 2014.
  * [30] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood.  Data-efficient and weakly supervised computational pathology on whole-slide images.  Nature biomedical engineering, 5(6):555–570, 2021.
  * [31] Oded Maron and Tomás Lozano-Pérez.  A framework for multiple-instance learning.  Advances in neural information processing systems, 10, 1997.
  * [32] Nikolaos Passalis and Anastasios Tefas.  Learning deep representations with probabilistic knowledge transfer.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 268–284, 2018.
  * [33] Zhuchen Shao and et al.  Transmil: Transformer based correlated multiple instance learning for whole slide image classification.  Advances in neural information processing systems, 34:2136–2147, 2021.
  * [34] Ashish Vaswani and et al.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [35] Chunyu Wang and et al.  A cancer survival prediction method based on graph convolutional network.  IEEE transactions on nanobioscience, 19(1):117–126, 2019.
  * [36] Zhikang Wang and et al.  Surformer: An interpretable pattern-perceptive survival transformer for cancer survival prediction from histopathology whole slide images.  Computer Methods and Programs in Biomedicine, 241:107733, 2023.
  * [37] Zhikang Wang and et al.  Targeting tumor heterogeneity: multiplex-detection-based multiple instance learning for whole slide image classification.  Bioinformatics, 39(3):btad114, 2023.
  * [38] Zhiqin Wang, Ruiqing Li, Minghui Wang, and Ao Li.  Gpdbn: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction.  Bioinformatics, 37(18):2963–2970, 2021.
  * [39] Xiaohan Xing and et al.  Multi-level attention graph neural network based on co-expression gene modules for disease diagnosis and prognosis.  Bioinformatics, 38(8):2178–2186, 2022.
  * [40] Xiaoyi Xu, Ya Zhang, Liang Zou, Minghui Wang, and Ao Li.  A gene signature for breast cancer prognosis using support vector machine.  In 2012 5th International conference on biomedical engineering and informatics, pages 928–931. IEEE, 2012.
  * [41] Yingxue Xu and Hao Chen.  Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction.  arXiv preprint arXiv:2306.08330, 2023.
  * [42] Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang.  Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks.  Medical Image Analysis, 65:101789, 2020.
  * [43] Jiawen Yao, Xinliang Zhu, Feiyun Zhu, and Junzhou Huang.  Deep correlational learning for survival prediction from multi-modality data.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 406–414. Springer, 2017.
  * [44] Shekoufeh Gorgi Zadeh and Matthias Schmid.  Bias in cross-entropy-based training of deep survival networks.  IEEE transactions on pattern analysis and machine intelligence, 43(9):3126–3137, 2020.
  * [45] Hongrun Zhang and et al.  Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18802–18812, 2022.
  * [46] Fengtao Zhou and Hao Chen.  Cross-modal translation and alignment for survival analysis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21485–21494, 2023.


# Improving Large-scale Language Models and Resources for Filipino

Jan Christian Blaise Cruz and Charibeth Cheng
Center for Language Technologies
College of Computer Studies
De La Salle University, Manila
{jan_christian_cruz<EMAIL_ADDRESS>

###### Abstract

In this paper, we improve on existing language resources for the low-resource
Filipino language in two ways. First, we outline the construction of the
TLUnified dataset, a large-scale pretraining corpus that serves as an
improvement over smaller existing pretraining datasets for the language in
terms of scale and topic variety. Second, we pretrain new Transformer language
models following the RoBERTa pretraining technique to supplant existing models
trained with small corpora. Our new RoBERTa models show significant
improvements over existing Filipino models in three benchmark datasets with an
average gain of 4.47% test accuracy across the three classification tasks of
varying difficulty.

##  1 Introduction

Unlike High-resource Languages (HRL) such as English, German, and French, Low-
resource Languages (LRL) suffer from a lack of benchmark datasets, databases,
linguistic tools, and pretrained models that impede the progress of research
within those languages.

Despite the growing success of methods that intrinsically learn from little
data Deng et al. (2020); Lee et al. (2021), creating “more data” remains a
very significant fundamental task in NLP. Given the data-hungry nature of the
neural networks that are prevalent in NLP today, creating new datasets to
train from is the most efficient way to improve model performance. In
addition, cleverly-constructed datasets also reveal new insights into the
models we commonly use, letting us gauge their true performance and expose
hidden weaknesses Maudslay and Cotterell (2021).

In this paper, we improve upon the existing resources for Filipino, a low-
resource language spoken in the Philippines. We create a larger, more
topically-varied large-scale pretraining dataset that improves upon the
existing WikiText-TL-39 Cruz and Cheng (2019) that is too small and too
topically-narrow to create robust models that perform well in modern NLP. We
also produce new RoBERTa pretrained models using our pretraining dataset that
supplant existing models trained with less data Cruz and Cheng (2020).

##  2 Resource Creation

In this section, we outline our full methodology for resource creation. First,
we introduce the construction of our new large-scale pretraining dataset.
Next, we detail the pretraining steps for our new RoBERTa models. Lastly, we
introduce the task datasets that we use to benchmark performance for our new
pretrained models.

###  2.1 The TLUnified Dataset

To effectively pretrain a large transformer for downstream tasks, we require
an equally large pretraining corpus of high-quality Filipino text. We
construct our pretraining corpus by combining a number of available Filipino
corpora, including:

  * •

Bilingual Text Data – Bitext datasets are used for training Machine
Translation models and contain crawled and aligned data from multiple sources.
We collected multiple bitexts, extracted Filipino text, then deduplicated the
extracted data to add to our pretraining corpus. Datasets we collected from
include bible-uedin Christodouloupoulos and Steedman (2015), CCAligned El-
Kishky et al. (2020), ELRC 2922111https://elrc-share.eu/, MultiCCAligned El-
Kishky et al. (2020),ParaCrawl 222https://www.paracrawl.eu/, TED2020 Reimers
and Gurevych (2020), WikiMatrix Schwenk et al. (2019), tico-19, Ubuntu,
OpenSubtitles, QED, Tanzil, Tatoeba, GlobalVoices, KDE4, and WikiMedia
Tiedemann (2012).

  * •

OSCAR – The Open Super-Large Crawled Aggregated Corpus (OSCAR) Ortiz Suárez et
al. (2019) is a massive dataset obtained from language identification and
filtering of the Common Crawl dataset. We use the deduplicated version of the
Filipino (Tagalog) portion of OSCAR and add it to our pretraining corpus.

  * •

NewsPH – The NewsPH Cruz et al. (2021) corpus is a large-scale crawled corpus
of Filipino news articles, originally used in automatically creating the
NewsPH-NLI benchmark dataset. Since we plan on using an NLI dataset derived
from NewsPH for benchmarking in this paper, we opted to only use a 60% subset
of the NewsPH corpus to add to TLUnified.

Since a large portion of our corpus is crawled and artificially aligned, we
expect that out-of-the-box data quality would be low. To clean our dataset, we
apply a number of preprocessing filters to it, including:

  1. 1.

Non-latin Filter – We filter out sentences whose characters are composed of
more than 15% non-latin letters.

  2. 2.

Length Filter – We remove sentences that have a number of tokens $N$ where
$4<=N<=150$.

  3. 3.

Puncutation Filter – All sentences that have tokens composed of too many
succeeding punctuations (eg. “///”) are all removed.

  4. 4.

Average Word Length Filter – If a sentence has tokens that are significantly
longer than the other tokens in the sentence, we remove the sentence entirely.
We first take the sum of the character lengths of each token, then divide it
by the number of tokens to get a ratio $r$. Only sentences with ratio
$3<=r<=18$ are kept in the corpus.

  5. 5.

HTML Filter – All sentences with HTML and URL-related tokens (e.g. “.com” or
“http://”) are removed.

After filtering the dataset, we perform one additional deduplication step to
ensure that no identical lines are found in the dataset. The final result is a
large-scale pretraining dataset we call TLUnified.

We then train tokenizers using TLUnified, limiting our vocabulary to a fixed
32,000 BPE subwords Sennrich et al. (2015). Our tokenizers are trained with a
character coverage of 1.0. We also do not remove casing to ensure that
capitalization is kept after tokenization.

###  2.2 Pretraining

We then pretrain transformer language models that can serve as bases for a
variety of downstream tasks later on. For this purpose, we use the RoBERTa Liu
et al. (2019) pretraining technique. Previous pretrained transformers in
Filipino Cruz and Cheng (2020); Cruz et al. (2021) used BERT Devlin et al.
(2018), and ELECTRA Clark et al. (2020) as their method of choice.

We choose RoBERTa as it retains state-of-the-art performance on multiple NLP
tasks while keeping its pretraining task simple unlike methods such as
ELECTRA. As a reproduction study of BERT, RoBERTa optimizes and builds up on
the BERT pretraining scheme to improve training efficiency and downstream
performance.

Two size variants are trained in this study following the original RoBERTa
paper: a Base model (110M parameters) and a Large model (330M parameters).
Both size variants use the same BPE tokenizer trained with TLUnified. Our
hyperparameter choices also follow the original RoBERTa paper closely. A
summary of our models’ hyperparameters can be found in Table 1.

| Base | Large
---|---|---
Hidden Size | 768 | 1024
Feedforward Size | 3072 | 4096
Max Sequence Length | 512 | 512
Attention Heads | 12 | 16
Hidden Layers | 12 | 24
Droput | 0.1 | 0.1
Table 1: Base and Large RoBERTa hyperparameters.

During training, we construct batches by continually filling them with tokens
until we reach a maximum batch size of 8192 tokens. Both variants are trained
using the Adafactor Shazeer and Stern (2018) optimizer with $\beta_{2}=0.98$
and a weight decay of 0.01. The base model is trained for 100,000 steps with a
learning rate of 6e-4, while the large variant is trained for 300,000 steps
with a learning rate of 4e-4. We also use a learning rate schedule that
linearly warms up for 25,000 steps, then linearly decays for the rest of
training. All experiments are done on a server with 8x NVIDIA Tesla P100 GPUs.

###  2.3 Benchmark Datasets

We test the efficacy of our RoBERTa models on three Filipino benchmark
datasets:

  * •

Filipino Hatespeech Dataset – 10,000 tweets labelled as “hate” and “non-hate”
collected during the 2016 Philippine Presidential Elections. Originally
published in Cabasag et al. (2019) and benchmarked with modern Transformers in
Cruz and Cheng (2020).

  * •

Filipino Dengue Dataset – Low-resource multiclass classification dataset with
4000 samples that can be one or many of five labels. Originally published in
Livelo and Cheng (2018) and benchmarked in Cruz and Cheng (2020) using
pretrained Transformers.

  * •

NewsPH-NLI – An automatically-generated dataset constructed by exploiting the
“inverted-pyramid” structure of news articles, causing every sentence to
naturally entail the sentence that came before it. Originally created in Cruz
et al. (2021).

| Hatespeech | Dengue | NewsPH-NLI Med.
---|---|---|---
Model | Val. Acc | Test Acc. | Val. Acc | Test Acc. | Val. Acc | Test Acc.
BERT Base Cased | 0.7479 | 0.7417 | 0.7720 | 0.7580 | 0.8838 | 0.8874
ELECTRA Base Cased | 0.7491 | 0.7250 | 0.7400 | 0.6920 | 0.9094 | 0.9106
RoBERTa Base | 0.7866 | 0.7807 | 0.8180 | 0.8020 | 0.9492 | 0.9501
RoBERTa Large | 0.7897 | 0.7824 | 0.8281 | 0.8110 | 0.9499 | 0.9510
Table 2: Finetuning results for all Transformer variants on the three
benchmark datasets.

For this study, we do not use the original NewsPH-NLI created in Cruz et al.
(2021) as it has significant overlap with the subset of the NewsPH corpus that
we used for pretraining. We instead re-generated a version of NewsPH-NLI
(which we call “NewsPH-NLI Medium”) using 40% of the NewsPH corpus, using the
other 60% as part of the TLUnified pretraining data. This ensures that no test
data is present in the training data, which will significantly inflate the
benchmark scores.

Preprocessing for the downstream benchmark datasets is kept simple and non-
destructive to preserve the linguistic structures and information present in
the original data.

For the Hatespeech and the Dengue datasets, we follow the preprocessing used
in Cruz and Cheng (2020), with a number of changes. Since both are datasets
composed mainly of tweet data, the following preprocessing steps are done:

  * •

Moses detokenization Koehn and Hoang (2010) was applied on all Moses-tokenized
text.

  * •

All HTML meta text and link texts are collapsed into a special [LINK] token.
This is to reduce the noise in the dataset as images in the tweets are
naturally converted into links.

  * •

All substrings that start with an @ character that are greater than length 1
are automatically treated as a “mention” and are replaced with a [MENTION]
special token.

  * •

All substrings that start with a # character that are greater than length 1
are automatically treated as a “hashtag” and are replaced with a [HASHTAG]
special token.

  * •

We renormalize apostrophes (e.g. it ’s $\rightarrow$ it’s) and punctuation
that were spaced out (e.g. one - two $\rightarrow$ one-two) during the
preprocessing in the Cruz and Cheng (2020) paper.

  * •

Characters that were converted into unicode (e.g. &) are converted back
into their encoded form (e.g. &).

For the Dengue dataset, we transform the multilabel, multiclass classification
setup into a multiclass classification problem by concatenating an example’s
labels and converting the resulting binary number into an integer. For
example, a sentence with the labels 1, 1, 0, 1, 1 for absent, dengue,
healthclasses, mosquito, and sick will be converted into 27 (11011
$\rightarrow$ 27). This results in 32 possible labels and increases the
difficulty of the task.

For the NewsPH-NLI Medium dataset, we opted to not do any further
preprocessing as the released data from Cruz et al. (2021) is already
preprocessed and clean.

###  2.4 Finetuning Setups

We then finetune for the downstream benchmark tasks using our pretrained
RoBERTa models. Since the NewsPH-NLI version and the setup of the Dengue
dataset task is different from the previous benchmarking paper, we also
finetuned Tagalog BERT Cruz and Cheng (2019) and Tagalog ELECTRA Cruz et al.
(2021) to serve as baseline models against the new RoBERTa model.

| Base | Large
---|---|---
Max. Seq. Length | 128 | 256
Learning Rate | 2e-5 | 1e-5
Warmup Ratio | 0.1 | 0.06
Table 3: Unique finetuning hyperparameters for Base and Large transformer
variants.

All models are trained using the Adafactor Shazeer and Stern (2018) optimizer
with a learning rate scheduler that linearly increases from zero after a ratio
of steps-to-total-training-steps has reached, then linearly decays afterwards.
We use a batch size of 32 sentences for all models and use a weight decay of
0.1. We opted to use a larger maximum sequence length for the Large RoBERTa
models as it has more capacity due to its deeper encoder stack.
Hyperparameters that are different between Base and Large variants of the
pretrained Transformers used are found in Table 3.

We add the [LINK], [MENTION], and [HASHTAG] special tokens during finetuning
for the Hatespeech and Dengue datasets to the vocabularies of the Transformers
used, averaging the vectors of all subword embeddings in the embedding layer
to serve as initialization for the three added tokens.

Despite our RoBERTa having a full maximum sequence length allowance of 512, we
opted to use smaller maximum sequence lengths during finetuning. This speeds
up training (approximately 4x for the Base models and 2x for the Large models)
while losing zero information since no sentence or sentence pair in any task
reaches 256 subwords in length.

All experiments are done on a server with 8x NVIDIA Tesla P100 GPUs.

##  3 Results

We report the results for our finetuning for the three benchmark datasets in
terms of validation and test accuracy. A summary of the results can be found
on Table 2.

Our RoBERTa models outperformed both the BERT and the ELECTRA models across
all tasks. For the Hatespeech task, RoBERTa Large outperformed the best
previous model (BERT Base) by +4.07% test accuracy. RoBERTa large also had a
gain in performance in the Dengue dataset (+5.3% test accuracy over BERT Base)
and the NewsPH-NLI Medium dataset (+4.04% test accuracy over ELECTRA Base).

While marginally inferior to the Large variant, the Base RoBERTa variant still
outperforms the baseline models in all tasks. RoBERTa Base has an improvement
of +3.9% against BERT Base on the Hatespeech task, +4.4% against BERT Base on
the Dengue task, and +3.95% against ELECTRA Base on the NewsPH-NLI Medium
task.

The difference in performance between the Base and Large RoBERTa variants is
marginal in the current benchmarks. Large outperforms Base only by +0.17% for
Hatespeech, +0.9% for Dengue, and +0.09% for NewsPH-NLI Medium. We hypothesize
that this is due to the size of the pretraining dataset. While the size of
TLUnified is much larger than the previous WikiText-TL-39, it may still not be
enough to make full use of the capacity of a Large-variant Transformer. We
surmise that RoBERTa Large may need to be trained with more data to show
significant, non-marginal improvements in performance.

Overall, our new models show significant improvements over older pretrained
Filipino Transformer models. This is likely due to the improved pretraining
corpus, with TLUnified being larger and of more varied topics and sources than
the previous WikiText-TL-39.

##  4 Conclusion

Our work has two main contributions in terms of language resources for the
Filipino language. First, we construct TLUnified, a new large-scale
pretraining corpus for Filipino. This is an improvement over the much smaller
pretraining corpora currently available, boasting much larger scale and topic
variety. Second, we release new pretrained Transformers using the RoBERTa
pretraining method. Our new models outperform existing baselines on three
different classification tasks, with significant improvements of +4.07%,
+5.03%, and +4.04% test accuracy for the Hatespeech, Dengue, and NewsPH-NLI
Medium datasets respectively.

## References

  * Cabasag et al. (2019) Neil Vicente Cabasag, Vicente Raphael Chan, Sean Christian Lim, Mark Edward Gonzales, and Charibeth Cheng. 2019.  Hate speech in philippine election-related tweets: Automatic detection and classification using natural language processing.  _Philippine Computing Journal, XIV No_ , 1.
  * Christodouloupoulos and Steedman (2015) Christos Christodouloupoulos and Mark Steedman. 2015.  A massively parallel corpus: the bible in 100 languages.  _Language resources and evaluation_ , 49(2):375–395.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_.
  * Cruz and Cheng (2019) Jan Christian Blaise Cruz and Charibeth Cheng. 2019.  Evaluating language model finetuning techniques for low-resource languages.  _arXiv preprint arXiv:1907.00409_.
  * Cruz and Cheng (2020) Jan Christian Blaise Cruz and Charibeth Cheng. 2020.  Establishing baselines for text classification in low-resource languages.  _arXiv preprint arXiv:2005.02068_.
  * Cruz et al. (2021) Jan Christian Blaise Cruz, Jose Kristian Resabal, James Lin, Dan John Velasco, and Charibeth Cheng. 2021.  Exploiting news article structure for automatic corpus generation of entailment datasets.  In _Pacific Rim International Conference on Artificial Intelligence_ , pages 86–99. Springer.
  * Deng et al. (2020) Shumin Deng, Ningyu Zhang, Zhanlin Sun, Jiaoyan Chen, and Huajun Chen. 2020.  When low resource nlp meets unsupervised language model: Meta-pretraining then meta-learning for few-shot text classification (student abstract).  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 13773–13774.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * El-Kishky et al. (2020) Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020\.  CCAligned: A massive collection of cross-lingual web-document pairs.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020)_ , pages 5960–5969, Online. Association for Computational Linguistics.
  * Koehn and Hoang (2010) Philipp Koehn and Hieu Hoang. 2010.  Moses.  _Statistical Machine Translation System, User Manual and Code Guide_ , page 245.
  * Lee et al. (2021) Hung-yi Lee, Ngoc Thang Vu, and Shang-Wen Li. 2021.  Meta learning and its applications to natural language processing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Tutorial Abstracts_ , pages 15–20.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Livelo and Cheng (2018) Evan Dennison Livelo and Charibeth Cheng. 2018.  Intelligent dengue infoveillance using gated recurrent neural learning and cross-label frequencies.  In _2018 IEEE International Conference on Agents (ICA)_ , pages 2–7. IEEE.
  * Maudslay and Cotterell (2021) Rowan Hall Maudslay and Ryan Cotterell. 2021.  Do syntactic probes probe syntax? experiments with jabberwocky probing.  _arXiv preprint arXiv:2106.02559_.
  * Ortiz Suárez et al. (2019) Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019.  Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures.  Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim. Leibniz-Institut für Deutsche Sprache.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Schwenk et al. (2019) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019.  Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia.  _arXiv preprint arXiv:1907.05791_.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015.  Neural machine translation of rare words with subword units.  _arXiv preprint arXiv:1508.07909_.
  * Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _International Conference on Machine Learning_ , pages 4596–4604. PMLR.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in opus.  In _Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)_ , Istanbul, Turkey. European Language Resources Association (ELRA).