# PlasmoFAB: A Benchmark to Foster Machine Learning for Plasmodium falciparum
Protein Antigen Candidate Prediction

Jonas C. Ditz
Shared First Author
Methods in Medical Informatics
Department of Computer Science
University of Tübingen
Tübingen, Germany
<EMAIL_ADDRESS>
& Jacqueline Wistuba-Hamprecht
Shared First Author
Methods in Medical Informatics
Department of Computer Science
University of Tübingen
Tübingen, Germany
<EMAIL_ADDRESS>
Timo Maier
Computomics GmbH
Tübingen, Germany
& Rolf Fendel
Institute of Tropical Medicine
University Hospital Tübingen
Tübingen, Germany
<EMAIL_ADDRESS>
& Nico Pfeifer
Methods in Medical Informatics
Department of Computer Science
University of Tübingen
Tübingen, Germany
<EMAIL_ADDRESS>
& Bernhard Reuter
Methods in Medical Informatics
Department of Computer Science
University of Tübingen
Tübingen, Germany
<EMAIL_ADDRESS>

###### Abstract

Motivation: Machine learning methods can be used to support scientific
discovery in healthcare-related research fields. However, these methods can
only be reliably used if they can be trained on high-quality and curated
datasets. Currently, no such dataset for the exploration of Plasmodium
falciparum protein antigen candidates exists. The parasite Plasmodium
falciparum causes the infectious disease malaria. Thus, identifying potential
antigens is of utmost importance for the development of antimalarial drugs and
vaccines. Since exploring antigen candidates experimentally is an expensive
and time-consuming process, applying machine learning methods to support this
process has the potential to accelerate the development of drugs and vaccines,
which are needed for fighting and controlling malaria.
Results: We developed PlasmoFAB, a curated benchmark that can be used to train
machine learning methods for the exploration of Plasmodium falciparum protein
antigen candidates. We combined an extensive literature search with domain
expertise to create high-quality labels for Plasmodium falciparum specific
proteins that distinguish between antigen candidates and intracellular
proteins. Additionally, we used our benchmark to compare different well-known
prediction models and available protein localization prediction services on
the task of identifying protein antigen candidates. We show that available
general-purpose services are unable to provide sufficient performance on
identifying protein antigen candidates and are outperformed by our models that
were trained on this tailored data.
Availability: PlasmoFAB is publicly available on Zenodo with DOI
10.5281/zenodo.7433087. Furthermore, all scripts that were used in the
creation of PlasmoFAB and the training and evaluation of machine learning
models are open source and publicly available on GitHub here:
https://github.com/msmdev/PlasmoFAB.

##  1 Introduction

Malaria is a major health problem worldwide, causing more than 247 million
cases and approximately 619,000 deaths in 2021 (WHO et al.,, 2022). Almost all
malaria cases are caused by Plasmodium falciparum (Pf), predominantly in
Africa. Children, pregnant women, and malaria-naïve subjects are at high risk
to develop severe malaria (Riley and Stewart,, 2013; Wu,, 2019). Furthermore,
the increase in resistance to both insecticides that target the mosquito
vector and anti-malaria drugs, as well as the COVID-19 pandemic, led to an
increase of morbidity in several highly endemic countries in the past years
(WHO et al.,, 2021). Vaccines are very effective means in protecting against
infectious diseases as recently demonstrated in the case of COVID-19. The
RTS,S vaccine is the first malaria vaccine recommended by the World Health
Organization (WHO) for widespread use in children in endemic settings with a
substantial reduction of severe malaria cases, but limited reduction of
transmission of malaria(Rts,, 2015; Olotu et al.,, 2013). Besides this first
success in fighting severe malaria, there is still an urgent need to develop
an effective malaria vaccine that confers sterile protection and reduces
malaria transmission. However, developing an effective malaria vaccine is
still challenging due to the complex, multi-stage life-cycle of Pf, which is
genetically highly diverse and employs several immune evasion strategies. As a
result, our understanding of immune responses to Pf-specific antigens that
mediate naturally acquired or experimentally induced protection is incomplete.

More than 5,300 genes are expressed during the life-cycle of Pf (Obiero et
al.,, 2019). However, only a small subset of proteins that are expressed by Pf
is considered in current target candidate screening processes for an effective
malaria vaccine (Jagannathan and Kakuru,, 2022; Mordmüller et al.,, 2017).
Since most of the unused proteins have unknown function and experimental
validation remains costly and time-intensive, computational methods can be
used for pre-screening of proteins of interest. For example, trans-membrane
topology prediction is an established task in bioinformatics, where the aim is
to predict how and if a protein resides in the cell membrane, i.e., predict
the location and length of trans-membrane domains. The class of membrane
proteins is one of the most important classes of proteins for medical use.
About 25-30% of natural proteins reside in the cell membrane and are, thus,
often bound by antibodies during an immune reaction (Baker,, 2010). Another
class of relevant proteins for vaccine and drug development is the class of
exported proteins. Many of these fulfill important functions for parasite
survival. For example, certain proteins ensure that infected red blood cells
stick to the microvasculature, one of the factors that makes malaria a
potentially fatal disease (Tuteja,, 2007; Wahlgren et al.,, 2017). In recent
years, several scholars developed general-purpose models for sub-cellular
localization prediction and offered them as prediction services to be used by
the academic community (Krogh et al., 2001a, ; Hallgren et al.,, 2022; Almagro
Armenteros et al.,, 2017; Thumuluri et al.,, 2022; Käll et al.,, 2007). While
general-purpose models provide researchers with an easy-to-use solution for
performing prediction tasks, the lack of out-of-distribution generalization
capabilities of most general-purpose models leads to sub-optimal prediction
performances on novel datasets and misleading pre-screening results (Ye et
al.,, 2021). However, training supervised machine learning models for protein
antigen candidate prediction needs a sufficient amount of protein sequences
with high-quality labels. Currently, only a small fraction of publicly
available Pf protein sequences have high-quality labels, making the training
of models for identification of such antigens for vaccine and drug development
exponentially harder. With this work, we introduce the Plasmodium Falciparum-
specific Antigen candidate Benchmark (PlasmoFAB), a manually pre-processed and
curated dataset containing labeled protein sequences for Plasmodium falciparum
protein antigen candidate prediction.

This manuscript is structured as follows. We describe in detail the process of
creating PlasmoFAB including the used data sources, pre-processing, and
validation steps. Afterwards we present our experiments for predicting Pf
protein antigen candidates. Here we show the limitations of using established
tools and present approaches that provide solutions to overcome these
limitations. We conclude our work with a discussion about necessary actions
that have to be taken in order to further improve PlasmoFAB and, hence,
further foster the development of vaccines and drugs to control malaria.

##  2 PlasmoFAB: Plasmodium Falciparum-specific Protein Antigen Candidate
Benchmark

Figure 1: Schematic overview of the pre-processing steps for the creation of PlasmoFAB. The green workflow shows the pre-processing of the positive set, i.e., Pf proteins that are either extracellular or membrane-located which renders them eligible to be considered as antigen candidates. We used knowledge-driven techniques like algorithmic homology search, mass-spectrometry, string search, and validation by published literature to create sets of proteins containing antigen candidates. These sets were merged and duplicates were removed to create the positive set. The red workflow shows the pre-processing of the negative set. We combined enzymes validated by published literature with proteins that were assigned to be intracellular by a domain expert and UniProtKB/SwissProt (reviewed) to create the negative set. Table 1: Composition of the PlasmoFAB benchmark. The difference between the sum of sequences in each inclusion criterion and the total number of unique sequences in the positive set occurs due to the fact that some proteins fulfill more than one inclusion criterion. These proteins were not duplicated, resulting in the mismatch between the sum of proteins in each criterion and the total number of proteins in PlasmoFAB. Positive Set (Unique Total = 438) |  | Negative Set (Unique Total = 384)
---|---|---
Inclusion Criterion | Identified By | # Proteins |  | Inclusion Criterion | Identified By | # Proteins
IEDB epitope | BLAST match (high confidence) | 57 |  | Intracellular Proteins |  Combined string and literature search; Domain expertise | 384
IEDB epitope | BLAST match (medium confidence) | 60 |  |  |
PEXEL/HT motif |  PlasmoDB query | 265 |  |  |  |
Sporozoite proteins |  Mass-spectrometry (Swearingen et al.,, 2016) | 13 |  |  |  |
VSA family / Membrane proteins | Combined string and literature search | 302 |  |  |  |

The term supervised machine learning (SL or supervised ML) summarizes
techniques that correlate patterns within datasets to desired output
variables, i.e., labels for classification or continuous values for
regression. The foundation of using supervised ML methods for scientific
discovery in medical research are curated datasets with validated and
biologically meaningful output variables. Currently, there is no benchmark
that fulfills this prerequisite for the exploration of Pf protein antigen
candidates. With this manuscript, we tackle this fundamental obstacle for
supporting Pf protein antigen candidate exploration with supervised ML
techniques.

In the humoral immune response, the production of antibodies is an important
step in getting rid of pathogens. To enable this response chain, pathogen-
specific antigens activate B-cells and their differentiation into antibody
secreting plasma cells. Therefore, an antigen candidate has to be visible by
the humoral immune system of the host. Pf protein antigen candidates can be
considered visible, if they are present on the outside of infected host cells,
like surface proteins, transmembrane proteins, membrane-located proteins or
exported proteins. The VEuPathDB database PlasmoDB (Amos et al.,, 2022)
contains the complete genome of different Plasmodium species. The protein
sequences of the reference strain 3D7 of Pf, available in PlasmoDB, are the
data source for our curated benchmark. We only selected sequences with
experimental evidence, i.e., the corresponding Pf protein has to be referenced
in published work with a unique publication identifier. However, these
sequences do not have a sub-cellular location label. We combined an extensive
literature search with domain expertise to create high-quality sub-cellular
location labels that can be used to train ML models on the task of protein
antigen candidate prediction for Pf. In other words, PlasmoFAB’s positive set
contains Pf proteins that are accessible at the surface or the exterior of
infected cells, like surface proteins, transmembrane proteins, membrane-
located proteins, or exported proteins. On the other hand, PlasmoFAB’s
negative set contains intracellular proteins, which are needed by the parasite
to maintain the intracellular life cycle in hepatocytes or erythrocytes. The
executed pre-processing steps for the creation of PlasmoFAB are detailed in
the following section. A schematic overview of our pre-processing can be found
in Figure 1 and the basic statistics of PlasmoFAB are shown in Table 1.

###  2.1 IEDB Epitopes

An epitope is the part of an antigen that is recognized by the immune system
of a host organism, i.e., the binding site of an antibody. The Immune Epitope
Database (IEDB, https://www.iedb.org/, Vita et al., (2019)) contains sequences
of known epitopes. We used exact string matching and BLAST similarity matching
to compare Pf protein sequences with sequences contained in the IEDB. Proteins
that either contained exact matches of epitope sequences or a positive BLAST
hit with high or medium confidence score were labeled as antigen candidates
for our benchmark.

###  2.2 PEXEL/HT Motif

The majority of Pf proteins that are either exported into the extracellular
space by the parasite or integrated into the membrane of infected erythrocytes
contain a specific amino acid sequence called Plasmodium exported element
(PEXEL) or host targeting (HT) (Osborne et al.,, 2010; Jonsdottir et al.,,
2021). Therefore, the presence of this motif is a strong indicator of a
protein antigen candidate. PlasmoDB indicates the presence of the PEXEL/HT
motif within a sequence by a flag in one of its data fields. For PlasmoFAB, we
included all proteins with the PEXEL/HT motif as positive antigen candidates.

###  2.3 VSA families and known membrane proteins

Variant surface antigen (VSA) families describe proteins that are typically
located on cell surfaces. There are three known VSA families in the Pf genome:
Plasmodium falciparum erythrocyte membrane protein 1 (PfEMP1), repetitive
interspersed family (RIFIN), and sub-telomeric variable open reading frame
(STEVOR). The first family, PfEMP1, summarizes proteins that are expressed on
the surface of infected erythrocytes during the trophozoite and schizont stage
of the infection cycle. These proteins are mainly responsible for effective
evasion of immune responses (Wahlgren et al.,, 2017). Proteins belonging to
the RIFIN family are exported onto the cell surface of infected erythrocytes
as well. They mediate the sequestration of erythrocytes which results in
erythrocyte rosetting that further helps parasites to evade immune responses
and can block the blood flow (Wahlgren et al.,, 2017). Similar to the other
two VSA families, STEVOR proteins are also used by Pf parasites to evade host
immune responses. They play active roles in the trophozoite, schizont,
merozoite, and gametocyte stages of the infection cycle (Gardner et al.,,
2002; Wahlgren et al.,, 2017). Beside the members of VSA families, there are a
number of known membrane proteins. In the sporozoite stage, those include
thrombospondin-related anonymous protein (TRAP), also known as sporozoite
surface protein 2 (SSP2), apical membrane antigen 1 (AMA1), liver stage
antigen 1 (LSA1), and exported protein 1 (Exp-1), also known as
circumsporozoite-related antigen (CRA). Additionally, we included known
surface proteins that can be found in other stages of the infection cycle like
the family of monomeric serine-threonine protein kinases (FIKK, AK et al.,
(2021)), the helical intersperse sub-telomeric family of exported proteins
(PHIST, Tarr et al., (2014)), and the multigene family of cytoadherence linked
asexual gene (CLAG, Gupta et al., (2015)).

Each entry in PlasmoDB has a textual product description field containing
information about the sample in textual form. We performed a string search on
the textual product description field using the names of the VSA families as
search terms: ’PfEMP1’, ’RIFIN’, ’STEVOR’. For additional known membrane and
exported proteins, we did not only included the names but also descriptive
search terms since the textual product description field is not standardized.
The additional search terms were ’surface’, ’circumsporozoite’, ’membrane’,
’exported’, ’serine repeat antigen’, ’TRAP’, ’FIKK’, ’GLURP’, ’CLAG’, ’PHIST’,
and ’GPI-anchor’. However, the source and rationale behind the annotation in
PlasmoDB’s textual product description field are not always disclosed. To
ensure that only validated membrane and exported proteins are included in our
benchmark, we performed a literature search for each protein that was selected
by our string search and included only proteins with published experimental
evidence into our benchmark. To further enrich the set of known membrane
proteins, we added a list of sequences validated by the UniProtKB/SwissProt
(reviewed) database. This database contains high quality, manually annotated
proteins sequences (uniprot2023uniprot).

###  2.4 Sporozoite surface-exposed proteins

The authors in (Swearingen et al.,, 2016) used mass-spectrometry to identify
potential surface-exposed sporozoite proteins of Pf. They assigned priority
scores to each investigated protein ranging from 1 (high confidence) to 6 (low
confidence). We downloaded the publicly available data from (Swearingen et
al.,, 2016) and selected all proteins with a priority score from 1 to 3. We
used the unique transcript ID of these proteins to merge this information into
the PlasmoDB data table and included them into our benchmark as antigen
candidates.

###  2.5 Intracellular proteins

The pre-processing steps described above added positive samples, i.e., Pf
protein antigen candidates, to our benchmark. However, PlasmoFAB needs
negative samples, i.e., proteins that are not Pf protein antigen candidates,
to be usable for training of supervised ML methods. A model can only learn to
detect true protein antigen candiates, if a set of high-quality negative
samples, a so-called negative set, is available. Similar to the positive
samples, we curated the negative samples to ensure that only intracellular Pf
proteins are included into the negative set. Intracellular proteins can only
leave the cytoplasm in specific situations that do not reliably occur in the
infection cycle, like the burst of an infected erythrocyte or if macrophages
digest an infected erythrocyte and subsequently present an intracellular
protein as an antigen. However due to the unreliability of these incidents and
the fact that both can only occur late in the infection cycle, intracellular
proteins are not suitable as antibody targets. Enzymes constitute a subset of
intracellular proteins. We performed a string search with the term ’*ase’ on
PlasmoDB’s textual product description field and included all proteins with
published experimental evidence of being enzymes into the negative set of our
benchmark. While there is a small number of enzymes that are exported to the
cell membrane, we made sure to exclude all enzymes from the negative set for
which published experimental evidence of being membrane-located exists.
Furthermore, we included a list of known intracellular proteins compiled by a
domain expert and a list of intracellular proteins validated by
UniProtKB/SwissProt (reviewed).

##  3 Utilizing Machine Learning for Plasmodium Falciparum Protein Antigen
Candidate Exploration

Manually exploring Pf proteins for potential antigen candidates is a time
consuming and expensive procedure. With the help of our curated benchmark, we
can utilize supervised ML to accelerate the process with a pre-screening of
potential proteins that reduces the required workload of researchers in the
laboratory. The usefulness of such a pre-screening process highly depends on
the accuracy that prediction models are able to achieve. We compared the
performance of several ML approaches that are commonly used for textual data,
especially for biological sequences. The used methods include a kernelized
support vector machine (SVM) utilizing the oligo kernel (Meinicke et al.,,
2004), the protein language model embedding ESM-1b (Rives et al.,, 2021)
combined with a logistic regression (LR) classifier as well as an SVM, and the
protein language model embedding ProtT5 (Elnaggar et al.,, 2020), which we
also combined with an LR classifier and an SVM. Furthermore, we also tested
the performance of existing protein localization prediction tools on the Pf
protein antigen candidate prediction task. These tools are publicly offered as
a service for protein localization prediction tasks and included TMHMM (Krogh
et al., 2001b, ), DeepTMHMM (Hallgren et al.,, 2022), DeepLoc 1.0 (Almagro
Armenteros et al.,, 2017), DeepLoc 2.0 (Thumuluri et al.,, 2022), and Phobius
(Käll et al.,, 2007). To ensure a fair comparison between pre-trained
prediction services and our self-trained models, we defined a test set that
was separated from the training data before model training was performed. We
used MMseqs2 (Zimmermann et al.,, 2018; Gabler et al.,, 2020) to ensure that
each sequence in the test set had at most 30% homology to sequences in the
training set, which is the default setting of MMseqs2. The test set consists
of 60 sequences (30 antigen targets and 30 intracellular proteins) with the
remaining 788 sequences in PlasmoFAB used as a training set. All performance
measures shown in this section are computed on the test set.

To assess the performance of each method, we used three performance measures
that are widely used in computational biology due to their ability to handle
imbalanced data with relative ease. First, we used balanced accuracy, which
has different definitions in literature. We used the arithmetic mean of
sensitivity and specificity (Pedregosa et al.,, 2011) given by

$\text{Acc}_{\text{bal}}=\frac{1}{2}\left(\frac{\text{TP}}{\text{TP}+\text{FN}}+\frac{\text{TN}}{\text{TN}+\text{FP}}\right),$
(1)

where TP is the number of correctly predicted protein antigen candidates
(i.e., true positives), FP is the number of wrongly predicted protein antigen
candidates (i.e., false positives), TN is the number of correctly predicted
intracellular proteins (i.e., true negatives), and FN is the number of wrongly
predicted intracellular proteins (i.e., false negatives). Additionally, we
used the F1-score that is the harmonic mean of precision and recall (Taha and
Hanbury,, 2015) given by

$\text{F}_{1}=\frac{2\text{TP}}{2\text{TP}+\text{FP}+\text{FN}},$ (2)

with TP, FP, and FN defined in the same way as above. Finally, we also
included the Matthews correlation coefficient (MCC, Chicco and Jurman,
(2020)), which is widely recognized as one of the most reliable performance
measures for binary classification on biological data. The MCC is defined as

$\text{MCC}=\frac{\text{TP}\cdot\text{TN}-\text{FP}\cdot\text{FN}}{\sqrt{(\text{TP}+\text{FP})(\text{TP}+\text{FN})(\text{TN}+\text{FP})(\text{TN}+\text{FN})}}.$
(3)

Again, the definition of TP, FP, TN, and FN are the same as above. Since the
classes in PlasmoFAB are balanced, we also report precision, recall, and
specificity to provide a quick overview over the distribution of FN and FP for
the predictions of the tested models.

###  3.1 Using PlasmoFAB’s training sequences for model training

Hyperparameter optimization and model selection was exclusively performed on
PlasmoFAB’s training sequences to avoid information leakage from the test
sequences. As a baseline model, we trained a kernelized SVM utilizing the
oligo kernel, a kernel function that was specifically developed for biological
sequences (Meinicke et al.,, 2004). This kernel computes the similarity of two
sequences based on $k$-mer occurrence with a tunable degree of positional
uncertainty. The SVM that was trained for Pf protein antigen candidate
prediction had three hyperparameters that needed to be optimized: the $k$-mer
length, the positional uncertainty parameter $\sigma$, and the regularization
parameter $C_{\text{SVM}}$. We performed a grid search utilizing repeated
nested cross-validation to optimize all three hyperparameters. The resulting
choices were $k=1$, $\sigma=18$, and $C_{\text{SVM}}=0.001$.

Additionally, we used two more complex language embedding models that are
commonly used for biological sequences: ESM-1b and ProtT5. The first, ESM-1b,
is a pre-trained transformer model (Rives et al.,, 2021), which is offered as
a feature generator for downstream prediction models. It was developed to be
used on biological sequences. ESM-1b follows the self-supervised bidirectional
encoder representation from transformation (BERT) pre-training procedure. This
language model is a transformer architecture with 33 layers and utilizes self-
attention with 20 attention heads. The resulting features have a
dimensionality of 1280 with a token context size of 1024. ESM-1b was trained
on sequence clusters derived from the UniProt database (Bairoch et al.,,
2005). We refer the interested reader to the original publication for all
technical details about ESM-1b. The token context size together with a
positional encoding of fixed length limits input sequences to a maximum of
1024 characters. Since there is a significant number of sequences in PlasmoFAB
that exceed this character limit, we followed published recommendations to cut
the middle part of sequences that exceed the 1024 character limitation
(Thumuluri et al.,, 2022) to be able to use ESM-1b on our benchmark. In total,
261 sequences were affected by this cutting procedure. The computed feature
embeddings were used as inputs for the two tested downstream prediction
models, LR and SVM. Again, we exclusively optimized the regularization
parameters $C_{\text{LR}}$ and $C_{\text{SVM}}$, respectively. After
performing the grid search, the optimal parameter choices were
$C_{\text{LR}}=0.15$ and $C_{\text{SVM}}=20$.

The second language embedding that we used was ProtT5-XL-UniRef50 (ProtT5,
Elnaggar et al., (2020)). This transformer model, based on the language model
T5 (Raffel et al.,, 2020), is specifically developed for biological data and
prediction tasks. Similar to ESM-1b, ProtT5 acts as a feature generator for
downstream prediction models. In contrast to other language models, ProtT5
follows an encoder-decoder approach and uses a simplified BERT training
objective. The architecture employs 24 layers and also utilizes self-attention
with 32 attention heads. ProtT5 has an embedding dimensionality of 1024. Since
ProtT5 does not use a positional encoding of fixed length but learns a
positional encoding for each attention head, the length of input sequences is
not limited in theory. ProtT5 was first pre-trained on the BFD database
(Steinegger et al.,, 2019) and fine-tuned on UniRef50 (Suzek et al.,, 2015).
We refer the interested reader to the original publication for all technical
details about ProtT5. Although sequence length is not limited when using
ProtT5, finite computation power limits the usable sequence length in
practice. With the computing resources available to us, an Nvidia Tesla V100
with 32GB RAM, the maximal usable sequence length was 6000 residues. Longer
sequences were shortened in the same way we shortened sequences for ESM-1b.
Five sequences in PlasmoFAB were affected by this reduction of sequence
length. Again, we used the feature embedding as inputs for the two downstream
prediction models, LR and SVM, and optimized the regularization parameter via
a grid search. The optimal parameters were $C_{\text{LR}}=0.2$ and
$C_{\text{SVM}}=2.0$.

###  3.2 Evaluating prediction models on PlasmoFAB’s test sequences

Table 2: Performance of trained prediction models and prediction services on PlasmoFAB’s test set. We trained different models on PlasmoFAB’s training set including a support vector machine utilizing the oligo kernel ($\text{SVM}_{\text{oligo}}$), a combination of the a linear regression with either ESM1b or ProtT5 language model embedding ($\text{LR}_{\text{ESM1b}}$ and $\text{LR}_{\text{ProtT5}}$), and a support vector machine combined with either ESM1b or ProtT5 language model embedding ($\text{SVM}_{\text{ESM1b}}$ and $\text{SVM}_{\text{ProtT5}}$). Furthermore, we used publicly available, pre-trained prediction services on PlasmoFAB’s test set. These services include Phobius, TMHMM, DeepTMHMM, Deeploc 1.0, and Deeploc 2.0. Model | MCC | F1 | Bal. Acc. | Precision | Recall | Specificity
---|---|---|---|---|---|---
$\text{SVM}_{\text{oligo}}$ | 0.3145 | 0.5882 | 0.6500 | 0.7143 | 0.5000 | 0.8000
$\text{LR}_{\text{ESM1b}}$ | 0.7071 | 0.8000 | 0.8333 | 1.0000 | 0.6667 | 1.0000
$\text{SVM}_{\text{ESM1b}}$ | 0.7071 | 0.8000 | 0.8333 | 1.0000 | 0.6667 | 1.0000
$\text{LR}_{\text{ProtT5}}$ | 0.7338 | 0.8235 | 0.8500 | 1.0000 | 0.7000 | 1.0000
$\text{SVM}_{\text{ProtT5}}$ | 0.6917 | 0.8077 | 0.8333 | 0.9545 | 0.7000 | 0.9666
DeepTMHMM | 0.4395 | 0.6909 | 0.7167 | 0.7600 | 0.6333 | 0.8001
DeepLoc 2.0 | 0.4009 | 0.7079 | 0.7009 | 0.6000 | 0.6923 | 0.7095
DeepLoc 1.0 | 0.2691 | 0.6071 | 0.6357 | 0.5667 | 0.6538 | 0.6176
TMHMM | 0.3015 | 0.6316 | 0.6500 | 0.6667 | 0.6000 | 0.7000
Phobius | 0.2722 | 0.6667 | 0.6333 | 0.6111 | 0.7333 | 0.5333

Table 2 shows the performance of all models on PlasmoFAB’s test set. The
models trained by ourselves can be directly applied to the test set. Since the
publicly available prediction services do not always provide a binary output,
we converted the prediction output for each service into a binary label. TMHMM
and Phobius provide topology predictions for input sequences and we assigned a
positive label to all samples with at least one predicted trans-membrane helix
or at least one predicted extracellular region. Otherwise the sample was
assigned a negative label. DeepTMHMM refines the prediction of TMHMM by
providing a label for each residue in an input sample. For the DeepTMHMM
output, we assigned a positive label to all samples with residues that had the
membrane domain label (’M’) assigned. Furthermore, a positive label was
assigned to samples where DeepTMHMM predicted the outside cell label (’O’) for
all residues. If none of these conditions was fulfilled, the sample was
assigned a negative label. DeepLoc 1.0 and 2.0 are tools for subcellular
localization prediction and, hence, offer a multi-label output. Each label
corresponds to a different subcellular localization. We used the top predicted
label for each input sample. If this label was ’cell membrane’ or
’extracellular’, the sample was assigned a positive label, otherwise a
negative label was assigned.

Our results show that models directly trained on PlasmoFAB training set
clearly outperform the available prediction services. The best performance was
achieved by combining ProtT5 feature embedding with logistic regression. None
of the tested prediction services was able to achieve a comparable performance
to the specialized models.

##  4 Discussion

Computational antigen pre-screening with machine learning methods can
drastically reduce time- and resource-consuming experimental exploration
procedures and, thereby, accelerate development of drugs and vaccines.
However, these computational pre-screening methods heavily depend on high-
quality data to produce reliable results. In this work, we take important
steps towards utilizing computational pre-screening for Malaria drug and
vaccine development by providing PlasmoFAB, a benchmark that consists of
Plasmodium falciparum-specific protein sequences with curated labels that
distinguish between protein antigen candidates and intracellular proteins.

Experimental validation is the gold standard to determine subcellular
localization labels for proteins. We ensured that each label in PlasmoFAB
achieves this gold standard or, if experimental validation is not feasible,
comes as close to the gold standard as possible. As detailed in section 2, the
biggest subgroup of proteins that were assigned as antigen candidates was the
group of VSA family members and known membrane proteins. We performed an
exhaustive literature search and only included proteins into this subgroup for
which published experimental evidence exists. Other subsets with
experimentally validated labels are sporozoite proteins and proteins that
contain the PEXEL/HT motif. Sporozoite proteins were validated by mass-
spectrometry (Swearingen et al.,, 2016). PEXEL/HT motif occurrence is a
property of the protein sequence. This property is experimentally validated
since PlasmoFAB only includes experimentally validated protein sequences.
Furthermore, there is experimental evidence that Pf parasites use the PEXEL/HT
motif to export proteins (Jonsdottir et al.,, 2021; Osborne et al.,, 2010).
This supports our decision to include PEXEL/HT motif occurrence as an
indication of protein antigen candidates. The last remaining subgroup in
PlasmoFAB’s positive set are proteins with known epitopes. IEDB only includes
epitopes that are experimentally validated and we used BLAST to perform
similarity matching between IEDB entries and Pf protein sequences. Although
BLAST does not fulfill the gold standard of experimental validation, it is
widely considered as the gold standard for sequence similarity matching. By
restricting ourselves to BLAST matches with high or medium confidence, we
ensured that the reduction in label quality of proteins in this subgroup is
minimized. PlasmoFAB’s negative set contains two groups of proteins: enzymes
and intracellular proteins. We performed an exhaustive literature search to
ensure that all included enzymes have experimental evidence of being
intracellular. We excluded enzymes, if there is at least one publication with
experimental evidence that suggests that the enzyme is being exported outside
the cell. The other subgroup, intracellular proteins, were classified by a
domain expert. While this does not fulfill the gold standard of experimental
validation, we ensured to minimize the reduction in label quality by using
domain expertise.

PlasmoFAB uses data that belongs to the Plasmodium falciparum strain 3D7. The
genome of this specific strain of the Pf parasite was the first to be
published by Gardner and colleagues in 2002 (Gardner et al.,, 2002). It is
still today one of the most important information sources for malaria research
(Olotu et al.,, 2013; Rts,, 2015; Mordmüller et al.,, 2017; Jagannathan and
Kakuru,, 2022). Therefore, we made the decision to concentrate on Pf strain
3D7 for the first version of PlasmoFAB. For future work, we want to further
refine PlasmoFAB by deriving high-quality labels for protein sequences of
other Pf strains in order to incorporate as much information about Pf protein
antigen candidates as possible into our benchmark.

One potentially surprising result is the sub-optimal performance of publicly
available prediction services, like DeepTMHMM or DeepLoc 2.0, even though
these services are relatively new and show impressive performance capabilities
in their respective manuscripts. Our results do not provide evidence that the
published performance capabilities of these models are overly optimistic or
that they should not be used in general. On the contrary, we would like to
emphasize that prediction services provide a fast and easy-to-use way for
researchers without a strong background in machine learning to utilize
prediction models in their research or the possibility to use prediction
models even if not enough data for model training is available. However, our
results highlight one common problem of general purpose models: their lack of
out-of-distribution generalization (Ye et al.,, 2021). Models learn certain
aspects of the training data’s distribution and allow trained models to
achieve high prediction performance of unseen data as long as these data
points came from the same distribution. However, if those unseen data points
came from a different distribution, there is no guarantee that the model will
be able to reliably make predictions on the new data. We see this out-of-
distribution generalization issue in the relatively poor performance of the
used prediction services. Since the Pf proteins are likely to be differently
distributed than the proteins used to train the prediction services, these
services perform poorly when applied to our test set. This result supports our
claim that providing curated datasets with high-quality labels for model
training is essential for maximising the potential of computational prediction
methods on biological prediction tasks like the pre-screening of Pf protein
antigen candidates. Therefore, our proposed PlasmoFAB benchmark offers a
solution to one fundamental obstacle in utilizing computational prediction
methods in the development process of drugs and vaccines against malaria.

One goal of developing PlasmoFAB was to provide the malaria research community
with a tool to utilize machine learning in protein antigen exploration
processes. However, the potential target user group of PlasmoFAB can only
benefit from the data if it fulfils two basic requirements. First, potential
users have to be enabled to reliably find, access, and reuse data. And second,
potential users have to be able to make an informed decision whether the data
is applicable for their specific problem. We tackle the first problem by
making PlasmoFAB publicly available via Zenodo, which is a platform by
researchers for researchers that aims to support open science. By uploading
our dataset to Zenodo we ensure that the FAIR principles (Wilkinson et al.,,
2016) are taken into account. Additionally, we release PlasmoFAB in form of
comma-separated values (CSV) files. This file format is universally used in
different research communities and should maximize the number of researchers
that can use our dataset. Furthermore, we created a datasheet for PlasmoFAB as
described in (Gebru et al.,, 2021). With this datasheet, we provide
information about the motivation behind creating PlasmoFAB, the creation
process, the assumptions made, and applicable use cases. Users who are
interested in using PlasmoFAB can use the datasheet to make an informed
decision about the applicability.

##  5 Conclusion

With this work, we introduce PlasmoFAB, a new and carefully curated benchmark
for the training of models for Plasmodium falciparum protein antigen candidate
prediction. The benchmark was created by manually validating extracellular,
surface-exposed, and intracellular Pf proteins to ensure high-quality labels
for every sample in the dataset. Such a curated benchmark is an important
prerequisite to incorporate learning models into pre-screening protocols for
protein antigen candidates.

We furthermore compared commonly used prediction models with publicly
available prediction services on the Pf protein antigen candidate prediction
task. Our results show the limitations of existing prediction services, which
are vastly outperformed by simpler prediction models that are specifically
trained for Pf protein antigen candidate prediction.

We are confident that our contribution provides a tool that can be used to
help the research community to explore the vast number of Plasmodium
falciparum proteins with unknown functionality and identify new targets for
drugs and vaccines against malaria.

## Acknowledgments and Disclosure of Funding

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under Germany’s Excellence Strategy – EXC number 2064/1 – Project
number 390727645. This research was supported by the German Federal Ministry
of Education and Research (BMBF) project ’Training Center Machine Learning,
Tübingen’ with grant number 01|S17054. This work was supported by the German
Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ:
01IS18039A.

## References

  * AK et al., (2021) AK, D., Shrivastava, D., Sahasrabuddhe, A. A., Habib, S., and Trivedi, V. (2021).  Plasmodium falciparum fikk9. 1 is a monomeric serine-threonine protein kinase with features to exploit as a drug target.  Chemical Biology & Drug Design.
  * Almagro Armenteros et al., (2017) Almagro Armenteros, J. J., Sønderby, C. K., Sønderby, S. K., Nielsen, H., and Winther, O. (2017).  Deeploc: prediction of protein subcellular localization using deep learning.  Bioinformatics, 33(21):3387–3395.
  * Amos et al., (2022) Amos, B., Aurrecoechea, C., Barba, M., Barreto, A., Basenko, E. Y., Belnap, R., Blevins, A. S., Böhme, U., Brestelli, J., Brunk, B. P., et al. (2022).  Veupathdb: the eukaryotic pathogen, vector and host bioinformatics resource center.  Nucleic Acids Research, 50(D1):D898–D911.
  * Bairoch et al., (2005) Bairoch, A., Apweiler, R., Wu, C. H., Barker, W. C., Boeckmann, B., Ferro, S., Gasteiger, E., Huang, H., Lopez, R., Magrane, M., et al. (2005).  The universal protein resource (uniprot).  Nucleic acids research, 33(suppl_1):D154–D159.
  * Baker, (2010) Baker, M. (2010).  Making membrane proteins for structures: a trillion tiny tweaks.  Nature methods, 7(6):429–434.
  * Chicco and Jurman, (2020) Chicco, D. and Jurman, G. (2020).  The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation.  BMC genomics, 21(1):1–13.
  * Elnaggar et al., (2020) Elnaggar, A., Heinzinger, M., Dallago, C., Rihawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2020).  Prottrans: towards cracking the language of life’s code through self-supervised deep learning and high performance computing.  arXiv preprint arXiv:2007.06225.
  * Gabler et al., (2020) Gabler, F., Nam, S.-Z., Till, S., Mirdita, M., Steinegger, M., Söding, J., Lupas, A. N., and Alva, V. (2020).  Protein sequence analysis using the mpi bioinformatics toolkit.  Current Protocols in Bioinformatics, 72(1):e108.
  * Gardner et al., (2002) Gardner, M. J., Hall, N., Fung, E., White, O., Berriman, M., Hyman, R. W., Carlton, J. M., Pain, A., Nelson, K. E., Bowman, S., et al. (2002).  Genome sequence of the human malaria parasite plasmodium falciparum.  Nature, 419(6906):498–511.
  * Gebru et al., (2021) Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D., and Crawford, K. (2021).  Datasheets for datasets.  Communications of the ACM, 64(12):86–92.
  * Gupta et al., (2015) Gupta, A., Thiruvengadam, G., and Desai, S. A. (2015).  The conserved clag multigene family of malaria parasites: essential roles in host–pathogen interaction.  Drug Resistance Updates, 18:47–54.
  * Hallgren et al., (2022) Hallgren, J., Tsirigos, K. D., Pedersen, M. D., Armenteros, J. J. A., Marcatili, P., Nielsen, H., Krogh, A., and Winther, O. (2022).  Deeptmhmm predicts alpha and beta transmembrane proteins using deep neural networks.  bioRxiv.
  * Jagannathan and Kakuru, (2022) Jagannathan, P. and Kakuru, A. (2022).  Malaria in 2022: Increasing challenges, cautious optimism.  Nature communications, 13(1):1–3.
  * Jonsdottir et al., (2021) Jonsdottir, T. K., Gabriela, M., Crabb, B. S., de Koning-Ward, T. F., and Gilson, P. R. (2021).  Defining the essential exportome of the malaria parasite.  Trends in Parasitology, 37(7):664–675.
  * Käll et al., (2007) Käll, L., Krogh, A., and Sonnhammer, E. L. (2007).  Advantages of combined transmembrane topology and signal peptide prediction—the phobius web server.  Nucleic acids research, 35(suppl_2):W429–W432.
  * (16) Krogh, A., Larsson, B., Von Heijne, G., and Sonnhammer, E. L. (2001a).  Predicting transmembrane protein topology with a hidden markov model: application to complete genomes.  Journal of molecular biology, 305(3):567–580.
  * (17) Krogh, A., Larsson, B., Von Heijne, G., and Sonnhammer, E. L. (2001b).  Predicting transmembrane protein topology with a hidden markov model: application to complete genomes.  Journal of molecular biology, 305(3):567–580.
  * Meinicke et al., (2004) Meinicke, P., Tech, M., Morgenstern, B., and Merkl, R. (2004).  Oligo kernels for datamining on biological sequences: a case study on prokaryotic translation initiation sites.  BMC bioinformatics, 5(1):1–14.
  * Mordmüller et al., (2017) Mordmüller, B., Surat, G., Lagler, H., Chakravarty, S., Ishizuka, A. S., Lalremruata, A., Gmeiner, M., Campo, J. J., Esen, M., Ruben, A. J., et al. (2017).  Sterile protection against human malaria by chemoattenuated pfspz vaccine.  Nature, 542(7642):445–449.
  * Obiero et al., (2019) Obiero, J. M., Campo, J. J., Scholzen, A., Randall, A., Bijker, E. M., Roestenberg, M., Hermsen, C. C., Teng, A., Jain, A., Davies, D. H., et al. (2019).  Antibody biomarkers associated with sterile protection induced by controlled human malaria infection under chloroquine prophylaxis.  Msphere, 4(1):e00027–19.
  * Olotu et al., (2013) Olotu, A., Fegan, G., Wambua, J., Nyangweso, G., Awuondo, K. O., Leach, A., Lievens, M., Leboulleux, D., Njuguna, P., Peshu, N., et al. (2013).  Four-year efficacy of rts, s/as01e and its interaction with malaria exposure.  New England Journal of Medicine, 368(12):1111–1120.
  * Osborne et al., (2010) Osborne, A. R., Speicher, K. D., Tamez, P. A., Bhattacharjee, S., Speicher, D. W., and Haldar, K. (2010).  The host targeting motif in exported plasmodium proteins is cleaved in the parasite endoplasmic reticulum.  Molecular and biochemical parasitology, 171(1):25–31.
  * Pedregosa et al., (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011).  Scikit-learn: Machine learning in python.  the Journal of machine Learning research, 12:2825–2830.
  * Raffel et al., (2020) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P. J., et al. (2020).  Exploring the limits of transfer learning with a unified text-to-text transformer.  J. Mach. Learn. Res., 21(140):1–67.
  * Riley and Stewart, (2013) Riley, E. M. and Stewart, V. A. (2013).  Immune mechanisms in malaria: new insights in vaccine development.  Nature medicine, 19(2):168.
  * Rives et al., (2021) Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021).  Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences.  Proceedings of the National Academy of Sciences, 118(15):e2016239118.
  * Rts, (2015) Rts, S. (2015).  Efficacy and safety of rts, s/as01 malaria vaccine with or without a booster dose in infants and children in africa: final results of a phase 3, individually randomised, controlled trial.  The Lancet, 386(9988):31–45.
  * Steinegger et al., (2019) Steinegger, M., Mirdita, M., and Söding, J. (2019).  Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold.  Nature methods, 16(7):603–606.
  * Suzek et al., (2015) Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., and Consortium, U. (2015).  Uniref clusters: a comprehensive and scalable alternative for improving sequence similarity searches.  Bioinformatics, 31(6):926–932.
  * Swearingen et al., (2016) Swearingen, K. E., Lindner, S. E., Shi, L., Shears, M. J., Harupa, A., Hopp, C. S., Vaughan, A. M., Springer, T. A., Moritz, R. L., Kappe, S. H., et al. (2016).  Interrogating the plasmodium sporozoite surface: identification of surface-exposed proteins and demonstration of glycosylation on csp and trap by mass spectrometry-based proteomics.  PLoS pathogens, 12(4):e1005606.
  * Taha and Hanbury, (2015) Taha, A. A. and Hanbury, A. (2015).  Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool.  BMC medical imaging, 15(1):1–28.
  * Tarr et al., (2014) Tarr, S. J., Moon, R. W., Hardege, I., and Osborne, A. R. (2014).  A conserved domain targets exported phistb family proteins to the periphery of plasmodium infected erythrocytes.  Molecular and biochemical parasitology, 196(1):29–40.
  * Thumuluri et al., (2022) Thumuluri, V., Almagro Armenteros, J. J., Johansen, A. R., Nielsen, H., and Winther, O. (2022).  Deeploc 2.0: multi-label subcellular localization prediction using protein language models.  Nucleic Acids Research.
  * Tuteja, (2007) Tuteja, R. (2007).  Malaria- an overview.  The FEBS journal, 274(18):4670–4679.
  * Vita et al., (2019) Vita, R., Mahajan, S., Overton, J. A., Dhanda, S. K., Martini, S., Cantrell, J. R., Wheeler, D. K., Sette, A., and Peters, B. (2019).  The immune epitope database (iedb): 2018 update.  Nucleic acids research, 47(D1):D339–D343.
  * Wahlgren et al., (2017) Wahlgren, M., Goel, S., and Akhouri, R. R. (2017).  Variant surface antigens of plasmodium falciparum and their roles in severe malaria.  Nature Reviews Microbiology, 15(8):479–491.
  * WHO et al., (2021) WHO et al. (2021).  World malaria report 2021.  World Health Organization.
  * WHO et al., (2022) WHO et al. (2022).  World malaria report 2022.  World Health Organization.
  * Wilkinson et al., (2016) Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016).  The fair guiding principles for scientific data management and stewardship.  Scientific data, 3(1):1–9.
  * Wu, (2019) Wu, H. M. (2019).  Evaluation of the sick returned traveler.  In Seminars in diagnostic pathology, volume 36, pages 197–202. Elsevier.
  * Ye et al., (2021) Ye, H., Xie, C., Cai, T., Li, R., Li, Z., and Wang, L. (2021).  Towards a theoretical framework of out-of-distribution generalization.  Advances in Neural Information Processing Systems, 34:23519–23531.
  * Zimmermann et al., (2018) Zimmermann, L., Stephens, A., Nam, S.-Z., Rau, D., Kübler, J., Lozajic, M., Gabler, F., Söding, J., Lupas, A. N., and Alva, V. (2018).  A completely reimplemented mpi bioinformatics toolkit with a new hhpred server at its core.  Journal of molecular biology, 430(15):2237–2243.


# Multi-model Ensemble Learning Method for Human Expression Recognition

Jun Yu
University of Science and Technology of China
Hefei, China
<EMAIL_ADDRESS>Zhongpeng Cai
University of Science and Technology of China
Hefei, China
<EMAIL_ADDRESS>Peng He
University of Science and Technology of China
Hefei, China
<EMAIL_ADDRESS>Guochen Xie
University of Science and Technology of China
Hefei, China
<EMAIL_ADDRESS>Qiang Ling
University of Science and Technology of China
Hefei, China
<EMAIL_ADDRESS>
###### Abstract

Analysis of human affect plays a vital role in human-computer interaction
(HCI) systems. Due to the difficulty in capturing large amounts of real-life
data, most of the current methods have mainly focused on controlled
environments, which limit their application scenarios. To tackle this problem,
we propose our solution based on the ensemble learning method. Specifically,
we formulate the problem as a classification task, and then train several
expression classification models with different types of backbones—ResNet,
EfficientNet and InceptionNet. After that, the outputs of several models are
fused via model ensemble method to predict the final results. Moreover, we
introduce the multi-fold ensemble method to train and ensemble several models
with the same architecture but different data distributions to enhance the
performance of our solution. We conduct many experiments on the AffWild2
dataset of the ABAW2022 Challenge, and the results demonstrate the
effectiveness of our solution.

##  1 Introduction

The analysis of human affect, which aims to transfer the understanding of
human feelings to computers, has been a topic of major research in human-
computer interaction (HCI) systems. Human affective states can be decided by a
wide range of sources in three main categories: visual, auditory and
biological signals. Among these three categories, visual information is the
most important data in analyzing human affect. Ekman [19] was the first to
systematically study human facial expressions who introduced six basic
emotions, i.e., anger, disgust, fear, happiness, sadness and surprise.
Furthermore, facial expressions are related to specific movements of facial
muscles, called Action Units (AUs). The Facial Action Coding System (FACS) was
developed, in which facial changes are described in terms of AUs [2].

The expression recognition task has been under exploration for a long time,
and many effective methods are proposed with remarkable performance. However,
most of them only concentrate on the controlled environments [7, 25, 26, 6]
and few of them propose effective solution for the in-the wild scenarios. And
in fact, there are few datasets collected in the wild environments. To deal
with these problems, the ABAW2022 challenge [16, 12, 15, 14, 11, 13, 24] is
held to promote the development of the in-the-wild expression recognition task
with a new video dataset AffWild2 [10]. The dataset includes 2,603,921 frames
from 548 different videos. Each frame is annoted with one of the 8
expressions(6 basic expressions, neutral and others). The subjects in the
daaset are 431 in total(265 male and 166 female). All the videos are captured
in the wild, and is thus the dataset is suitable for the corresponding
research.

Figure 1: The overall pipeline of our method. It consists of two components:
(a) Fusion block within the fold, (b) Fusion across the folds.

In this work, we propose our solution based on the ensemble learning to deal
with the problems of inconspicuous facial features and blurred expression
classification boundaries in AffWild2 dataset. In order to fully extract
facial features from each frames of the videos and improve the sensitivity of
recognition to different emotional expressions, we formulate the problem as a
classification task and train several human expression classification models
with different types of backbones—ResNet [4], EfficientNet [23] and
InceptionNet [22] to get the outputs, which are fused via model ensemble
method to predict the fused results. Considering the impact of different data
distributions on model training, we introduce the multi-fold ensemble method
to train and ensemble several models with the same architecture but different
data distributions to enhance the performance of our solution.

In sum, the contributions of this work are two-folds:

  * •

We ensemble three models with different architectures to improve the
efficiency of our proposed model in expression recognition task. Meanwhile,
focal loss [17] is introduced to deal with the imbalanced problem in the
AffWild2 dataset.

  * •

We split the whole training set into five folds and then train several models
of the same architecture but with different folds. After that, we fuse the
output of models trained with each fold via model ensemble method to further
enhance the performance.

##  2 Related Works

Ensemble learning is an imperative study in machine learning [20]. Over the
previous years, ensemble learning has drawn considerable attention in the
field of artificial intelligence, pattern recognition, machine learning,
neural network and data mining. The ensemble method can be roughly divided
into two categories, homogenous ensemble and heterogeneous ensemble. The
homogenous ensemble means ensemble the models trained with the same algorithm,
while the heterogeneous ensemble always ensemble models different in
architectures or training methods. Huang et.al [5] proposes the ensemble
learning framework on face recognition tasks and concludes that ensemble
method performs better than traditional method. Giacinto et.al [3] introduced
ensemble approaches for intrusion detection system and came to conclusion that
the ensembled method has more excellent detection ability for known attacks
than single model.

The benefits of the ensemble learning is that they are capable of boosting the
weak learners so that overall accuracy of the learning algorithm on training
data can be enhanced. Inspired by the fact, we constructs several human
expressions classifiers and fuse their outputs so that the overall variance
can be reduced and the evaluation result can be improved compared to the
single classifier.

##  3 Proposed Method

In this section, we introduce our method for the 2nd track (Expression
Classification Challenge) of the 3rd Workshop and Competition on Affective
Behavior Analysis IN-THE-WILD(ABAW). The overall pipeline is illustrated in
Figure 1. The entire framework consists of two components: (a) Fusion block
within the fold, (b) Fusion across the folds.

###  3.1 Preprocessing

In AffWild2 dataset, all videos are splitted into independent frames [9] and
it provides the cropped and aligned images that are extracted from the videos.
Each training image is resized to 112 $\times$ 112 and normalized before
inputting into the models, and then we use the methods of
RandomHorizontalFlipping and ColorJitter to enlarge our dataset.

Table 1: Five folds training results (F1-score) of three models. Model |  Fold 1 |  Fold 2 |  Fold 3 |  Fold 4 |  Fold 5
---|---|---|---|---|---
ResNet50 |  0.317 |  0.232 |  0.227 |  0.266 |  0.271
EfficientNet-b0 |  0.302 |  0.270 |  0.286 |  0.307 |  0.314
InceptionNet-v1 |  0.275 |  0.242 |  0.236 |  0.297 |  0.300
Table 2: Five folds training results (F1-score) of three model fusion methods.(InceptionNet-v1 : ResNet50 : EfficientNet-b0) Fusion Method | Fold 1 | Fold 2 | Fold 3 | Fold 4 | Fold 5
---|---|---|---|---|---
Fusion 1 | 0.331 (1:1:1) | 0.265 (1:1:1) | 0.246 (1:1:1) | 0.313 (1:1:1) | 0.326 (1:1:1)
Fusion 2 | 0.323 (0.5:1.1:0.5) | 0.274 (0.6:0.4:1.2) | 0.280 (0.5:0.5:2) | 0.325 (0.6:0.4:1.1) | 0.331 (0.6:0.4:1.2)
Fusion 3 | 0.324 (0.4:1:0.6) | 0.278 (0.6:0:0.7) | 0.285 (0.5:0:2) | 0.328 (0.4:0:0.6) | 0.326 (0.5:0:1.1)

###  3.2 Model Structure

As it shown in Figure 1, the method we proposed consists of two components:
(a) Fusion block within the fold, (b) Fusion across the folds.

For the fusion block within the fold module, We choose ResNet50 [4],
EfficientNet-b0 [23] and InceptionNet-v1 [22] as the backbones to extract the
human facial expression features. In order to help the backbone learn more
abundant facial features, we opt for the pretrained weights on the VGGFace2
dataset[1] for ResNet50 and InceptionNet-v1, and use the pretrained weights on
ImageNet[21] for EfficientNet-b0, because the VGGFace2 dataset covers a large
range of pose, age and ethnicity which can improve the robustness of our
models. These three models have great differences in structure, so different
features of the same image can be extracted, which is helpful for the
classification of human expressions. By adjusting the fusion weights
$W_{1},W_{2},W_{3}$ of the three networks, we can choose the best models to
get the outputs, showing the advantages of ensemble learning.

As for the fusion block across the folds, as shown in the Figure 1(b), we
merge all the data in the training set and the validation set, and split the
dataset according to the idea of five-fold cross-validation in the other
module. This method can extract as much effective information as possible from
existing data, obtain a more reasonable and accurate evaluation of the models,
and avoid overfitting. Finally, we fuse the outputs of the five models to get
the best performance of our method.

###  3.3 Loss Function

We apply the Focal loss [17] to deal with the imbalanced problem of the
AffWild2 dataset. For the $m$ classes(include background), one-hot vector $y$
specifies the ground truth class $i(1\leq i\leq m)$ when $y_{i}=1$. The output
$p$ of model for each voxel is a vector of length $m$, which representing the
prediction probability for each class. Define the multi-class focal loss as:

$\mathbf{F}\mathbf{L}(p)=\sum_{i=1}^{m}y_{i}\alpha_{i}\left(1-p_{i}\right)^{\gamma_{i}}log(p_{i})$
(1)

where super parameter $\alpha_{i}$ is the balanced weight of class $i$ and
super parameter $\gamma$, which is called focusing parameter, is shared by
each class.

##  4 Experimental Results

Figure 2: The process to get the expression prediction sequence.

###  4.1 Training Setup

The training process is optimized by the Adam optimizer[8]. All the
experiments are implemented on a NVIDIA Tesla A100 with a batch size of 256.
To regularize the training process and accelerate the convergence of the
model, we use the Cosine Annealing[18] as the learning rate scheduler with the
initial learning rate of 0.001. Most of the testing models achieve the best
validation index (F1-score) after 30 epochs training.

###  4.2 Results

We follow the process shown in Figure 2 to get the prediction results for each
frame in the video.

The performance index is the average F1-score across all 8 categories. The
F1-score is a weighted average of the recall and precision and takes values in
the range $[0,1]$. It is defined as:

$\mathbf{F_{1}}=\frac{2\times precision\times recall}{precision+recall}$ (2)

The evaluation criterion for the Expression Recognition Challenge is:

$\mathbf{P}_{EXPR}=\frac{\sum_{expr}F_{1}^{expr}}{8}$ (3)

We merge the training set and the validation set and divide it into five
subsets with similar size according to the data distribution characteristics.
When conducting the five-fold experiment, four of the subsets are used as
training set and the remaining subset is used as test set. By applying this
division, we can conduct five experiments on the dataset and obtain five
different trainning results.

Table 1 shows the training results (F1-score) of three models——ResNet50,
EfficientNet-b0 and InceptionNet-v1 in each fold experiment. Table 2 shows the
training results (F1-score) of different fusion methods. We conduct a series
of experiments with different fusion weights and record the weights
configuration of the best model. According to the final results, the
indicators have been improved on the AffWild2 dataset and it can be noticed
that the multi-model ensemble learning method plays an important role in the
human expressions recognition task. Our final model will fuse the best results
of each fusion method, which will further improve the final metric.

##  5 Conclusion

In this paper, we propose an ensemble learning method to deal with the
chellange of human expression recognition by training several expression
classification models with different types of backbones—ResNet, EfficientNet
and InceptionNet and fuse the outputs of several models to predict the final
results. Moreover, we introduce the multi-fold ensemble method to train and
ensemble several models with the same architecture but different data
distributions to enhance the performance. We conduct many experiments on the
AffWild2 dataset of the ABAW2022 Challenge, and the results demonstrate the
effectiveness of our solution.

## References

  * [1] Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman.  Vggface2: A dataset for recognising faces across pose and age.  In 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), pages 67–74, 2018.
  * [2] Charles Darwin.  The Expression of the Emotions in Man and Animals.  University of Chicago Press, 2015.
  * [3] Giorgio Giacinto, Fabio Roli, and Luca Didaci.  Fusion of multiple classifiers for intrusion detection in computer networks.  Pattern Recognition Letters, 24:1795–1803, 08 2003.
  * [4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [5] Fu Jie Huang, Zhihua Zhou, Hong-Jiang Zhang, and Tsuhan Chen.  Pose invariant face recognition.  In Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), pages 245–250, 2000.
  * [6] Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu.  Dfew: A large-scale database for recognizing dynamic facial expressions in the wild, 08 2020.
  * [7] Miyuki Kamachi, Michael Lyons, and Jiro Gyoba.  The japanese female facial expression (jaffe) database.  01 1997.
  * [8] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  International Conference on Learning Representations, 12 2014.
  * [9] Dimitrios Kollias.  Abaw: Valence-arousal estimation, expression recognition, action unit detection & multi-task learning challenges, 02 2022.
  * [10] Dimitrios Kollias, Attila Schulc, Elnar Hajiyev, and Stefanos Zafeiriou.  Analysing affective behavior in the first abaw 2020 competition.  In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pages 637–643, 2020.
  * [11] Dimitrios Kollias, Viktoriia Sharmanska, and Stefanos Zafeiriou.  Face behavior a la carte: Expressions, affect and action units in a single network.  arXiv preprint arXiv:1910.11111, 2019.
  * [12] Dimitrios Kollias, Viktoriia Sharmanska, and Stefanos Zafeiriou.  Distribution matching for heterogeneous multi-task learning: a large-scale face study.  arXiv preprint arXiv:2105.03790, 2021.
  * [13] Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, Björn Schuller, Irene Kotsia, and Stefanos Zafeiriou.  Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond.  International Journal of Computer Vision, pages 1–23, 2019.
  * [14] Dimitrios Kollias and Stefanos Zafeiriou.  Expression, affect, action unit recognition: Aff-wild2, multi-task learning and arcface.  arXiv preprint arXiv:1910.04855, 2019.
  * [15] Dimitrios Kollias and Stefanos Zafeiriou.  Affect analysis in-the-wild: Valence-arousal, expressions, action units and a unified framework.  arXiv preprint arXiv:2103.15792, 2021.
  * [16] Dimitrios Kollias and Stefanos Zafeiriou.  Analysing affective behavior in the second abaw2 competition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3652–3660, 2021.
  * [17] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2999–3007, 2017.
  * [18] I. Loshchilov and F. Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  2016\.
  * [19] Emily B. Prince, Katherine B. Martin, and Daniel S. Messinger.  Facial action coding system.  2015\.
  * [20] Thomas N. Rincy and Roopam Gupta.  Ensemble learning techniques and its efficiency in machine learning: A survey.  In 2nd International Conference on Data, Engineering and Applications (IDEA), pages 1–6, 2020.
  * [21] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 115, 09 2014.
  * [22] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–9, 2015.
  * [23] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  arXiv e-prints, page arXiv:1905.11946, May 2019.
  * [24] Stefanos Zafeiriou, Dimitrios Kollias, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, and Irene Kotsia.  Aff-wild: Valence and arousal ‘in-the-wild’challenge.  In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, pages 1980–1987. IEEE, 2017.
  * [25] Xing Zhang, Lijun Yin, Jeffrey F. Cohn, Shaun Canavan, Michael Reale, Andy Horowitz, Peng Liu, and Jeffrey M. Girard.  Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database.  Image and Vision Computing, 32(10):692–706, 2014.  Best of Automatic Face and Gesture Recognition 2013.
  * [26] Zheng Zhang, Jeffrey M. Girard, Yue Wu, Xing Zhang, Peng Liu, Umur Ciftci, Shaun Canavan, Michael Reale, Andrew Horowitz, Huiyuan Yang, Jeffrey F. Cohn, Qiang Ji, and Lijun Yin.  Multimodal spontaneous emotion corpus for human behavior analysis.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3438–3446, 2016.


# TECHNIQUES AND RESULTS OF NEUTRAL LONG-LIVED PARTICLE SEARCHES IN ATLAS AND
CMS IN LHC RUN 2

CLAUDIA-ELISABETH WULZ  for the ATLAS and CMS Collaborations  Institute of
High Energy Physics of the Austrian Academy of Sciences,
Nikolsdorfergasse 18, 1050 Vienna, Austria

###### Abstract

Selected results of searches for neutral long-lived particles with displaced
and delayed jets, trackless jets and emerging jets, as well as displaced
muons, obtained with data recorded at a centre-of-mass energy of 13 TeV by the
ATLAS and CMS Collaborations during LHC’s Run 2 are presented, as well as
challenges and techniques.

00footnotetext: Copyright 2019 CERN for the benefit of the ATLAS and CMS
Collaborations. CC-BY-4.0 license.

##  1 Introduction

The standard model (SM) of particle physics includes a number of uncharged
long-lived particles (LLP) such as neutrinos, neutrons or neutral B, D, and K
mesons. Exotic, but so far undetected long-lived particles emerge in many
theories extending the standard model. We focus on detecting such neutral
particles and present selected recent results of searches by the ATLAS and CMS
collaborations [1, 2] at the LHC in proton-proton collisions at a centre-of-
mass energy of $\sqrt{s}=13$ TeV. In the context of these searches LLP’s are
defined as particles that do not decay instantly at the interaction point,
where they are produced, but at some measurable distance within the detector.
They may also remain stable within the detector volume.

A plethora of models predict long-lived particles. Supersymmetry (SUSY) in
various incarnations, dark matter scenarios, or portal interactions between a
hidden sector and the SM are sources of LLP’s. Among the SUSY models are
R-parity violating SUSY, gauge- and anomaly-mediated SUSY breaking scenarios,
split SUSY, or stealth SUSY. In gauge-mediated scenarios the gravitino is the
lightest supersymmetric particle (LSP). The striking feature of split
supersymmetry is that the gluino is a quasi-stable particle with a lifetime
that could be up to a hundred seconds long. Scalars except the Higgs boson are
ultra-heavy. In stealth SUSY R-parity is preserved, but missing energy
signatures are lacking. This becomes possible through a new hidden sector of
particles at the weak scale. Since SUSY breaking is small, the fermion and
boson pairs will be almost degenerate in mass. In hidden valley scenarios
particles of the hidden sector such as long-lived dark photons or dark hadrons
may decay to standard model particles. Left-right symmetric models predict
heavy right-handed neutrinos. Axions or axion-like particles are also expected
to have long lifetimes.

A particle’s proper lifetime is inversely proportional to its mass, and
proportional to the squared matrix element for the decay into its daughter
particles and the phase space for the decay. Heavy particles normally decay
quickly into lighter ones, unless their decay is suppressed by some mechanism.
Suppression can occur if couplings are small, if the decaying particle and the
decay products are almost degenerate in mass, or if the decay is mediated by
very heavy virtual exchange particles.

##  2 Challenges and techniques

Searches for particles with proper lifetimes $c\tau>1$ mm have been ongoing
for many years at the ATLAS and CMS experiments, even though the detectors
were a priori designed and optimised for promptly produced particles. However,
no signals have been observed so far. Therefore preparations for Run 3 of the
LHC and the corresponding detector upgrades have a strong focus on LLP’s. In
parallel with new theoretical developments clever ideas for triggering, data
acquisition, event reconstruction and analysis have been and are being worked
out, exploiting the detectors in ways they were not necessarily designed for.

Several challenges have to be overcome. The first challenge comes from physics
itself. Unusual or unexpected signatures of long-lived particles may arise.
For example, decays could occur outside the usual, dedicated detectors. Monte-
Carlo generators need to be developed in accord with theoretical progress, and
therefore samples with simulated data may not be readily available. In order
to be as model-independent as possible, searches are usually signature-driven.
One looks for displaced or delayed decays, or even particles trapped in
detector material. The MoEDAL experiment [3] at CERN, for example, will search
for magnetic monopoles in sections of an old CMS beam pipe. Displaced decays
with respect to the primary interaction vertex can manifest themselves in a
number of ways, for instance as displaced multi-track vertices, photons,
dileptons or lepton-jets. Displaced jets may have particular properties. Jets
with multi-track vertices may appear not only in the inner tracking system,
but also in the muon chambers. The characteristic signature of emerging jets
[4] are displaced tracks and many different vertices inside the jet cone.
Trackless jets, on the other hand, have a low fraction of electromagnetic
energy besides no tracks in the inner tracking detector. Figure 1 shows a
sketch of the different displaced decays in a collider detector. Another
signature for LLP’s are delayed decays, which require out-of-time detection.

Figure 1: Displaced decays in a collider detector (adapted from Heather
Russell, with permission).

Trigger, reconstruction and data analysis pose further challenges. Events with
LLP’s may be lost due to inadequate triggers or triggers with low efficiency.
A possible solution is to relax requirements, taking care not to exceed
trigger rate limits. Another method is to trigger on prompt particles produced
in association with long-lived ones. For particles with very long lifetimes
one can also investigate bunch crossings subsequent to the collision of
interest. A few special runs with triggers during gaps in LHC bunch trains
have already been taken. Most subdetectors do not provide timing information,
or their time resolution is not good enough. CMS, for example, will therefore
implement a novel muon timing detector (MTD) with a time resolution of 30 ps
[5]. Standard event records will have to be expanded to add timing
information, where available. Novel ideas for data acquisition that have
already been used by the LHC experiments can particularly benefit the search
for LLP’s. These are data parking and data scouting, also called trigger-level
analysis. Parking means that full raw data are stored, but not immediately
processed. Scouting on the other hand means that only a reduced event
information is stored, but at rates that can be much higher than the usual
bandwidth allocations would permit. For example, searches for dimuon
resonances with scouting data could thus be expanded to displaced dimuon
searches.

The standard physics object reconstruction is often inadequate for long-lived
particles, and systematic uncertainties, relevant for the data analysis, have
to be specially calculated. Initial basic requirements need to be revisited in
order to avoid that events get rejected at early stages of the reconstruction.
The interaction point constraint in triggering or reconstruction is not
usable. Track finding is generally optimised for prompt particles, therefore
dedicated tracking algorithms need to be developed. Secondary vertex finding
is usually tailored to b hadrons, so the b tagging algorithms have to be
extended to work also for impact parameters larger than a few millimetres. The
particle flow algorithm [6] that CMS is using needs to be adapted, since by
design it uses and combines all available event information from the
subdetectors, under the assumption of promptly produced particles. For all
experiments, the reconstruction of non-promptly produced electrons and taus
needs particular future development efforts. Instrumental effects, where
possible, should be estimated from data.

Last but not least, the treatment of backgrounds is very challenging. Standard
model particles such as KL or b hadrons may contribute. Both in-time and out-
of-time pileup have to be considered as well. While in-time pileup –
additional proton-proton collisions occurring in the same bunch crossing as
the collision of interest – is also a background in the detection of prompt
particles, out-of-time pileup occurring in adjacent bunch crossings is of
particular concern in the search for particles with very long lifetimes. But
one can actually also use in-time pileup events to search for LLP’s [7], for
example by studying displaced particles with low transverse momenta, which
could arise from $u\bar{u}d\bar{d}s\bar{s}$ sexaquarks [8] scattering in
detector material or the beam pipe. While it is practically impossible to
trigger on such particles directly, they could be present in pileup
interactions that are anyway recorded along with a primary interaction that
was triggered by a standard algorithm. The added benefit is that there is
nearly no selection bias. Cosmic rays are of course a major background in LLP
searches, since they can easily lead to false displaced decay vertices.
Accelerator backgrounds are also relevant, although they are usually well
under control. Examples of these are beam halo remnants or satellite bunches.
The first come from protons scattered off LHC collimators upstream of the
experiments, and the latter originate from collisions of low-luminosity
bunches resulting from the bucket structure generated by the accelerator’s
radio frequency cavities. Interactions with detector material and electronic
noise can also be sources of background. Care must be taken for instance that
genuine decays in calorimeters are not mistaken for noise. Special mitigation
procedures for anomalous detector signals, for example spike-cleaning
algorithms in the electromagnetic calorimeter of CMS, must be checked and
adapted in order to avoid rejecting interesting long-lived particle
signatures.

##  3 Results

The following section presents selected results on displaced and delayed jets,
trackless jets and emerging jets, as well as displaced muons, obtained with
data recorded at a centre-of-mass energy of 13 TeV.

###  3.1 Displaced jets

CMS has performed a search for long-lived particles decaying into jets [9],
testing a GMSB SUSY model with a long-lived gluino decaying to a gluon and a
gravitino, and the so-called Jet-Jet model, where long-lived scalar neutral
particles X are pair-produced through a $2\rightarrow 2$ scattering process,
mediated by an off-shell Z boson propagator. Each LLP can have a decay vertex
displaced from the production vertex by up to 55 cm in the plane transverse to
the LHC beams. The jets were reconstructed from energy deposits in the
calorimeter towers. The data sample corresponds to an integrated luminosity of
35.9 fb-1, collected with a dedicated displaced jet-trigger, which required
$H_{\mathrm{T}}$ to be larger than 350 GeV, with $H_{\mathrm{T}}$ defined as
the scalar sum of the transverse momenta ($p_{\mathrm{T}}$) of all jets
satisfying $p_{\mathrm{T}}>$ 40 GeV and $|\eta|<$ 2.5. The online event
selection also required at least two jets with $p_{\mathrm{T}}>$ 40 GeV within
$|\eta|<$ 2.0, at most two associated prompt tracks with transverse impact
parameters with respect to the leading primary vertex smaller than 1 mm, and
at least one associated displaced track with a transverse impact parameter
larger than 0.5 mm, and an impact parameter significance larger than 5. The
main background comes from QCD jets, which are suppressed using a likelihood
discriminant based on three variables based on track, jet and vertex
information. Figure 2 (left) shows the likelihood discriminants for data,
simulated QCD multijet events, and simulated signal events.

Figure 2: Likelihood discriminant for data, simulated QCD multijet events, and
simulated signal events (left), expected and observed 95% CL upper limits on
the pair production cross section of the long-lived particle X, assuming a
100% branching fraction for X to decay to a quark-antiquark pair, shown at
different X masses and proper decay lengths for the Jet-Jet model (centre),
and expected and observed 95% CL limits for the long-lived gluino model in the
mass-lifetime plane, assuming a 100% branching fraction for
$\tilde{g}\rightarrow g\tilde{G}$ decays (right).[9]

The cutoff value of 0.9993 for the event selection was determined by
optimising the signal sensitivity for the Jet-Jet model across proper decay
lengths from 1 to 1000 mm and X masses from 100 to 1000 GeV. Figure 2 (centre)
shows the expected and observed 95% confidence level (CL) upper limits on the
pair production cross section of the long-lived particle X, assuming X to
decay exclusively to a quark-antiquark pair, for different particle masses
$m_{\mathrm{X}}$ and proper decay lengths for the Jet-Jet model, while Fig. 2
(right) depicts the corresponding limits for the GMSB SUSY model, assuming a
100% branching fraction for $\tilde{g}\rightarrow g\tilde{G}$ decays. The
solid (dashed) lines represent the observed (median expected) limits. The
shaded bands represent the regions containing 68% of the distributions of the
expected limits under the background-only hypothesis. For the Jet-Jet model
the limits are most stringent for proper decay lengths between 3 and 100 mm.
For smaller decay lengths, the limits become less restrictive because of the
vetoes on prompt activity. Since the tracking efficiency decreases with larger
displacement, the limits also become less stringent for larger decay lengths.
For the GMSB model gluino masses up to 2300 GeV are excluded for proper decay
lengths between 20 and 110 mm.

###  3.2 Delayed jets

CMS has also searched for delayed jets resulting from displaced decays of
long-lived particles in the electromagnetic calorimeter (ECAL) [10]. The delay
is expected to be a few nanoseconds for a TeV scale particle which travels a
distance of the order of 1 m before decaying. Events with such particles would
escape reconstruction in a tracker-based search because of the inability to
reconstruct tracks for decay points separated from the primary vertex by more
than about 50 cm in the plane perpendicular to the beam axis. A similar GMSB
benchmark model as in the previous subsection was used as a benchmark. This
model contains pairs of long-lived gluinos that each decay into a gluon, which
forms a jet, and a gravitino, which escapes the detector causing significant
missing energy in the event. The analysis exploits the full Run 3 dataset of
137 fb-1. Its specificity is the use of timing information, which leads to a
significant gain in sensitivity [11]. It is provided by the lead tungstate
crystal calorimeter in the barrel detector region, read out by silicon
avalanche photodiodes (APD) and yielding a time resolution of 200 ps. The
online trigger required missing energy larger than 120 GeV. Particle flow was
not used for the delayed jet reconstruction due to the non-standard tracker
component, only calorimeter clustering. In addition to missing energy, the
signal selection required jets to have a delay $t_{\mathrm{jet}}>3$ ns.
Crystal cells within a distance $\Delta R<0.4$ with respect to the jet axis
have been taken into account, where $t_{\mathrm{jet}}$ is given by the median
cell time, and $t=0$ ns is defined as the arrival time of a particle
travelling from the origin to a cell with velocity $c$.

It was possible to perform the analysis of the full LHC Run 2 dataset at this
time, because no Monte Carlo simulations of the backgrounds had been
necessary. They were estimated solely from control regions with data.
Backgrounds include ECAL resolution tails, direct APD hits, in-time and out-
of-time pileup, cosmic muon deposits in the electromagnetic calorimeter, beam
halo and satellite bunches. The latter occur in steps of 2.5 ns. The main
backgrounds are primarily reduced by jet cleaning, applying several
requirements. The number of ECAL cells in a jet must exceed 25, to reject
noise and direct APD hits. The hadronic energy fraction, defined as
$E_{\mathrm{ECAL}}/(E_{\mathrm{ECAL}}+E_{\mathrm{HCAL}}$), has to be larger
than 0.2. The fraction of tracks associated to the primary vertex must be less
than 1/12, to reject jets with a mismeasured time or originating from
satellite bunch collisions. The spread in the time of the constituent cells of
a signal jet is required to be small, since all the component cells originate
from the same delayed jet. The good efficiency of the jet cleaning procedure
can be seen from Fig. 3.

Figure 3: Jet timing distributions for 2016, 2017 and 2018 data, before
(left), and after jet cleaning (right).[10]

The jet timing distribution of the backgrounds in the signal region, compared
to signals predicted by the above-mentioned GMSB model for different gluino
masses and lifetimes is depicted in Fig. 4 (left). The obtained results in
Fig. 4 (right) show that the signal reach can be significantly extended
compared to tracker-based searches by using timing information of
electromagnetic energy deposits [12], [13], [14]. The interpretation within
the model yields exclusions of gluino masses up to 2500 GeV (2150 GeV) for
proper lifetimes of 1m (30 m).

Figure 4: Timing distribution of the backgrounds in the signal region,
compared to predictions for the GSMB model with representative parameters
(left); 95% CL observed upper limits on $\sigma$ in the gluino mass and
$c\tau_{0}$ plane after all selections (right).[10] The area enclosed by the
thick black curve represents the observed exclusion region, while the dashed
red lines indicate the expected limits and their $\pm 1$ standard deviation
ranges. The thin black lines show the effect of the theoretical uncertainties
on the signal cross section.

###  3.3 Trackless jets

A search for pairs of neutral, long-lived scalar particles decaying to
trackless jets in the calorimeter has been performed by the ATLAS
Collaboration [15]. The signature consists of displaced jets in the hadron
calorimeter of the outer edge of the electromagnetic calorimeter. The analysis
probes proper decay lengths between a few centimetres and a few tens of
metres. As a benchmark, a simplified model with a hidden sector that is
connected to the standard model through a heavy neutral boson $\Phi$. It
decays to two scalars $s$, which in turn decay to fermion-antifermion pairs.
The data consisted of two samples of 10.8 fb-1 and 33.0 fb-1, recorded with a
low- and a high-$E_{\mathrm{T}}$ trigger, respectively. The event selection
required at least two trackless jets with a low fraction of electromagnetic
energy, also called CalRatio (CR) jets. To select events with trackless jets,
an event-level variable, $\Delta R_{\mathrm{min}}$(jet, tracks), is used,
where $\Delta R_{\mathrm{min}}$(jet, tracks) is defined as the angular
distance between the jet axis and the closest track with $p_{\mathrm{T}}>2$
GeV, and $\Sigma\Delta R_{\mathrm{min}}$(jet, tracks) is calculated by summing
this distance over all the clean jets with $p_{\mathrm{T}}>50$ GeV. Machine
learning techniques have been used for the analysis, namely a neural network
to determine the jet origins, and a boosted decision tree (BDT) classifier for
jets. The backgrounds, estimated with an ABCD method, consist mainly of QCD
multijets and beam-induced backgrounds. Fig. 5 (left) demonstrates the
efficiency of the selection.

Figure 5: $\Sigma\Delta R_{\mathrm{min}}$(jet, tracks) versus per-event BDT
for a simulated signal sample after event selection for the
high-$E_{\mathrm{T}}$ sample (left); observed cross section times branching
fraction limits for a mediator mass of $m_{\Phi}=600$ GeV (centre); combined
results for CalRatio and muon system analyses for $m_{\Phi}=600$ GeV
(right).[15]

Limits at 95% CL have been set on the cross section times branching fraction
$\sigma(\Phi)\times\mathcal{B}(\Phi\rightarrow s\bar{s})$. For mediator masses
$m_{\Phi}$ of 400, 600, and 1000 GeV values above 0.1 pb are excluded between
12 cm and 9 m, 7 cm and 20 m, and 4 cm and 35 m respectively, depending on
$m_{s}$. The limit curve for $m_{\Phi}=600$ GeV and $m_{s}=150$ GeV is shown
in Fig. 5 (centre). The corresponding Run 1 curve [16] is scaled by the ratio
of parton luminosities for gluon-gluon fusion between 13 and 8 TeV. The right-
hand figure shows results for this CalRatio jet analysis as well as a
combination with a displaced jet analysis in the muon system (MS) [17]. For
masses $m_{\Phi}\geq 600$ GeV, the CR analysis is in general more sensitive
than the MS analysis. The combination provides a modest improvement on the CR-
only limit at long decay lengths.

###  3.4 Emerging jets

Emerging jets are characteristic for models with a composite dark sector. CMS
has studied such a model [4], which contains a heavy mediator
$\mathrm{X_{DK}}$ that couples both to dark sector and SM particles. The
experimental signature consists of four high-$p_{\mathrm{T}}$ jets, two from
down quarks and two from dark quarks [18]. The dark quark jets contain many
displaced vertices arising from the decays of dark pions ($\pi_{\mathrm{DK}}$)
produced in the dark parton shower and fragmentation.The trigger was a simple
$H_{\mathrm{T}}$ trigger with a threshold of 900 GeV, and the event selection
required two jets with displaced tracks and many different vertices within the
jet cone, and two regular jets. The main background is SM four-jet production,
where jets are tagged as emerging either because they contain long-lived B
mesons or reflect track misreconstruction. It is estimated with a data-driven
method and depends on parton flavour and track multiplicity, as can be seen
from Fig. 6 (left).

Figure 6: Misidentification rate as a function of track multiplicity for b
jets and light-quark jets (left); $\alpha_{3\mathrm{D}}$ for background
(black) and for signals with a dark mediator mass of 1 TeV and a dark pion
mass of 5 GeV for dark pion proper decay lengths ranging from 1 to 300 mm
(centre); upper limits at 95% CL on the signal cross section and signal
exclusion contours derived from theoretical cross sections for a model with a
dark pion mass of 5 GeV (right).[18]

The central figure shows the variable $\alpha_{3\mathrm{D}}$, which quantifies
the fraction of jet $p_{\mathrm{T}}$ associated with prompt tracks, for
ordinary QCD jets and for signal jets arising from dark pions of mass 5 GeV
with different lifetimes. Figure reffig:emergingjets (right) shows upper
limits at 95% CL on the signal cross section and signal exclusion contours
derived from theoretical cross sections for a model with a dark pion mass of 5
GeV in the plane of the mediator mass and the dark pion lifetime. The solid
red contour is the expected upper limit, with its one standard- deviation
region enclosed in red dashed lines. The solid black contour is the observed
upper limit. The region to the left of the observed contour is excluded.
Mediator masses between 400 and 1250 GeV are thus excluded for decay lengths 5
to 225 mm.

###  3.5 Displaced dimuons

A search for long-lived neutralinos and long-lived dark photons ZD has been
studied by ATLAS [19], using displaced muons as the signature. In the first
model the neutralino of a GMSB scenario decays into a gravitino and a Z boson,
which in turn decays to dimuons. In the second model two ZD arise from the
decay of a Higgs boson, with at least one of them decaying to two displaced
muons. The trigger was as inclusive as possible, requiring two opposite-sign
muons with a vertex displacement of up to 4 m from the interaction point. Both
a high-mass and a low-mass trigger were used, with the first one targeting the
decay Z$\rightarrow\mu^{+}\mu^{-}$, and the second one
Z${}_{\mathrm{D}}\rightarrow\mu^{+}\mu^{-}$. The high-mass trigger required
missing energy and a single muon with $p_{\mathrm{T}}>60$ GeV, while the low-
mass trigger required collimated dimuons or trimuons with lower transverse
momenta, between 20 and 6 GeV. Requiring missing energy compensates the loss
in single muon trigger efficiency, which drops from 70% at the interaction
point to 10% at a distance of 4 m. Backgrounds come mainly from Drell-Yan,
$t\bar{t}$, single top and multijet production. The dimuon mass and vertex
distance distributions for the two trigger samples are shown in Fig. 7 (left).
The events with the highest masses have been identified as a Z and a cosmic
ray particle, respectively. No signal has thus been found. Exclusion contours
for the two models in the cross section times branching fraction and lifetime
plane are depicted in Fig. 7 (right). Lower and upper lifetime limits of 0.3
and 2400 cm, depending on the model parameters, have been set.

Figure 7: Dimuon mass and distance of dimuon vertex from interaction point
(left); exclusion contours for the GMSB model and the dark photon model
(right).[19]

##  4 Conclusions

No signals for long-lived neutral particles have yet been found in the
analysis of the LHC Run 2 data analysed so far by the ATLAS and CMS
collaborations. Non-standard trigger, data acquisition and analysis strategies
and techniques are being further developed, and results for the full dataset
will be published soon.

## References

## References

  * [1] ATLAS Collaboration, JINST 3, S08003 (2008).
  * [2] CMS Collaboration, JINST 3, S08004 (2008).
  * [3] MoEDAL Collaboration, CERN-LHCC-2009-006,
https://cds.cern.ch/record/1181486 (2009).

  * [4] P. Schwaller, D. Stolarski, and A. Weiler, JHEP 1505, 059 (2019).
  * [5] CMS Collaboration, CERN-LHCC-2019-003, https://cds.cern.ch/record/2667167 (2019).
  * [6] CMS Collaboration, JINST 12, 10003 (2017).
  * [7] B. Nachman, F. Rubbo, Phys. Rev. D 97, 092002 (2018).
  * [8] G. R. Farrar, arXiv:1708.08951 (2017).
  * [9] CMS Collaboration, Phys. Rev. D 99, 032911 (2019).
  * [10] CMS Collaboration, CMS-PAS-EXO-19-001, to be published in PLB,
http://cds.cern.ch/record/2667508 (2019).

  * [11] J. Liu, Z. Liu, and L.-T. Wang, arXiv:1805.05957.
  * [12] ATLAS Collaboration, Phys. Rev. D 97, 052012 (2018).
  * [13] CMS Collaboration, Phys. Rev. D 99, 032011 (2019).
  * [14] CMS Collaboration, Phys. Rev. D 99, 092011 (2018).
  * [15] ATLAS Collaboration, arXiv:1902.03094 (2019).
  * [16] ATLAS Collaboration, Phys. Lett. B 743, 15 (2015).
  * [17] ATLAS Collaboration, Phys. Rev. D 99, 052005 (2018).
  * [18] CMS Collaboration, JHEP 02, 179 (2019).
  * [19] ATLAS Collaboration, Phys. Rev. D 99, 012001 (2019).