# On the Evaluation of Sequential Machine Learning for Network Intrusion
Detection

Andrea Corsini<EMAIL_ADDRESS>University of Modena and Reggio
EmiliaVia Pietro VivarelliModenaItaly41125 ,  Shanchieh Jay Yang
<EMAIL_ADDRESS>Rochester Institute of Technology8 Lomb Memorial
DriveRochester, NYUSA14623 and  Giovanni Apruzzese<EMAIL_ADDRESS>University of LiechtensteinFürst-Franz-Josef-StrasseVaduzLiechtenstein9490

(2021)

###### Abstract.

Recent advances in deep learning renewed the research interests in machine
learning for Network Intrusion Detection Systems (NIDS). Specifically,
attention has been given to sequential learning models, due to their ability
to extract the temporal characteristics of Network traffic Flows (NetFlows),
and use them for NIDS tasks. However, the applications of these sequential
models often consist of transferring and adapting methodologies directly from
other fields, without an in-depth investigation on how to leverage the
specific circumstances of cybersecurity scenarios; moreover, there is a lack
of comprehensive studies on sequential models that rely on NetFlow data, which
presents significant advantages over traditional full packet captures. We
tackle this problem in this paper. We propose a detailed methodology to
extract temporal sequences of NetFlows that denote patterns of malicious
activities. Then, we apply this methodology to compare the efficacy of
sequential learning models against traditional static learning models. In
particular, we perform a fair comparison of a ‘sequential’ Long Short-Term
Memory (LSTM) against a ‘static’ Feedforward Neural Networks (FNN) in distinct
environments represented by two well-known datasets for NIDS: the CICIDS2017
and the CTU13. Our results highlight that LSTM achieves comparable performance
to FNN in the CICIDS2017 with over 99.5% F1-score; while obtaining superior
performance in the CTU13, with 95.7% F1-score against 91.5%. This paper thus
paves the way to future applications of sequential learning models for NIDS.

Long Short Term Memory, Machine Learning, Network Intrusion Detection,
Cybersecurity, Network Flows, Deep Learning

††copyright: acmcopyright††journalyear: 2021††conference: The 16th
International Conference on Availability, Reliability and Security; August
17–20, 2021; Vienna, Austria††booktitle: The 16th International Conference on
Availability, Reliability and Security (ARES 2021), August 17–20, 2021,
Vienna, Austria††price: 15.00††doi: 10.1145/3465481.3470065††isbn:
978-1-4503-9051-4/21/08††ccs: Security and privacy Intrusion detection
systems††ccs: Security and privacy Network security††ccs: Computing
methodologies Temporal reasoning

##  1\. Introduction

Network Intrusion Detection Systems (NIDS) are of paramount importance for the
protection of network infrastructures. Many detailed works (e.g., (Mishra et
al., 2018; Liao et al., 2013; Buczak and Guven, 2016; Khraisat et al., 2019))
highlighted the significant advances of NIDS, which are subject of continuous
improvements. Indeed, NIDS must face significant challenges such as the ever-
changing network environments as well as skilled and motivated adversaries. In
particular, the recent promising successes in machine and deep learning
renewed the interest in devising autonomous defensive systems (Buczak and
Guven, 2016; Khraisat et al., 2019), giving life to a next generation of NIDS.

Among the plethora of proposals that exploit Machine Learning (ML) techniques,
one promising direction seeks to leverage sequential learning methods that
rely on the enticing idea that specific network events exhibit temporal
patterns. When properly applied, sequential ML methods—and especially their
“deep” variants—offer the appreciable opportunity to automatically extract
these temporal patterns. These patterns can then be used to create
conventional anomaly- or signature-based detection systems. Despite abundant
literature, an effective and realistic deployment of sequential learning
models for NIDS is yet a distant goal. For instance, identifying temporal
patterns is challenging because many factors must be considered, such as, the
adopted network protocols; the web-services employed by the network; and the
possible threats that may target the network.

Moreover, the massive size and variability of modern network traffic data
makes timely analyses based on traditional packet captures (PCAP) a difficult
objective – which is further aggravated by the increasing usage of encrypted
traffic. To mitigate this issue, many proposals (e.g., (Sperotto et al.,
2010)) favor the inspection of Network Flows (NetFlows) instead of PCAP data.
In this context, we make an intriguing observation: although it has been
empirically shown that cyber-attacks exhibit temporal patterns that can be
easily extracted at the packet level, it is less clear whether this is still
true at the ‘higher’ NetFlow level111NetFlows are a statistical summary of
packets, grouped according to a set of rules (Cisco, 2021).. Indeed, temporal
patterns within packets are not guaranteed to exist also in the corresponding
NetFlows, which are metadata that summarize the corresponding PCAP data.

In this paper we devise a novel methodology for (i) investigating the
existence of temporal patterns in malicious NetFlows; and, if such patterns
exist, (ii) assess their effectiveness for detection purposes. The primary
focus of past literature on sequential ML-NIDS is the proposal of solutions
that “outperform” the state-of-the-art according to some performance metrics;
however, the investigation of the actual benefits brought by these solutions
is often neglected. In contrast, our objective is a preliminary analysis
focused on the investigation of the existence, advantages and shortcoming
brought by using temporal patterns at the NetFlow level for NIDS. To the best
of our knowledge, this paper is the first effort that analyzes and performs a
fair comparison of sequential ML models against their ‘static’ counterparts.

We experimentally evaluate the proposed methodology on two well-known and
recent datasets for NIDS, the CTU13 (Garcia et al., 2014) and the CICIDS2017
(Sharafaldin et al., 2018). We apply our method to extract temporal patterns,
and use such patterns to train and compare two Deep Learning (DL) algorithms:
one based on Long-Short-Term-Memory (LSTM), which leverages sequential
learning; and one based on Feedforward Neural Networks (FNN), which is a
traditional ‘static’ learning method. Our results highlight that the patterns
extracted by our method yield proficient detectors in the CICIDS2017 scenario,
with both LSTM and FNN achieving over 99.5% F1-score; whereas the LSTM
significantly outperforms the FNN detector in the CTU13 scenario, with 95.7%
F1-score against 91.5%.

We are confident that this paper will shed some light on the application of
sequential learning models for NetFlow-based NIDS. Our work thus paves the way
to more reliable defensive platforms, which can jointly combine existing
detection techniques with novel solutions that also consider the temporal
axis.

To summarize the main contributions of this work:

  * •

we devise an original method to extract temporal patterns from NetFlows that
can be leveraged by sequential ML;

  * •

we evaluate the proposed method to explore its benefits by performing a fair
comparison of a ‘static’ FNN against a ‘sequential’ LSTM on two well-known
datasets for NIDS.

The remainder of this paper is structured as follows. Section 2 discusses
related work. Section 3 describes the proposed methodology for extracting
temporal patterns in malicious NetFlows. Section 4 presents the experimental
settings. Section 5 evaluates our proposed method. Section 6 concludes the
paper with final remarks and future work.

##  2\. Related Work

There exist many proposals for detecting malicious traffic by means of machine
learning techniques (e.g., (Buczak and Guven, 2016; Mishra et al., 2018)).
There is also abundant literature on time-based analyses for NIDS (e.g.,
(Apruzzese et al., 2017; Zhou and Lang, 2003)).

This paper lies at the intersection of these two application fields. Our
objective is investigating the effectiveness of temporal relationships for ML-
based NIDS (ML-NIDS), hence our literature review focuses on those proposals
that leverage machine- and, specifically, deep-learning by considering also
the temporal axis, usually through Recurrent Neural Networks (RNN).

Among the earliest successful applications of sequential learning models to
NIDS there is the 2017 proposal in (Yin et al., 2017). Since then, several
works have proposed learning models that extract temporal relationships in the
network traffic and use them for ML purposes. We categorize and discuss such
works on the basis of the approach used to extract the temporal patterns,
namely: automatic approaches, where the temporal patterns are extracted
through a pipeline that inspects the input data and automatically learns the
most significant temporal features; manual approaches, where the temporal
patterns are extracted exclusively on the basis of the input features.

We provide an overview of the major differences among related work in Table 1.
This table shows the datasets considered, the employed DL techniques (CNN
stands for Convolutional Neural Networks), and whether the approach is
tailored for anomaly- or misuse-based222Note that within the ‘misuse’ category
we also include approaches that use the trained ML models to perform the
detection. NIDS.

Table 1. Overview of related work on sequential ML-NIDS.

| Paper | Datasets |  Detection method | CNN | RNN | FNN
---|---|---|---|---|---|---
Automatic | (Han et al., 2020) | ISCX2012, CICIDS2017 | misuse | ✓ | ✓ | $\times$
(Pektaş and Acarman, 2019) | CTU13, ISOT | misuse | ✓ | ✓ | ✓
(Zhang et al., 2019) | CICIDS2017, CTU13 | anomaly | ✓ | ✓ | $\times$
(Sinha et al., 2019) | CTU13 | misuse | $\times$ | ✓ | $\times$
(Wu and Guo, 2019) | NSLKDD, UNSW-NB15 | misuse | ✓ | ✓ | $\times$
Manual | (Radford et al., 2018b) | CICIDS2017 | anomaly | $\times$ | ✓ | ✓
(Radford et al., 2018a) | ISCX2012 | anomaly | $\times$ | ✓ | ✓
(Gwon et al., 2019) | UNSW-NB15 | misuse | $\times$ | ✓ | $\times$
(Kim et al., 2020) | CTU13 | anomaly | $\times$ | ✓ | $\times$
(Torres et al., 2016) | CTU13 | misuse | $\times$ | ✓ | $\times$

We do not consider works whose proposal is evaluated exclusively on the
(outdated and widely deprecated (Sharafaldin et al., 2018)) NSLKDD dataset
(e.g. (Boukhalfa et al., 2020)), and that have been published recently (since
2017). Finally, although many works use RNN for NIDS (e.g., (Khan et al.,
2019)), they often neglect considering temporal dependencies, hence we do not
consider such works in our analysis.

###  2.1. Automatic extraction of features

These works rely on an end-to-end pipeline that leverages spatio-temporal
dependencies, usually by means of CNN and RNN. The CNN automatically learns
both spatial and temporal features, thus removing the need of manually
extracting the features used to devise the detector. In some of these
proposals (i.e., (Wu and Guo, 2019; Zhang et al., 2019)), the network traffic
is inspected at the packet level by transforming each packet into an image.
These images are first spatially analyzed through a CNN, and then, the feature
maps describing packets are used to extract temporal patterns by means of
sequential models.

The authors of (Han et al., 2020) proposed a Spatial and Temporal Aware
Intrusion Detection Model (STIDM) composed of LeNet-5333LeNet-5 is a well-
known CNN architecture: http://yann.lecun.com/exdb/lenet and a modified
version of the LSTM. LeNet-5 was used to extract spatial features, while the
modified LSTM was used to extract temporal patterns by taking into account the
time intervals between packet exchanges. STIDM was trained on truncated flows
of multiple packets created by grouping on the tuple (SourceIP, SourcePort,
DestinationIP, DestinationPort, Protocol). We observe that it is difficult to
attribute the outstanding performance of STIDM to the CNN and/or the LSTM from
the results shown in (Han et al., 2020). Furthermore, STIDM works on truncated
sequences of packets, which makes determining the existence of temporal
patterns in NetFlows unfeasible.

The approach in (Zhang et al., 2019) is similar to (Han et al., 2020), as they
use the same STIDM structure to extract spatial features and temporal
patterns, but the LSTM architecture is different. In this work, the overall
model was used to analyze sequences of packets composed of only the first 160
bytes of every packet; then, the packets are aggregated by following the same
five-tuple of STIDM, but only the first 10 packets in each sequence are
considered. Similarly to (Han et al., 2020), from (Zhang et al., 2019) it is
difficult to conclude anything about the existence temporal patterns among
NetFlows and about the true benefits brought by the LSTM, since the results of
the tested models (CNN, LSTM and CNN+LSTM) showed comparable performance.

A different proposal for a end-to-end pipeline is in (Wu and Guo, 2019). Here,
instead of chaining a CNN with a LSTM (as in (Han et al., 2020)), the authors
first create a block composed by these two techniques, and then stack many of
these blocks obtaining higher abstraction levels. Although the final
performance of such architecture achieves excellent results, the method for
processing the input data is poorly described, and no information is provided
as to how the temporal domain is used.

In (Sinha et al., 2019), the authors used a sequence of graphs to model the
time evolution of the network traffic, in which nodes are the hosts (in the
form of IP addresses) of the network, and edges are the packets exchanged by
hosts. After generating these graphs, time series are extracted from each node
by computing a set of graph-based features (e.g., in/out-degree, or in/out-
neighbors). Then, such time series are divided in five overlapped time
intervals, which are used to train a LSTM to detect malicious (i.e.,
anomalous) hosts. We observe that the approach in (Sinha et al., 2019)
presents a heavy burden in terms of computational resources for its feature
extraction; furthermore, dividing a time series into fixed intervals does not
allow to infer long-lasting temporal patterns–despite obtaining encouraging
detection results on the considered testbed. Finally, this approach focuses on
analyzing PCAP data, and not on NetFlows.

The authors of (Pektaş and Acarman, 2019) propose an approach similar to
(Sinha et al., 2019), but assume NetFlows as input. A single graph was
extracted from the dataset, where nodes represent hosts (as IP addresses),
while edges represent the NetFlows exchanged by the hosts; then, a statistical
summary of the NetFlows between each pair of nodes is created. Such
statistical summary is integrated with temporal features extracted from the
sequence of connection states between each pair of nodes. Then, both the
statistical summary and the temporal features are used to make the final
inference throughout a FNN. What makes (Pektaş and Acarman, 2019) intriguing
is that there is an actual benefit from using temporal features extracted from
the connection states of NetFlows. However, it also demonstrated that using
only the temporal features extracted from the connection states was not enough
to achieve good detection.

To summarize, the major drawbacks of these approaches is their specificity to
packet level analyses. Inferring spatial features from NetFlow data may still
give a benefit, but the overall detection process must take into account the
complex NetFlow semantic. Finally, none of these works prove the existence of
malicious temporal patterns in NetFlows that are usable for ML-NIDS.

###  2.2. Manual extraction

These approaches use the features as they are provided in the input data; some
preprocessing may occur, but there is no automatic feature extraction
mechanism, that is, the features are not produced by any ML method.

In (Radford et al., 2018a), the authors use a bidirectional LSTM model to
predict whether a communication was atypical or not. Specifically, they
leverage Natural Language Processing by transforming NetFlows into ‘tokens’,
where a token is defined as either a network port or a byte-port tuple. Then,
these tokens were grouped by IP-pairs within hourly bins to form time-ordered
sequences. This allowed the LSTM to learn a model of the benign traffic, which
is used to detect anomalous communications. We observe, however, that the
NetFlow semantic was used only to extract information on the byte-port tuple,
which is a limitation in large networks with high diversity in the traffic.
Moreover, having sequences focused on IP-pair and of limited duration does not
imply that the learned temporal patterns (if any) are truly effective for
NIDS.

In (Radford et al., 2018b), the authors extend (Radford et al., 2018a) by
evaluating five different methods for aggregating NetFlow tokens in time-
ordered sequences. They showed that different aggregation methods do not play
an important role in training sequential models, due to negligible performance
differences. While this observation might be true in their specific context,
it is not applicable for our case. Indeed, we want our ML-NIDS to learn the
temporal pattern of attacks, thus splitting the corresponding attack NetFlows
is not viable: by doing so, we would break the temporal patterns (if they
exist).

In (Kim et al., 2020), the authors propose a Recurrent Variational Autoencoder
to detect botnet connections through anomaly detection. Their model is trained
on certain botnet types and tested on unseen ones, showing good detection
performance and demonstrating generalization capabilities. We point out that
the generalization capabilities of some ML methods (and, specifically, of RNN)
are an appreciable characteristic. However, the sequences used in (Kim et al.,
2020) are created by aggregating NetFlows on their source IP, and then each
sequence is reduced by summarizing NetFlows, therefore inducing loss of data
which may break any temporal pattern.

In (Torres et al., 2016) a behavioral LSTM is devised to counter botnet
attacks by analyzing the long-term traffic characteristics. Here, NetFlows are
first aggregated into sequences according to a 4-tuple (source IP, destination
IP, destination part, protocol); then, each NetFlow is vectorized with the
behavioural model on the basis of three features: size, duration, periodicity.
Although this work evaluates the prominent issues in training LSTM (e.g., data
imbalance), it manually cherry picked the best features for detection. We do
not make such assumption in this paper.

The LSTM is used also in (Gwon et al., 2019) to detect and classify malicious
network packets. The authors evaluate the semantic of categorical features
through embedding layers. Their results show that such layers do not improve
the detection, and that the binary classification of LSTM achieves optimal
performance ($99.75\%$ F1-score). However, the LSTM is trained on sequences of
fixed length, and no information on how to determine such length is provided.
As in (Torres et al., 2016), this paper cherry-picks the optimal thresholds
that ensure best results on their specific testbed, with low practical value.

To summarize, these works have little realistic usage, and do not allow to
assess the effectiveness of the extracted temporal patterns.

##  3\. Proposed Solution

Our goal is to investigate the existence of temporal patterns in malicious
NetFlows, and then explore the benefits of using these patterns in detecting
malicious activities via machine learning. Hence, our first objective is the
extraction of such temporal patterns. To this purpose, we propose an original
method based on expert knowledge and network data analytics. Then, we need to
verify if the extracted patterns are useful for detection purposes.

As explained in Section 2, many state-of-the-art approaches for extraction of
temporal patterns have significant limitations. Thus, we propose an original
method that aims at mitigating such deficiencies. The intention is to give
more guarantees that the underlying temporal characteristics of different
attacks are truly captured by the generated sequences, and are hence usable
for efficient detection. In the remainder of this work, we will use the term
‘scenario’ to denote a complete ‘attack scenario’, i.e., a sequence of
NetFlows that describes a specific attack scenario.

We first describe the threat model, the characteristics of the network
environment and the assumptions required by our method (Section 3.1). Then we
present our proposed method for creating scenarios (Section 3.2 and outline
the characteristics of the sequential ML model (Section 3.3).

###  3.1. Threat Model and Assumptions

This paper makes the following assumptions about the attacker’s side, the
considered network environment, and the input data required for creating the
attack scenarios.

Attacker. We assume an attacker with the goal of either stealing sensitive
data from, or disrupting the services provided or used by, the targeted
organization. As is common in NIDS scenarios, the attacker is assumed to have
established a foothold within the targeted network by compromising one (or
more) of its most vulnerable hosts. The attacker does not have any information
about the defensive mechanisms monitoring the organization; for example, the
attacker does not know the datasets used to train ML-based NIDS. The attacker
can interact with the network services accessible from within the organization
network, but they do not have direct access to the NetFlow-exporter nor to any
defensive system.

Environment. We assume the typical network environment spanning from dozens to
hundreds of hosts that perform multiple network activities. The traffic
generated by these hosts is first captured as PCAP data, and then transformed
into NetFlows by means of any NetFlow-generation software444Exemplary NetFlow
software: https://qosient.com/argus/.. These extracted NetFlows are then
analyzed by the NIDS, possibly after some preprocessing operations. We assume
that the traffic generated by the most critical servers of the organization
(to which the attacker has no access) is analyzed by dedicated mechanism.

Input Data. Our method requires a collection of NetFlows that are labelled
according to the malicious network activities that they represent (or as
‘benign’ if they are not related to illegitimate actions). For each NetFlow,
our method requires the following pieces of information: the timestamp, the
Source/Destination ports, the protocol, the duration, the direction of the
connection with respects to the network environment (i.e., inbound, outbound,
or bidirectional), and optionally the specific attack type denoted by the
label. Any additional piece of information is not required, but – if available
– it can be freely added to the list of considered features to provide more
fine-grained results.

In summary, our proposed method makes the assumptions typical of NIDS
scenarios, and only requires the essential pieces of information obtainable
from any NetFlow-generation software. The availability of labelled data is
also a standard assumption in machine learning tasks.

###  3.2. Extraction of Attack Scenarios

Our method uses expert knowledge that can be applied to existing labelled
NetFlow data to extract the attack scenarios.

The fundamental idea is to use the available information on a given set of
NetFlow data to extract these scenarios. Such information can be readily
acquired from the labels or the documentation of a dataset; or after
exploratory data-analytics procedures. Our intuition is that if a specific
attack presents temporal patterns, then such patterns should not depend on the
benign traffic. Hence, we propose to isolate the specific attack types and,
for each type, focus on the time-interval containing the malicious samples.

However, many obstacles must be overcome to increase the likelihood that the
extracted scenarios are effective for detection.

A well-known issue in the application of ML-based approaches for NIDS is the
strong unbalancing between the normal ‘benign’ traffic, and the malicious
traffic. Indeed, contrarily to other domains where ML is applied, in
cybersecurity the illegitimate samples represent rare events (e.g., (Apruzzese
et al., 2017; Chuang and Wu, 2019)) which are several orders of magnitude
inferior to the normal events occurring in a network. A strong data imbalance
affects all types of ML applications; however, our time-sensitive context
further aggravates this problem, because finding temporal patterns in the
minority class is more difficult.

Therefore, in designing our proposed methodology for extracting the attack
scenarios, we must first mitigate the unbalancing in the data. By doing so, we
can improve the quality (in terms of detection efficacy) of the extracted
temporal patterns. To mitigate the unbalancing problem, we leverage two
observations:

  1. (1)

the majority of the benign traffic that skews the distribution of a dataset
presents similar characteristics, i.e., same protocols or same (service)
ports555This is the assumption underlying anomaly-based NIDS, where a
representative model of the benign network traffic is constructed through
statistical analyses (Khraisat et al., 2019; Mishra et al., 2018). , and can
be partially discarded to increase the ‘weight’ of the malicious patterns;

  2. (2)

some attacks exhibit periodic behaviours (also referred to as beaconing
(Apruzzese et al., 2017)), and are executed in time intervals that, if found,
can be used to “split” the scenario.

By combining these two observations, we propose to identify one or more
separate scenarios for each attack type; such scenarios contain only those
samples (benign and malicious) generated at the precise time intervals. Hence,
we focus on pinpointing one or more time intervals for each attack, and then
create a dedicated scenario for each time interval by including only those
NetFlow samples occurring within the corresponding intervals. The remaining
traffic samples are then discarded.

Figure 1. Example of attack scenarios creation. Any NetFlow not included in
the attack scenarios is discarded.

Let us illustrate our idea with a concrete example, graphically depicted in
Figure 1. Here, we assume that there exists a “Botnet Scenario” that starts
from time $t+5$ (denoted with the NetFlow $x_{t+5}$), and ends at time $t+14$
(denoted with the NetFlow $x_{t+14}$). Our approach retains all NetFlows
occurring within such time interval (spanning over 10 time-steps), regardless
of their nature (benign or malicious). Specifically, the extracted “Botnet
Scenario” will include two botnet NetFlows ($x_{t+5}$, $x_{t+6}$) followed by
two benign NetFlows ($x_{t+7}$, $x_{t+8}$), and then by, in order, another
botnet NetFlow ($x_{t+9}$), two benign NetFlows ($x_{t+10}$, $x_{t+11}$), a
botnet NetFlow ($x_{t+12}$), a benign NetFlow ($x_{t+13}$), and the final
botnet NetFlow ($x_{t+14}$) The NetFlows not included in such scenario (e.g.,
those from $x_{t}$ to $x_{t+4}$ and from $x_{t+15}$ to $x_{t+18}$) are
discarded. Then, at time $t+19$, the “SQL Injection Scenario” begins.

The time intervals of an attack can be identified by analyzing the temporal
distribution of malicious NetFlows (e.g., through the use of autocorrelation
(Shirani et al., 2015)). However, such intervals can be obtained also from a
dataset documentation (e.g., (Sharafaldin et al., 2018)).

When identifying the time-intervals of the attack scenarios, it is important
to avoid ‘removing’ excessive amounts of benign samples: such occurrences may
lead to oversimplified attack scenarios that are not sufficiently
representative for detection purposes. If the extracted scenarios do not
include at least some benign samples, we recommend increasing the length of
the time-intervals to include more benign activities; alternatively, it is
even possible to create dedicated ‘benign’ scenarios. We anticipate that, in
our evaluation, we demonstrate that this latter technique can be used without
affecting the learning phase of the sequential models.

Nonetheless, there may also exist some circumstances that lead to scenarios
with an overabundance of benign samples (e.g., long duration attacks such as
DDoS). Completely solving this issue is unfeasible. Hence, to mitigate this
problem, we recommend to configure the loss function (i.e., the weights) of
the ML model to assign a heavier penalty to misclassifications of malicious
scenarios.

Let us explain the motivation behind such design choices. The ML literature
proposes many techniques for dealing with imbalanced datasets. Some prominent
examples involve: under-sampling the majority class or over-sampling the
minority class (Cieslak et al., 2006); Synthetic Minority Over-sampling
Technique (SMOTE) (e.g., (Zhang et al., 2020)); and weighing the loss
function. However, with the exception of weighing of the loss function (which
is the approach adopted in the proposed method), all these techniques
artificially modify the input data and cannot be readily applied. Indeed,
manipulating the network data with under/over-sampling may lead to biased and
overfit models that learn unrealistic patterns, which may present good results
on the ‘modified’ source dataset, but with impractical performance in
realistic applications. We argue that these techniques are effective for
training successful ML models that do not take into account temporal
relationships. However, in our time-sensitive context, we have to ensure that
the generation of synthetic data - or the removal of real data - does not
interfere with the actual temporal patterns that denote the network
activities.

Our method is among the first attempts in this domain, and presents much room
for improvement. The major advantage of the proposed method, that is, creating
specific attack scenarios, is that it should improve the quality of the
learning phase by reducing the noise caused by the benign traffic. However, we
stress that there is no guarantee that the extracted scenarios capture
meaningful temporal relationships that are truly useful for detection
purposes. This is why we must resort on ML methods to validate if the proposed
method is effective or not.

###  3.3. Sequential ML for NIDS

To assess the quality of the temporal patterns hopefully contained in the
attack scenarios (described in Section 3.2), we employ the de-facto sequence
model for practical applications, _gated RNN_ (Dey and Salem, 2017). Among the
possible architectures of gated RNNs, we chose the Long Short-Term Memory
(LSTM), due to its successful results achieved in literature (e.g., (Yeom et
al., 2021; Mirza and Cosan, 2018)).

The successful principle at the base of the LSTM is an internal and context-
aware self-loop, which dynamically adjusts its time scale on the basis of the
sequence provided as input (Sainath et al., 2015). The ability of
automatically adjusting the time scale makes LSTM a suitable candidate for our
objective. Indeed, we stress that our method does not know whether temporal
patterns (among malicious NetFlow data) exist or not. Moreover, even if such
patterns exist, the method does not know when they occur and how long they
last. Hence, we require a technique that is able to learn both long-term and
short-term dependencies in the input scenarios. As an additional benefit, LSTM
are among the best models for dealing with the so called vanishing gradient
problem (Fei and Tan, 2018), which is a major obstacle when training RNN
methods.

We propose the use of an unidirectional $n$-stacked version of the LSTM. We
provide a schematic depiction of such architecture (in a 2-stacked version) in
Figure 2, which reports the exemplary case of the Botnet attack scenario shown
in Figure 1. As described in (Pascanu et al., 2013), having a stacked version
of a RNN helps the model to look at the input data from different time-scales.
This is possible since the input of each layer is the hidden state of the
previous layer, with the sole exception of the very first layer which receives
the actual input $x_{t}$.

Figure 2. Exemplary architecture of a 2-stacked LSTM.

As shown in Figure 2, in a $n$-stacked version of a LSTM, the input at time-
step $t$ is fed into the first layer, and it is then used (alongside the
hidden state of the previous time-step $h_{t-1}^{1}$) to produce the Cell
state $c_{t}^{1}$ (we use the superscript for identifying each layer). Then,
for all the upper $n-1$ layers, the input at time-step $t$ is the hidden state
of the previous layer, that is, for a layer $i\geq 2$, the input at time-step
$t$ is $h_{t}^{i-1}$. This mechanism allows to bring the actual input into
different time scales before making the final inference.

A common problem in any ML approach is the risk of realizing an overfit model.
Depending on the input data and the chosen ML algorithm, different techniques
can be used to remove (or mitigate) such issues (e.g., (Jindal et al., 2016)).
Similarly, the activation function of the LSTM layers can be chosen
arbitrarily: common solutions are, e.g., the sigmoid or the rectified linear
unit (ReLU).

The model is used in a many-to-many fashion, that is, each NetFlow in a
scenario is classified as either benign or malicious. This final
classification is achieved through a single fully-connected neuronal layer
placed after the last LSTM layer.

##  4\. Experimental Settings

We experimentally evaluate the proposed methodology. Recall the twofold goal
of our paper: investigating the existence of malicious temporal patterns in
NetFlows; and verifying whether sequential learning models really learn and
benefit from these patterns. To this purpose, we need to first devise a
sequential learning model using some input data, and then compare its
performance against a baseline ‘static’ ML model (using the same input data),
but that does not consider any temporal relationship.

We describe the experimental settings to highlight the realistic value of our
testbed. We present the chosen datasets in Section 4.1, the implementation of
the proposed method and the realization of the final detectors in Section 4.2.

###  4.1. Datasets and Preprocessing

We base our evaluation on two well-known and labelled datasets of enterprise
network traffic that include different types of attacks and that are
appreciated in the NIDS domain: the CTU13 (Garcia et al., 2014) and the
CICIDS2017 (Sharafaldin et al., 2018). We present the motivations of our
choice below.

CTU13. This dataset is extensively used in literature (see Table 1) and
contains thirteen collections of real network traffic data (in the form of
NetFlows) mixed with traffic generated by different botnet families. We choose
the CTU13 for two reasons that make it as a valid benchmark for realistic
evaluations. First, in each collection, a specific botnet family performs
multiple malicious actions (e.g., Port Scan, Click Fraud, DDoS); hence, each
collection embeds a large variance of malicious events–and potentially many
different temporal patterns. Second, the volume of benign traffic with respect
to the malicious one is several orders of magnitude greater, making the
dataset highly imbalance.

We perform a preliminary analysis of the CTU13. We discover that the majority
of the benign traffic in CTU13 is produced by 20 internal hosts and involve
few services. Hence, we exclude the NetFlows (which are all benign) of these
hosts to partially mitigate the unbalancing problem. We recall that our focus
is investigating the existence of temporal patterns among malicious NetFlows,
and removing such samples does not interfere with our methodology.
Furthermore, these hosts can be seen as more critical to the organization, and
in real networks they would be protected by dedicated defensive mechanisms,
therefore aligning with the assumptions made in our threat model (see Section
3.1). As successfully done in related NIDS literature (e.g., (Apruzzese et
al., 2020)), we select the following NetFlow features666Note that our chosen
features extend the basic essential features required by our method (see
Section 3.2).: duration, source/destination ports, direction, protocol, total
packets, packets per second, total amount of bytes, amount of source bytes,
ratio of bytes per second, ratio of bytes per packet.

CICIDS2017. This benchmark dataset contains synthetic benign traffic and
common types of attacks, with the peculiarity that different types of attacks
are carried out in distinct time intervals. We choose the CICIDS2017 because,
with respect to the CTU13, it is more recent, it has a larger variety of
attacks and it is less class imbalance. Moreover, its documentation specifies
the time interval of each type of attack, which we leverage for our method.

We conduct an exploratory analysis of the CICIDS2017, where we determine the
features for our experiments: duration, protocol, source/destination IP (as
either internal or external to derive the direction of the connection), total
packets, source/destination ports, total amount of bytes, ratio of packets per
second, ratio of bytes per packet, minimum/maximum/average inter arrival time.
This feature set differs from the one adopted in the CTU13, but it includes
the features required by our approach.

Table 2. The distribution of benign and malicious NetFlows in the datasets
before and after the application of the method described in Section 3.2. In
the table, #B stands for number of benign and #M for number of malicious.

| CICIDS 2017
---
NetFlows in the dataset | NetFlows in processed scenarios
File | # NetFlows | # Mal | Scenario | #B train | #M train | #B test | #M test
|  |  | Benign1 | 191235 | 0 | 81958 | 0
Day 1 | 516243 | 0 | Benign2 | 170135 | 0 | 72915 | 0
|  |  | Botnet1 | 105288 | 1460 | 45538 | 212
|  |  | Botnet2 | 9607 | 107 | 4019 | 145
|  |  | Port Scan | 88585 | 117607 | 47173 | 41196
Day 2 | 687858 | 284758 | DDoS | 42946 | 101443 | 39294 | 22588
|  |  | DoS GoldenEye | 10202 | 7690 | 24456 | 2384
|  |  | DoS Hulk | 16802 | 153596 | 14358 | 58670
|  |  | DoS slowhttptest | 19655 | 3974 | 14227 | 1527
Day 3 | 663612 | 233635 | DoS slowloris | 54111 | 3878 | 8323 | 1911
|  |  | FTP-Patator | 73355 | 5027 | 23262 | 2866
Day 4 | 436088 | 13782 | SSH-Patator | 55133 | 3522 | 22772 | 2366
|  |  | Web Brute Force | 43062 | 867 | 18188 | 640
Day 5 | 448382 | 2214 | Web XSS | 15984 | 417 | 6794 | 235
Summary | 2752183 | 534389 | Summary | 896100 | 399588 | 423277 | 134740
| CTU 13
---
NetFlows in the dataset | NetFlows in processed scenarios
File | # NetFlows | # Mal | Scenario | #B train | #M train | #B test | #M test
Menti | 558919 | 4630 | Menti | 62306 | 2986 | 26369 | 1614
Murlo | 2954230 | 6126 | Murlo | 273249 | 3641 | 116251 | 2417
Neris1 | 2824636 | 40959 | Neris1 | 156026 | 24280 | 68013 | 9261
Neris2 | 1808122 | 20941 | Neris2 | 99614 | 16821 | 46569 | 3333
Neris3 | 2753884 | 184979 | Neris3 | 86236 | 51890 | 33908 | 25290
Nsisay | 325471 | 2168 | Nsisay | 32228 | 1602 | 14140 | 359
Rbot1 | 4710638 | 26822 | Rbot1 | 317008 | 12451 | 126869 | 14329
Rbot2 | 1121076 | 1768 | Rbot2 | 53924 | 825 | 22288 | 1177
Rbot3 | 1309791 | 106352 | Rbot3 | 112684 | 74743 | 48939 | 31388
Rbot4 | 107251 | 8164 | Rbot4 | 5598 | 5070 | 1500 | 3073
Virut1 | 129832 | 901 | Virut1 | 8331 | 549 | 3501 | 306
Virut2 | 1925149 | 39993 | Virut2 | 141886 | 23225 | 62048 | 8714
Summary | 20528999 | 443803 | Summary | 1349090 | 218083 | 570395 | 101261

###  4.2. Implementation and Detectors

After preprocessing the datasets, the next step is the extraction of the
attack scenarios with the proposed method (see Section 3.2). Its application
produces a set of attack scenarios, which we summarize in Table 2. Let us
describe the table by focusing on its leftmost part: the dataset is the
CICIDS2017 for which precise information on the time interval of the attacks
is available. We use such information to create the scenarios. For instance,
_Day 2_ is split into 4 scenarios (Botnet1, Botnet2, Port Scan, and DDoS). We
observe that our method reduces the amount of benign NetFlows: in _Day 2_ they
decrease from $687$k to $382$k. Furthermore, instead of dropping Day 1, we use
its benign traffic to create two benign scenarios to demonstrate that their
use can yield better performance.

To assess the quality of the extracted patterns for detection purposes, we
devise a baseline detector that relies only on the “static” features of
NetFlows for the final inference, that is, it does not take into account any
temporal relationship. The rationale is that, if the baseline ‘static’
detector and the proposed ‘sequential’ detector have comparable overall
performance, but their specific classification patterns differ, we can prove
that temporal patterns among malicious NetFlows indeed exist in our scenarios.

For the sequential detector, we use a 2-stacked LSTM. To mitigate overfitting,
the LSTM leverages the dropout technique to yield a smoother model that is
less sensitive to small variations in the (unseen) input data. Moreover, we
use the sigmoid as activation function. The LSTM has the same structure as in
Figure 2.

For the static detector, we consider a Feedforward Neural Network (FNN). To
avoid bias and favor a fair comparison, we build the FNN with the same number
of hidden layers of the LSTM; we apply the dropout mechanism after each layer,
and we use ReLU as activation function. We report such FNN architecture in
Figure 3, which can be directly compared to the LSTM in Figure 2.

We observe that both the LSTM and the FNN project the NetFlows in two latent
spaces, the blue ellipsis. While the LSTM projects sequences of NetFlows in
two distinct and subsequent time scales, the FNN separately transforms each
NetFlow into two subsequent projection spaces. The models should have a
similar expressiveness (i.e., optimization functions) which is beneficial for
a fair comparison.

Figure 3. FNN architecture.

We focus on binary classification, that is, each sample can be predicted to be
either benign or malicious. We consider two datasets, so we devise a total of
four models, i.e., one FNN and one LSTM per dataset. We report in Table 2, the
partitioning of the training and testing sets. We use the first $70\%$ of each
scenario for training and the remaining for testing (to avoid temporal bias
(Pendlebury et al., 2019)); the only exceptions are some scenarios of
CICIDS2017, where we must resort to different splits to ensure at that at
least $70\%$ of the malicious NetFlows are put in the training set. We remark
that both the LSTM and FNN are trained on the same training data, and tested
on the same test data. Note also that we keep the Benign1-2 scenarios of
CICIDS2017 only for training the models.

##  5\. Evaluation and Results

The goal of the experiments is verifying the existence and effectiveness of
the temporal patterns extracted with the proposed method; we do not want to
propose a superior technique. Hence, we train777We report more experimental
details in Appendix A. the LSTM and FNN on each dataset (as described in
Section 4.2), and we perform two types of experiments, which we briefly
outline.

Direct comparison. Here, the focus is studying the classification patterns
learned by the two models (LSTM and FNN). We do so by analyzing the results of
both models on the same testing set. The idea is that different performance
implies different learned patterns used for detection. We present and discuss
the results of these experiments in Section 5.1.

Ensemble comparison. To truly verify if the two models (FNN and LSTM) learn
different patterns, we create a third ‘ensemble’ model that includes both
models. More specifically, we join the FNN with the LSTM model through the
logical or. Then, we compare this ensemble (denoted as FNN+LSTM) with the
stand-alone LSTM model, and test them on the same test data. The intuition is
as follows: if we combine the LSTM with the FNN and the performance matches
the performance of the stand-alone LSTM, then it will mean that the FNN and
LSTM learn the same detection patterns (and vice versa). These experiments are
discussed in Section 5.2.

###  5.1. Initial Assessment (FNN vs LSTM)

We first compare the performance of the sequential LSTM against the static
FNN. We evaluate the two classifiers on every scenario of each dataset. We
report the results in Table 3, showing the performance metrics of interest:
the F1-score; and the complete confusion matrix, where fp (fn) stands for
false positives (negatives), and tp (tn) stands for true positives
(negatives). We consider a positive as a malicious sample. We split the
classification results for each scenario to allow a more fine-grained
comparison of the two approaches. The green rows report the aggregated results
on the corresponding dataset.

Table 3. Results of the LSTM and FNN on the two datasets.

|  | CICIDS 2017
---|---
| Scenario | F1-score % | tp | fn | tn | fp
| Botnet1 | 81.755 | 177 | 35 | 45494 | 44
| Botnet2 | 87.542 | 130 | 15 | 3997 | 22
| Port Scan | 99.558 | 41179 | 17 | 46824 | 349
| DDoS | 99.572 | 22588 | 0 | 39100 | 194
| DoS GoldenEye | 99.707 | 2384 | 0 | 24442 | 14
| DoS Hulk | 99.955 | 58653 | 17 | 14322 | 36
| DoS slowhttptest | 98.444 | 1487 | 40 | 14220 | 7
| DoS slowloris | 99.843 | 1910 | 1 | 8318 | 5
| FTP-Patator | 99.600 | 2866 | 0 | 23239 | 23
| SSH-Patator | 98.666 | 2366 | 0 | 22708 | 64
| Web Brute Force | 99.456 | 640 | 0 | 18181 | 7
| Web XSS | 99.153 | 234 | 1 | 6791 | 3
Long Short-Term Memory | Summary | 99.669 | 134614 | 126 | 267636 | 768
| CTU 13
---
Scenario | F1-score % | tp | fn | tn | fp
Menti | 95.153 | 1600 | 14 | 26220 | 149
Murlo | 95.640 | 2369 | 48 | 116083 | 168
Neris1 | 93.384 | 8793 | 468 | 67235 | 778
Neris2 | 90.524 | 3162 | 171 | 46078 | 491
Neris3 | 95.911 | 24901 | 389 | 32174 | 1734
Nsisay | 41.445 | 109 | 250 | 14082 | 58
Rbot1 | 98.105 | 14080 | 249 | 126574 | 295
Rbot2 | 94.873 | 1101 | 76 | 22245 | 43
Rbot3 | 99.909 | 31352 | 36 | 48918 | 21
Rbot4 | 100.000 | 3073 | 0 | 1500 | 0
Virut1 | 87.117 | 284 | 22 | 3439 | 62
Virut2 | 80.851 | 6863 | 1851 | 60648 | 1400
Summary | 95.703 | 97687 | 3574 | 565196 | 5199
|  | Botnet1 |  93.735 | 202 | 10 | 45521 | 17
---|---|---|---|---|---|---
| Botnet2 |  81.569 | 104 | 41 | 4013 | 6
| Port Scan |  99.773 | 41152 | 44 | 47030 | 143
| DDoS |  99.817 | 22588 | 0 | 39211 | 83
| DoS GoldenEye |  99.916 | 2384 | 0 | 24452 | 4
| DoS Hulk |  100.000 | 58670 | 0 | 14358 | 0
| DoS slowhttptest |  98.303 | 1477 | 50 | 14226 | 1
| DoS slowloris |  99.974 | 1911 | 0 | 8322 | 1
| FTP-Patator |  99.930 | 2866 | 0 | 23258 | 4
| SSH-Patator |  99.705 | 2366 | 0 | 22758 | 14
| Web Brute Force |  99.844 | 640 | 0 | 18186 | 2
| Web XSS |  99.788 | 235 | 0 | 6793 | 1
Feedforward Neural Network | Summary |  99.844 | 134595 | 145 | 268128 | 276
| Menti |  84.960 | 1610 | 4 | 25803 | 566
---|---|---|---|---|---
Murlo |  63.267 | 2413 | 4 | 113453 | 2798
Neris1 |  85.188 | 9087 | 174 | 65027 | 2986
Neris2 |  84.533 | 3189 | 144 | 45546 | 1023
Neris3 |  97.353 | 24605 | 685 | 33255 | 653
Nsisay |  44.673 | 174 | 185 | 13894 | 246
Rbot1 |  84.926 | 14299 | 30 | 121823 | 5046
Rbot2 |  79.619 | 1170 | 7 | 21696 | 592
Rbot3 |  98.688 | 31355 | 33 | 48138 | 801
Rbot4 |  99.854 | 3073 | 0 | 1491 | 9
Virut1 |  75.943 | 292 | 14 | 3330 | 171
Virut2 |  87.553 | 8047 | 667 | 60427 | 1621
Summary |  91.497 | 99314 | 1947 | 553883 | 16512

Let us discuss these results, starting from the CICIDS2017 dataset. By looking
at the overall results (green rows), the performance of both LSTM and FNN is
similar, and comparable to the state of the art (e.g., (Zhang et al., 2019;
Han et al., 2020)). However, by inspecting the performance in the individual
scenarios, we can observe some interesting phenomena. Consider the two Botnet
scenarios of CICIDS2017 (highlighted in yellow): here, the LSTM learns
different classification patterns than the FNN, achieving $81\%$ ($87\%$)
F1-score against a $93\%$ ($81\%$) in the Botnet1 (Botnet2) scenario. What is
remarkable in these two scenarios is the fact that the malicious samples are
from the same botnet type. If we turn the attention to the _DoS slowloris_ and
the _Port Scan_ scenarios, the LSTM has almost the same fn of the FNN in the
_DoS slowloris_ scenario, but fewer in the _Port Scan_ scenario. These two
different types of attacks further highlight that, against some attacks, the
LSTM appears to be more effective. The intriguing observation is that a Port
Scan typically generates a large volume of (malicious) traffic in a short
timeframe; on the contrary, the DoS slowloris generates a small volume of
traffic but in long timeframes. Despite these differences, the LSTM seems to
adjust its internal time scale (see Section 3.3) for achieving superior
performance.

We now focus on the CTU13. Here, on average, both models achieve good
performance with over $91\%$ F1-score, in-line with the state of the art
(e.g., (Apruzzese et al., 2020)). The LSTM performs better than the FNN in
terms of F1-score, and it exhibits a lower amount of fp, but also slightly
higher fn, making it a more precise detector for practical purposes. Let us
observe the results for the individual scenarios. In the Nsisay scenario both
models achieve similar poor performance (below $45\%$ F1-score). The most
remarkable differences involve the _Neris3_ and _Virut2_ scenarios
(highlighted in yellow). In _Neris3_ , the LSTM has a lower number of false-
negatives, but it also has a larger number of false-positives; on the other
hand, _Virut2_ is the only scenario where the FNN outperforms the LSTM, and
this is clear from the F1-scores of the models. Indeed, the Virut2 scenario is
the most ‘problematic’ scenario for the LSTM due to the huge fn. However,
everywhere else the LSTM learns different classification patterns that produce
superior detection results.

Takeaway: the different performance in the individual scenarios on both
datasets may suggest that the LSTM learns different classification patterns
than the FNN. Hence, for some specific attacks, the application of the
proposed method (and its usage for LSTM-detectors) may yield better NIDS.

###  5.2. Verification (FNN+LSTM vs LSTM)

Despite the differences shown in the previous experiments, we still cannot be
truly sure that the LSTM and FNN have learned different patterns for their
classification888The objective is verifying if the temporal relationships are
truly useful for ML-NIDS.. As an example, consider the case of Botnet1 in the
CICIDS2017 dataset: the FNN has 10 fn, while the LSTM has 35 fn. The question
is: are these 10 fn by the FNN also included in the 35 fn by the LSTM? If this
is true, then the two models will have learned (very) similar classification
patterns; on the other hand, if this is not true, then the two models will
have learned (arguably) different classification patterns.

To this purpose, we join the LSTM model with the FNN model with the logical or
operator to create a new ensemble (FNN+LSTM). More specifically, for a given
test-sample as input, the output will be malicious if at least one model of
the ensemble makes a malicious classification; conversely, the output will be
benign only if both models agree on a benign classification. Such design
choice should reduce the number of fn. We do not retrain the models of the
ensemble: we simply input a (test) sample to each trained model, and then join
the output with the logical or.

We test the LSTM+FNN ensemble on both datasets and report the results in Table
4. For each dataset, the three columns of the FNN+LSTM report the F1-score and
the number of fn and fp for each attack scenario; we summarize the overall
results in the last row (green row). The two rightmost columns report the
difference ($\Delta$) of each performance metric between the FNN+LSTM and the
stand-alone LSTM: hence, in the $\pm$fn and $\pm$fp columns, a negative
(positive) value means that the ensemble LSTM+FNN is better (worse) than the
stand-alone LSTM.

Table 4. Results of the FNN+LSTM ensemble, and comparison ($\Delta$) with the
stand-alone LSTM.

| CICIDS2017
---
| Testing data | FNN+LSTM | $\Delta$
Scenario | # NetFlows | # Mal | F1-score % | fn | fp |  $\pm$fn |  $\pm$fp
Botnet1 | 45750 | 212 | 89.936 | 2 | 45 | -33 | 1
Botnet2 | 4164 | 145 | 91.558 | 4 | 22 | -11 | 0
Port Scan | 88369 | 41196 | 99.556 | 9 | 358 | -8 | 9
DDoS | 61882 | 22588 | 99.572 | 0 | 194 | 0 | 0
Dos GoldenEye | 26840 | 2384 | 99.707 | 0 | 14 | 0 | 0
DoS Hulk | 73028 | 58670 | 99.969 | 0 | 36 | -17 | 0
DoS slowhttptest | 15754 | 1527 | 99.443 | 10 | 7 | -30 | 0
DoS slowloris | 10234 | 1911 | 99.869 | 0 | 5 | -1 | 0
FTP-Patator | 26128 | 2866 | 99.600 | 0 | 23 | 0 | 0
SSH-Patator | 25138 | 2366 | 98.666 | 0 | 64 | 0 | 0
Web Brute Force | 18828 | 640 | 99.456 | 0 | 7 | 0 | 0
Web XSS | 7029 | 235 | 99.366 | 0 | 3 | -1 | 0
Summary | 403144 | 134740 | 99.703 | 25 | 778 | -101 | 10
| CTU13
---
| Testing data | FNN+LSTM | $\Delta$
Scenario | # NetFlows | # Mal | F1-score % | fn | fp |  $\pm$fn |  $\pm$fp
Menti | 27983 | 1614 | 82.688 | 2 | 673 | -12 | 524
Murlo | 118668 | 2417 | 62.562 | 4 | 2884 | -44 | 2716
Neris1 | 77274 | 9261 | 84.099 | 130 | 3323 | -338 | 2545
Neris2 | 49902 | 3333 | 82.624 | 33 | 1355 | -138 | 864
Neris3 | 59198 | 25290 | 95.938 | 86 | 2048 | -303 | 314
Nsisay | 14499 | 359 | 50.228 | 139 | 297 | -111 | 239
Rbot1 | 141198 | 14329 | 84.654 | 11 | 5180 | -238 | 4885
Rbot2 | 23465 | 1177 | 79.268 | 7 | 605 | -69 | 562
Rbot3 | 80327 | 31388 | 98.664 | 33 | 816 | -3 | 795
Rbot4 | 4573 | 3073 | 99.854 | 0 | 9 | 0 | 9
Virut1 | 3807 | 306 | 74.568 | 4 | 202 | -18 | 140
Virut2 | 70762 | 8714 | 87.040 | 148 | 2403 | -1703 | 1003
Summary | 671656 | 101261 | 90.803 | 597 | 19795 | -2977 | 14596

Let us discuss Table 4, starting from the CICIDS2017 results. Here, we
highlight (in yellow) _Botnet1_ and _DoS slowhttptest_ scenarios because they
show the greatest improvement of the FNN+LSTM ensemble over the stand-alone
LSTM. In these scenarios, the ensemble has lower fn than the LSTM, and the
same fp (only 1 more for the Botnet1). This is possible because the FNN used
in the ensemble correctly identifies the malicious NetFlows that are
misclassified by the LSTM. We observe an interesting phenomenon in four
scenarios, i.e., _DoS slowhttptest_ , _Botnet1-2_ and _Port Scan_. The
ensemble has not only lower fn than the stand-alone LSTM, but also lower than
the stand-alone FNN (see Table 3). This means the ensemble also benefits from
the LSTM because it correctly identifies malicious NetFlows misclassified by
the FNN.

We now focus on the CTU13. Here, the FNN+LSTM ensemble does better than the
stand-alone LSTM in terms of number of false-negatives. However, the latter
has a higher F1-score, i.e., $95.703\%$ vs $90.803\%$. This is due to the more
conservative nature of the ensemble in detecting malicious events, which may
reduce the fn, but may increase the fp. A more in-depth analysis of each
scenario outlines that the ensemble reduces the fn in almost all cases.
Notably, in _Neris1_ and _Virut2_ scenarios (in yellow) the FNN+LSTM ensemble
has significantly lower fn than the stand-alone LSTM, but it also has over
$2$k more fp. The rationale is that the FNN is better in detecting malicious
NetFlows, which benefits the FNN+LSTM ensemble. However, we also observe that
the ensemble also benefits from the LSTM. This is clear if we compare the
overall number of false-negatives of the FNN+LSTM ensemble with those of the
stand-alone FNN (see Table 3): the former has only 597, while the latter has
almost four times that amount with 1947.

Takeaway: the LSTM and FNN appear to have learned different classification
patterns, confirming that the application of our method to sequential ML
models does influence their detection.

##  6\. Conclusions

In this work, we investigate the use of Long Short-Term Memory (LSTM) neural
networks to learn temporal patterns among NetFlows as a result of cyber-
attacks. We review and highlight the limitations of related work, which has
not truly studied the effectiveness of temporal patterns for NetFlow-based
NIDS. Hence, we propose an original method that can be used to extract
temporal patterns from labelled NetFlow data. We then apply this method to
train a ‘sequential’ LSTM classifier, and compare its performance against a
‘static’ Feedforward Neural Network (FNN) to verify if temporal patterns are
truly significant for ML-NIDS.

Our evaluation spans over two recent ML-NIDS datasets, CICIDS2017 and CTU13.
The results highlight that LSTM achieves comparable ($\sim\\!99\%$ F1-score)
or better ($95\%$ vs $91\%$ F1-score) performance in detecting malicious
NetFlows than the FNN. However, against some specific attacks, the LSTM proves
to be significantly better, yielding lower false negatives and lower false
positives.

We verify if the proposed method influences the detection by creating an
ensemble of the FNN+LSTM models, and compare it against the stand-alone LSTM.
This additional set of experiments further confirms that the LSTM and FNN
models learn different classification patterns, which translate to different
performance that can be exploited to mitigate some types of attacks. Such
comparison confirms that, if properly extracted (e.g., with the proposed
method and sequential ML-models), temporal patterns can truly be effective at
enhancing the performance of ML-NIDS.

Our paper paves the way to more secure and effective NIDS, which rely not only
on ‘static’ ML methods, but also by sequential approaches that leverage the
underlying relationships among distinct network traffic samples.

## References

  * (1)
  * Apruzzese et al. (2020) Giovanni Apruzzese, Mauro Andreolini, Mirco Marchetti, Andrea Venturi, and Michele Colajanni. 2020\.  Deep reinforcement adversarial learning against botnet evasion attacks.  _IEEE T. Netw. Serv. Manag._ 17, 4 (2020), 1975–1987.
  * Apruzzese et al. (2017) Giovanni Apruzzese, Mirco Marchetti, Michele Colajanni, Gabriele Gambigliani Zoccoli, and Alessandro Guido. 2017.  Identifying malicious hosts involved in periodic communications. In _Proc. IEEE Int. Symp. Netw. Comp. Appl._ 1–8.
  * Boukhalfa et al. (2020) Alaeddine Boukhalfa, Abderrahim Abdellaoui, Nabil Hmina, and Habiba Chaoui. 2020.  LSTM deep learning method for network intrusion detection system.  _Int. J. Elec. Comp. Eng._ 10 (2020).
  * Buczak and Guven (2016) Anna L Buczak and Erhan Guven. 2016.  A Survey of Data Mining and Machine Learning Methods for Cyber Security Intrusion Detection.  _IEEE Comm. Surv. Tut._ 18, 2 (2016), 1153–1176.
  * Chuang and Wu (2019) Po-Jen Chuang and Dong-Ye Wu. 2019.  Applying Deep Learning to Balancing Network Intrusion Detection Datasets. In _Proc. Int. Conf. Adv. InfoComm. Tech._ 213–217.
  * Cieslak et al. (2006) DA Cieslak, NV Chawla, and A Striegel. 2006.  Combating imbalance in network intrusion datasets. In _Proc. IEEE Int. Conf. Granular Comp._ 732–737.
  * Cisco (2021) Cisco. 2021.  _IOS NetFlow._ Technical Report.  https://www.ciscopress.com/articles/article.asp?p=2812391&seqNum=3
  * Dey and Salem (2017) Rahul Dey and Fathi M. Salem. 2017.  Gate-variants of Gated Recurrent Unit (GRU) neural networks. In _Proc. IEEE Int. Midwest Symp. Circ. Syst._ 1597–1600.
  * Fei and Tan (2018) Hongxiao Fei and Fengyun Tan. 2018.  Bidirectional grid long short-term memory (bigridlstm): A method to address context-sensitivity and vanishing gradient.  _Algorithms_ 11, 11 (2018), 172.
  * Garcia et al. (2014) Sebastian Garcia, Martin Grill, Jan Stiborek, and Alejandro Zunino. 2014.  An empirical comparison of botnet detection methods.  _Elsevier Comp. Secur._ 45 (2014), 100–123.
  * Gwon et al. (2019) Hyeokmin Gwon, Chungjun Lee, Rakun Keum, and Heeyoul Choi. 2019\.  Network Intrusion Detection based on LSTM and Feature Embedding.  _arXiv:1911.11552_ (2019).
  * Han et al. (2020) Xueying Han, Rongchao Yin, Zhigang Lu, Bo Jiang, Yuling Liu, Song Liu, Chonghua Wang, and Ning Li. 2020\.  STIDM: A Spatial and Temporal Aware Intrusion Detection Model. In _Proc. IEEE Int. Conf. Trust, Secur. Privacy Comp. Commun_. 370–377.
  * Jindal et al. (2016) Ishan Jindal, Matthew Nokleby, and Xuewen Chen. 2016\.  Learning deep networks from noisy labels with dropout regularization. In _Proc. IEEE Int. Conf. Data Mining_. 967–972.
  * Khan et al. (2019) Muhammad Ashfaq Khan, Md Karim, Yangwoo Kim, et al. 2019\.  A scalable and hybrid intrusion detection system based on the convolutional-LSTM network.  _Symmetry_ 11, 4 (2019), 583.
  * Khraisat et al. (2019) Ansam Khraisat, Iqbal Gondal, Peter Vamplew, and Joarder Kamruzzaman. 2019.  Survey of intrusion detection systems: techniques, datasets and challenges.  _Springer Cybersecurity_ 2, 1 (2019), 1–22.
  * Kim et al. (2020) Jeeyung Kim, Alex Sim, Jinoh Kim, and Kesheng Wu. 2020\.  Botnet Detection Using Recurrent Variational Autoencoder.  _arXiv:2004.00234_ (2020).
  * Liao et al. (2013) Hung-Jen Liao, Chun-Hung Richard Lin, Ying-Chih Lin, and Kuang-Yuan Tung. 2013.  Intrusion detection system: A comprehensive review.  _Elsevier J, Netw, Comp, Appl._ 36, 1 (2013), 16–24.
  * Mirza and Cosan (2018) Ali H Mirza and Selin Cosan. 2018.  Computer network intrusion detection using sequential LSTM neural networks autoencoders. In _Proc. IEEE Sign. Proc. Commun. Appl. Conf._ 1–4.
  * Mishra et al. (2018) Preeti Mishra, Vijay Varadharajan, Uday Tupakula, and Emmanuel S Pilli. 2018.  A detailed investigation and analysis of using machine learning techniques for intrusion detection.  _IEEE Comm. Surv. Tut._ 21, 1 (2018), 686–728.
  * Pascanu et al. (2013) Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2013.  How to construct deep recurrent neural networks.  _arXiv:1312.6026_ (2013).
  * Pektaş and Acarman (2019) Abdurrahman Pektaş and Tankut Acarman. 2019.  Deep learning to detect botnet via network flow summaries.  _Springer Neur. Comp. Appl._ 31, 11 (2019), 8021–8033.
  * Pendlebury et al. (2019) Feargus Pendlebury, Fabio Pierazzi, Roberto Jordaney, Johannes Kinder, and Lorenzo Cavallaro. 2019\.  $\\{$TESSERACT$\\}$: Eliminating experimental bias in malware classification across space and time. In _Proc. USENIX Secur. Symp.)_. 729–746.
  * Radford et al. (2018a) Benjamin J Radford, Leonardo M Apolonio, Antonio J Trias, and Jim A Simpson. 2018a.  Network traffic anomaly detection using recurrent neural networks.  _arXiv:1803.10769_ (2018).
  * Radford et al. (2018b) Benjamin J Radford, Bartley D Richardson, and Shawn E Davis. 2018b.  Sequence aggregation rules for anomaly detection in computer network traffic.  _arXiv:1805.03735_ (2018).
  * Sainath et al. (2015) Tara N Sainath, Oriol Vinyals, Andrew Senior, and Haşim Sak. 2015.  Convolutional, long short-term memory, fully connected deep neural networks. In _Proc. IEEE Int. Conf. Acoustics, Speech, Sig. Proc._ 4580–4584.
  * Sharafaldin et al. (2018) Iman Sharafaldin, Arash Habibi Lashkari, and Ali A Ghorbani. 2018.  Toward generating a new intrusion detection dataset and intrusion traffic characterization.. In _Proc. Int. Conf. Inf. Syst. Secur. Privacy_. 108–116.
  * Shirani et al. (2015) Paria Shirani, Mohammad Abdollahi Azgomi, and Saed Alrabaee. 2015.  A method for intrusion detection in web services based on time series. In _Proc. IEEE Canadian Conf. Elec. Comp. Eng._ 836–841.
  * Sinha et al. (2019) Kapil Sinha, Arun Viswanathan, and Julian Bunn. 2019\.  Tracking temporal evolution of network activity for botnet detection.  _arXiv:1908.03443_ (2019).
  * Sperotto et al. (2010) Anna Sperotto, Gregor Schaffrath, Ramin Sadre, Cristian Morariu, Aiko Pras, and Burkhard Stiller. 2010.  An overview of IP flow-based intrusion detection.  _IEEE Comm. Surv. Tut._ 12, 3 (2010), 343–356.
  * Torres et al. (2016) Pablo Torres, Carlos Catania, Sebastian Garcia, and Carlos Garcia Garino. 2016.  An analysis of recurrent neural networks for botnet detection behavior. In _Proc. IEEE Biennal Congress Argentina_. 1–6.
  * Wu and Guo (2019) Peilun Wu and Hui Guo. 2019\.  LuNET: a deep neural network for network intrusion detection. In _Proc. IEEE Symp. Series Comp. Int._ 617–624.
  * Yeom et al. (2021) Sungwoong Yeom, Chulwoong Choi, and Kyungbaek Kim. 2021\.  Source-side DoS attack detection with LSTM and seasonality embedding. In _Proc. ACM Symp, Appl. Comp._ 1130–1137.
  * Yin et al. (2017) Chuanlong Yin, Yuefei Zhu, Jinlong Fei, and Xinzheng He. 2017\.  A deep learning approach for intrusion detection using recurrent neural networks.  _IEEE Access_ 5 (2017), 21954–21961.
  * Zhang et al. (2020) Hongpo Zhang, Lulu Huang, Chase Q Wu, and Zhanbo Li. 2020\.  An effective convolutional neural network based on SMOTE and Gaussian mixture model for intrusion detection in imbalanced dataset.  _Elsevier Comp. Netw._ 177 (2020), 107315.
  * Zhang et al. (2019) Yong Zhang, Xu Chen, Lei Jin, Xiaojuan Wang, and Da Guo. 2019.  Network intrusion detection: Based on deep hierarchical network and original flow data.  _IEEE Access_ 7 (2019), 37004–37016.
  * Zhou and Lang (2003) Mian Zhou and Sheau-Dong Lang. 2003.  Mining frequency content of network traffic for intrusion detection. In _Proc. Int. Conf. Commun. Netw. Inf. Secur._ 101–107.

##  Appendix A Appendix A

When preprocessing the datasets, we one-hot encode the categorical features:
source/destination ports, the direction and the protocol. The numerical
features are normalized in the range $[0,1]$. We grid-search the most optimal
parameters of each model, which we report in Table 5).

Table 5. Hyperparameters (CICIDS2017: left; CTU13: right).

parameter | LSTM | FNN | LSTM | FNN
---|---|---|---|---
learning rate | 0.001 | 0.001 | 0.001 | 0.001
# epochs | 30 | 20 | 100 | 100
dropout | 0.2 | 0.3 | 0.2 | 0.2
batch size | 1 | 512 | 1 | 1024
optimizer | Adam | Adam | Adam | Adam
truncated BPTT window | N/A | N/A | 512 | N/A
LSTM1 size | 256 | N/A | 256 | N/A
LSTM2 size | 256 | N/A | 256 | N/A
FC1 size | N/A | 256 | N/A | 512
FC2 size | N/A | 256 | N/A | 512
FC size | 2 | 2 | 2 | 2


11institutetext: Departamento de Física, Universidad de Murcia, 30071 Murcia,
Spain

# Methods on compositeness and related aspects

José Antonio Oller 11<EMAIL_ADDRESS>
###### Abstract

In many physical applications, bound states and/or resonances are observed,
which raises the question whether these states are elementary or composite.
Here we elaborate on several methods for calculating the compositeness $X$ of
bound states and resonances in Quantum Mechanics, and in Quantum Field Theory
by introducing particle number operators. For resonances $X$ is typically
complex and we discuss how to get meaningful results by using certain phase
transformations in the $S$ matrix.

##  1 Introduction

We start by reviewing a few basic aspects of the problem of compositeness in
hadron physics Weinberg:1962hj ; Weinberg:1963zza ; Weinberg:1965zz . The
Hamiltonian $H$ is first split in the free $H_{0}$ and interaction $V$ parts,

$\displaystyle H=H_{0}+V\leavevmode\nobreak\ .$ (1)

Both $H$ and $H_{0}$ share the same spectrum. For $H_{0}$, one has the bare
eigenstates,

$\displaystyle H_{0}|\varphi_{\alpha}\rangle$
$\displaystyle=E_{\alpha}|\varphi_{\alpha}\rangle\leavevmode\nobreak\
,\leavevmode\nobreak\ \text{ continuum spectrum}\leavevmode\nobreak\ ,$
$\displaystyle H_{0}|\phi_{n}\rangle$
$\displaystyle=E_{B_{n}}|\phi_{n}\rangle\leavevmode\nobreak\
,\leavevmode\nobreak\ \text{ discrete spectrum}\leavevmode\nobreak\ ,$ (2)

where $|\phi_{n}\rangle$ is a bare “elementary” state, and
$|\varphi_{\alpha}\rangle$ is made up by the direct product of free particles
(Greek letters are used as subscripts to refer to quantum numbers, among which
one has the momentum, spin, etc). The physical spectrum is comprised by the
eigenstates of $H$,

$\displaystyle H|\psi^{\pm}_{\alpha}\rangle$
$\displaystyle=E_{\alpha}|\psi^{\pm}_{\alpha}\rangle\leavevmode\nobreak\
,\leavevmode\nobreak\ \text{ continuum spectrum}\leavevmode\nobreak\ ,$
$\displaystyle H|\psi_{B_{n}}\rangle$
$\displaystyle=E_{B_{n}}|\psi_{B_{n}}\rangle\leavevmode\nobreak\
,\leavevmode\nobreak\ \text{ discrete spectrum}\leavevmode\nobreak\ .$ (3)

Here, the $|\psi_{B_{n}}\rangle$ are the bound states of the theory and the
$|\psi^{\pm}_{\alpha}\rangle$ are the scattering in/out states, respectively.
Within this framework the definitions of compositeness $X$ and elementariness
$Z$ of a bound state $|\psi_{B}\rangle$ are as follows Weinberg:1962hj . Let
us consider the linear decomposition of this state in the basis of eigenstates
of $H_{0}$,

$\displaystyle|\psi_{B}\rangle$
$\displaystyle=\sum_{n}\langle\phi_{n}|\psi_{B}\rangle|\phi_{n}\rangle+\int
d\alpha\langle\varphi_{\alpha}|\psi_{B}\rangle|\varphi_{\alpha}\rangle\leavevmode\nobreak\
.$ (4)

Then, the Parseval identity implies

$\displaystyle\langle\psi_{B}|\psi_{B}\rangle=1$
$\displaystyle=\underbrace{\sum_{n}|\langle\phi_{n}|\psi_{B}\rangle|^{2}}_{\equiv
Z}+\underbrace{\int
d\alpha|\langle\varphi_{\alpha}|\psi_{B}\rangle|^{2}}_{\equiv
X}\leavevmode\nobreak\ ,$ (5) $\displaystyle 1$
$\displaystyle=Z+X\leavevmode\nobreak\ .$

A new interpretation based on the use of the number operators in the
interaction or Dirac picture was introduced in Ref. Oller:2017alp , where more
details can be found. The basic idea is to take two free particles of types
$A$ and $B$ ($H_{0}|AB_{\gamma}\rangle=E_{\gamma}|AB_{\gamma}\rangle$). The
standard creation and annihilation operators of the two free particles are
$a^{\dagger}_{\alpha}$, $a_{\alpha}$, $b^{\dagger}_{\beta}$, $b_{\beta}$,
respectively. In terms of them, the number operator $N_{D}$ for the total
number of these free particles is

$\displaystyle N_{D}=\int d\alpha a_{\alpha}^{\dagger}a_{\alpha}+\int d\beta
b_{\beta}^{\dagger}b_{\beta}=N_{D}^{A}+N_{D}^{B}\leavevmode\nobreak\ ,$ (6)

with $N_{D}^{A}$ and $N_{D}^{B}$ the number operators for each species of
particles separately. Here, the subscript $D$ refers to the Dirac picture
used. In nonrelativistic Quantum Field Theory (QFT) one can express these
operators using the fields $\psi_{A}$ and $\psi_{B}$ as Thirring:book1

$\displaystyle N_{D}=\int
d^{3}x\left[\psi_{A}^{\dagger}(x)\psi_{A}(x)+\psi_{B}^{\dagger}(x)\psi_{B}(x)\right]\leavevmode\nobreak\
,\leavevmode\nobreak\ x=(t,{\mathbf{x}})\leavevmode\nobreak\ ,$ (7)

Let us notice that in the interaction picture $[H_{0},N_{D}]=0$, so that
$N_{D}$ is actually time independent, $N_{D}(t)=N_{D}(0)$. In terms of $N_{D}$
one can give an alternative definition for the compositeness $X$ as
Oller:2017alp

$\displaystyle
X=\frac{1}{2}\langle\psi_{B}|N_{D}|\psi_{B}\rangle\leavevmode\nobreak\ .$ (8)

The factor 1/2 is introduced because we are considering that only two-particle
states made out of $A$ and $B$ may contribute to $|\psi_{B}\rangle$. Let us
now probe the equivalence between the new definition of $X$ and the previous
one in Eq. (5). From the linear decompositions of the bound-state
$|\psi_{B}\rangle$ in states $|AB_{\gamma}\rangle$ and $|\phi_{n}\rangle$
(eigenstates of $H_{0}$) we have

$\displaystyle|\psi_{B}\rangle$ $\displaystyle=\int d\gamma
C_{\gamma}|AB_{\gamma}\rangle+\sum_{n}C_{n}|\phi_{n}\rangle\leavevmode\nobreak\
,$ (9) $\displaystyle X$
$\displaystyle=\frac{1}{2}\langle\psi_{B}|N_{D}^{A}+N_{D}^{B}|\psi_{B}\rangle=\int
d\gamma|C_{\gamma}|^{2}\leavevmode\nobreak\ .$

This definition is specially suitable for Effective Field Theories (EFTs),
like e.g. ChPT and hadron physics in general. The point is that in a low-
energy EFT written in terms of the (pseudo-)Goldstone bosons as the only
degrees of freedom, it is valuable to define $X$ as in Eq. (8), since it does
not require to have explicit bare “elementary” states (fields) in the
theoretical set up.

This new definition is also the most adequate for the treatment in QFT since,
as deduced in Oller:2017alp , it can be rewritten as

$\displaystyle X=\frac{1}{2}\lim_{T\to+\infty}\frac{1}{T}\int
d^{4}x\langle\varphi_{B}|P\left[e^{-i\int_{-\infty}^{+\infty}dt^{\prime}V_{D}(t^{\prime})}\sum_{i}\psi_{A_{i}}^{\dagger}(x)\psi_{A_{i}}(x)\right]|\varphi_{B}\rangle\leavevmode\nobreak\
,$ (10)

by expressing the number operators in terms of the nonrelativistic fields
$\psi_{A_{i}}$. Compared with Eq. (9) here $A\equiv A_{1}$ and $B\equiv
A_{2}$. Making use of the LSZ formalism one can directly express $X$ in terms
of $S$-matrix elements,

$\displaystyle X$ $\displaystyle\\!=\frac{1}{2}\lim_{E\to
E_{B}}\frac{(E-E_{B})^{2}}{g_{\alpha}(k_{B})^{2}}\lim_{T\to+\infty}\\!\frac{1}{T}\int\\!d^{4}x\langle\varphi_{\alpha}|P\left[e^{-i\int_{-\infty}^{+\infty}dt^{\prime}V_{D}(t^{\prime})}\sum_{i}\psi_{A_{i}}^{\dagger}(x)\psi_{A_{i}}(x)\right]|\varphi_{\alpha}\rangle\leavevmode\nobreak\
,$ (11)

with $g_{\alpha}(k_{B})$ the coupling of the bound state to the continuum
in/out states. Of course, the previous equation clearly shows that $X$ is an
observable in nonrelativistic QFT since it is expressed in terms of $S$-matrix
elements in the presence of a local source term, made up by the number-
operator density.

##  2 Explicit formulas

Figure 1: Feynman diagrams for the calculation of $X$ for a bound state
represented by the double lines. The two filled dots correspond to insertions
of a number-operator density.

The Feynman diagrams appropriate for the calculation of Eq. (10) are depicted
in Fig. 1, where the insertion of the number-operator density in the
propagator of every particle is represented by the double dots. The resulting
expression for $X$ in terms of the off-shell coupling to $A$ and $B$ is

$\displaystyle X_{\ell S}$
$\displaystyle=\int\frac{d^{3}k}{(2\pi)^{3}}\frac{g_{\ell
S}^{2}(k)}{(k^{2}/2\mu-E_{B})^{2}}\leavevmode\nobreak\ ,$ (12) $\displaystyle
X$ $\displaystyle=\sum_{\ell S}X_{\ell S}\leavevmode\nobreak\ ,$

with $\mu$ the reduced mass. The subscripts $\ell$ and $S$ in Eq. (12) refer
to particular two-particle partial wave amplitude (PWAs) coupled to the bound
state (e.g. ${}^{3}S_{1}$ and ${}^{3}D_{1}$ for the deuteron in nucleon-
nucleon ($NN$) scattering). If in the integral of Eq. (12) we assume a shallow
bound state then the coupling $g_{\ell S}(k)$ can be factorized out with
$k=0$, or with its on-shell value (which is valid at this level of accuracy).
Thus, $X_{\ell S}$ becomes

$\displaystyle X_{\ell S}$ $\displaystyle=g_{\ell
S}^{2}\frac{d}{dE}\int\frac{d^{3}k}{(2\pi)^{3}}\frac{1}{k^{2}/2\mu-
E_{B}}=g_{\ell S}^{2}\frac{i\mu^{2}}{2\pi k_{B}}\leavevmode\nobreak\ ,$ (13)

with the binding momentum $k_{B}=\sqrt{2\mu E_{B}}$. This is one of the
results of Ref. Weinberg:1965zz . The equation for $g_{\ell S}(k)$ is obtained
from the Lippmann-Schwinger equation $T=V+VGT$ in the limit $E\to E_{B}$ and
by taking the residue at the pole, so that

$\displaystyle g_{\ell S}(k)$
$\displaystyle=\frac{1}{2\pi^{2}}\int_{0}^{\infty}{k^{\prime}}^{2}dk^{\prime}V_{\ell
S}(k,k^{\prime})\frac{1}{{k^{\prime}}^{2}/2\mu-E_{B}}g_{\ell
S}({k^{\prime}})\leavevmode\nobreak\ .$ (14)

For the case of energy-independent potentials, which corresponds to pure
potential scattering, it was shown in Ref. Oller:2017alp that the
compositeness is exactly equal to 1 for any bound state, that is,

$\displaystyle\sum_{\alpha=1}^{n}X_{\alpha}=1\leavevmode\nobreak\ ,$ (15)

with $n$ the number of coupled PWAs. We refer to Oller:2017alp for the
interesting detailed analysis, which is skipped here to avoid becoming too
technical. This result has interesting implications like e.g. in the case of
the deuteron, studied within ChPT up to N4LO in terms of energy-independent
potentials with an estimated error in the reproduction of its properties of
around a 4% RodriguezEntem:2020jgp ; Epelbaum:2014sza . Therefore, one would
conclude from Eq. (15) that for this bound state $X=0.96-1.0$.

We now discuss within similar terms the case of resonances following the basic
lines of scattering theory and the use of number operators Oller:2017alp . A
resonance stems from the analytic continuation in energy of in states with
energy $E+i\varepsilon$, and out states with energy $E-i\varepsilon$
($\varepsilon\to 0^{+}$). In this way, when evaluating an $S$-matrix element
$\langle\psi_{\beta}^{-}|\psi_{\alpha}^{+}\rangle$ both ket and bra involve an
energy of $E+i\varepsilon$. In performing the analytical continuation towards
the pole position of the resonance in the complex $E$-plane one has to cross
the real axis along $E>0$, borrowing in the second Riemann sheet (RS) where
the resonance pole lies with $\Im E<0$. For the calculation of $X$ for a
resonance in QFT we can then proceed similarly as for the case of a bound
state by isolating the double pole residue of $S$-matrix elements with the
external source $\sum_{i}\psi_{A_{i}}^{\dagger}(x)\psi_{A_{i}}(x)$. The
corresponding formula is

$\displaystyle X=\frac{1}{2}\lim_{E\to
E_{R}}\frac{(E-E_{R})^{2}}{g_{\alpha}(k_{R})^{2}}\lim_{T\to+\infty}\\!\frac{1}{T}\int\\!d^{4}x\langle\varphi_{\alpha}|P\left[e^{-i\int_{-\infty}^{+\infty}dt^{\prime}V_{D}(t^{\prime})}\sum_{i}\psi_{A_{i}}^{\dagger}(x)\psi_{A_{i}}(x)\right]|\varphi_{\alpha}\rangle\leavevmode\nobreak\
,$ (16)

with $E_{R}$ and $k_{R}$ the resonance pole position in energy and momentum,
respectively. Of course, one can also write down a formula similar to Eq. (12)
for a bound state, corresponding to analogous Feynman diagrams as those in
Fig. 1 with the replacement $|\psi_{B}\rangle\to|\psi_{R}\rangle$. The result
is

$\displaystyle X_{\ell S}$
$\displaystyle=\int\frac{d^{3}k}{(2\pi)^{3}}\frac{g_{\ell
S}^{2}(k)}{(k^{2}/2\mu-E_{n}^{2})^{2}}+\frac{i\mu^{2}}{\pi
k_{n}}\left[\frac{\partial}{\partial k}k\,g_{\ell
S}^{2}(k)\right]_{k=k_{n}}\leavevmode\nobreak\ ,$ (17) $\displaystyle X$
$\displaystyle=\sum_{\ell S}X_{\ell S}\leavevmode\nobreak\ .$

The extrapolation of the integral to the second RS is responsible for the
additional last term as compared with Eq. (12).

In pure potential scattering it was proved in Ref. Oller:2017alp that
$X=\sum_{\alpha=1}^{n}X_{\alpha}=1$, see also Ref. Hernandez:1984zzb . For
instance, this would imply that $X$ for the virtual or antibound state in the
${}^{1}S_{0}$ $NN$ scattering should be very close to 1, since $NN$ scattering
data can be described very precisely with energy-independent potentials
derived from ChPT RodriguezEntem:2020jgp ; Epelbaum:2014sza . If for the
deuteron, with a binding momentum of around 45 MeV, the possible uncertainty
was bounded to a 4%, for the virtual state in the ${}^{1}S_{0}$ PWA one would
expect it to be smaller because $|k_{R}|\approx 10$ MeV$\ll 45$ MeV.

##  3 Compositeness in the Heisenberg picture

Let us denote by $N_{H}(t)$ the number operator in the Heisenberg picture. The
scattering states $|\psi^{\pm}\rangle$ are eigenstates of $H$ that behave as
free states when acted with operators at $\tau\mp\infty$, respectively
Weinberg:book1 . Then,

$\displaystyle\exp(-iH\tau)N_{H}(\tau)|\psi_{\alpha}^{\pm}\rangle\to\exp(-iH_{0}\tau)N_{D}(\tau)|\varphi_{\alpha}\rangle\leavevmode\nobreak\
,$ (18)

for $\tau\to\mp\infty$. Taking into account in the previous equation that
$|\psi^{\pm}\rangle=U_{D}(0,\tau)|\varphi_{\alpha}\rangle=\exp(iH\tau)\exp(-iH_{0}\tau)|\varphi_{\alpha}\rangle$,
with $\tau\to\mp\infty$, we have the equality at the operational level

$\displaystyle\lim_{\tau\to\pm}\exp(-iH\tau)N_{H}(\tau)\exp(iH\tau)=N_{D}(\tau)\leavevmode\nobreak\
.$ (19)

Since $H|\psi_{\alpha}^{\pm}\rangle=E_{\alpha}|\psi_{\alpha}^{\pm}\rangle$ we
also have that

$\displaystyle\langle\psi_{\alpha}^{-}|N_{D}(\tau)|\psi_{\alpha}^{+}\rangle=\langle\psi_{\alpha}^{-}|N_{H}(\tau)|\psi_{\alpha}^{+}\rangle\leavevmode\nobreak\
,\leavevmode\nobreak\ \tau\to\pm\infty.$ (20)

Furthermore, as $dN_{H}(t)/dt=i[H,N_{H}(t)]$ it follows that

$\displaystyle\frac{d}{dt}\langle\psi_{\alpha}^{-}|N_{H}(t)|\psi_{\alpha}^{+}\rangle=0\leavevmode\nobreak\
,$ (21)

so that both expectation values in Eq. (20) are time independent, and we can
simply write that

$\displaystyle\langle\psi_{\alpha}^{-}|N_{D}(t)|\psi_{\alpha}^{+}\rangle=\langle\psi_{\alpha}^{-}|N_{H}(t^{\prime})|\psi_{\alpha}^{+}\rangle\leavevmode\nobreak\
,$ (22)

for arbitrary $t$ and $t^{\prime}$. The equality between the expectation
values of $N_{D}$ and $N_{H}$ is also clear by taking into account the QFT
expressions above for $X$ in Eqs. (11) and (16), since they correspond to the
$S$-matrix elements
$\displaystyle{\lim_{T\to\infty}\frac{1}{T}\langle\psi_{\alpha}^{-}|\int_{-T/2}^{T/2}dt\,N_{H}(t)|\psi_{\alpha}^{+}\rangle}$
when calculated in the interaction picture. Afterwards, the residue of the
double pole is isolated.

Now, the idea would be to use an appropriate operator normalized in lattice
QCD to mimic the number operators. For instance, if the operator in the
Heisenberg picture is $O(t)$ and we consider a bound state made out of
nucleons, like the deuteron, we could take the normalization constant to be
$\langle N|O(t)|N\rangle$, which indeed is time independent (to conclude this
one can follow an analogous reasoning as used for Eq. (21)). Of course, extra
discussions about the final adequate choice for the operator $O(t)$ in lattice
QCD are still required, and this should be the object of further work.

##  4 Sum rule

From a generic formula for two-particle PWAs one can deduce a sum rule for
compositeness and elementariness. We follow here Ref. Guo:2015daa , which
employs relativistic kinematics. Two-body unitarity along the right-hand cut,
and above the thresholds of channels $i$ and $j$, can be expressed as
Oller:2019rej

$\displaystyle\Im\left.T^{-1}\right|_{ij}=\delta_{ij}\rho_{i}\leavevmode\nobreak\
,$ (23)

where $\rho_{i}$ is the phase-space factor $\rho_{i}=p_{i}/8\pi\sqrt{s}$, with
$p_{i}$ the center of mass (CM) momentum for the channel $i$ and $s$ the CM
total energy squared (the usual Mandelstam variable). From this equation one
can resum the right-hand or unitarity cut and obtain a general expression for
a PWA in coupled channels. In matrix notation it reads Oller:2019opk ;
Oller:2019rej

$\displaystyle T(s)=\left[{\cal
K}(s)^{-1}+G(s)\right]^{-1}\leavevmode\nobreak\ ,$ (24)

where ${\cal K}$ is an interacting kernel and the unitary loop function
$G_{i}(s)$ is given by

$\displaystyle G_{i}(s)$
$\displaystyle=\frac{1}{16\pi^{2}}\left[a_{i}(\Lambda)+\log\frac{m_{i1}^{2}}{\Lambda^{2}}-x_{+}\log\frac{x_{+}-1}{x_{+}}-x_{-}\log\frac{x_{-}-1}{x_{-}}\right]\leavevmode\nobreak\
,$ (25) $\displaystyle x_{\pm}$
$\displaystyle=\frac{s+m_{i2}^{2}-m_{i1}^{2}}{2s}\pm\frac{1}{2s}\sqrt{(s+m_{i2}^{2}-m_{i1}^{2})^{2}-4s(m_{i2}^{2}-i0^{+})}\leavevmode\nobreak\
,$

where $m_{1i}$ and $m_{2i}$ are the masses of the two particles in the channel
$i$, and $a_{i}(\Lambda)$ is a subtraction constant (the combination
$a_{i}(\Lambda)-\log\Lambda^{2}$ is $\Lambda$ independent). Taking the
derivative of Eq. (24) with respect to $s$, in the limit $s\to s_{R}$ the
residue of the double pole implies that

$\displaystyle 1$
$\displaystyle=\underbrace{-\sum_{i}g_{i}^{2}\left.\frac{dG_{i}(s)}{ds}\right|_{s_{B}}}_{X_{i}}+\underbrace{g^{T}G(s_{B})\left.\frac{dK(s)}{ds}\right|_{s_{B}}G(s_{B})g}_{Z}\leavevmode\nobreak\
.$ (26)

The expression for $X_{i}$ does not depend on $a_{i}$ or $\Lambda$, since they
disappear in the derivative of $G_{i}(s)$. It is the same expression as in Eq.
(13) for shallow bound states, and also for separable potentials
Sekihara:2014kya ; Aceti:2012dd .

##  5 Resonances

For the case of resonances one should consider the unitarity loop function in
the 2nd RS for those channels in which the transit to this sheet is involved
to find the pole position at $s_{R}$. We indicate this by $G^{II}(s_{R})$.
Then, Eq. (26) implies

$\displaystyle X_{i}$
$\displaystyle=-g_{i}^{2}\left.\frac{dG_{i}^{II}(s)}{ds}\right|_{s_{R}}\leavevmode\nobreak\
.$ (27)

However, for resonances $X_{i}$ is typically complex. We argue next that one
should actually take its absolute value $|X_{i}|$. This based on certain phase
transformations of the $S$-matrix in PWAs, driving to a phase redefinition of
the couplings Guo:2015daa . Let us first consider a narrow resonance
$\Gamma\ll M_{R}-m_{\rm th}$, with $\Gamma$, $M_{R}$ the resonance mass and
width, and $m_{\rm th}$ the nearest threshold. Later on we generalize these
results for wider resonances. The $S$ matrix is split in a pole term plus
$S_{0}(s)$, which is assumed to be smooth in a certain domain around the
resonance pole. That is,

$\displaystyle S(s)=\frac{R}{s-s_{R}}+S_{0}(s)\leavevmode\nobreak\ ,$ (28)

with $R$ the residue matrix of the resonance pole at $s_{R}$. Then, we impose
unitarity of the $S$ matrix in PWAs,

$\displaystyle S(s)S(s)^{\dagger}=I\leavevmode\nobreak\ .$ (29)

The solution of this equation with $S(s)$ given by Eq. (28), with constant
$S_{0}$ for $s$ near $s_{R}$, is

$\displaystyle S(s)$ $\displaystyle={\cal
O}\underbrace{\left(I+\frac{i\lambda{\cal A}}{s-s_{R}}\right)}_{S_{R}(s)}{\cal
O}^{T}\leavevmode\nobreak\ ,$ (30) $\displaystyle{\cal O}{\cal O}^{\dagger}$
$\displaystyle=I\leavevmode\nobreak\ .$

Here ${\cal A}$ is a rank one symmetric projection operator and $S_{R}(s)$ is
the pure resonant $S$ matrix. The disposition of operators in Eq. (30) clearly
indicates the corrections to $S_{R}$ due to initial- and final-state
interactions from $S_{0}$.

Figure 2: Isoscalar scalar $\pi\pi$ phase shifts, $J^{PC}=0^{++}$. For the
experimental references see Oller:1997ti .

The dressing to $S_{R}$ by $S_{0}$ typically modifies the phases of the
resonance couplings in $S_{R}$. This is because the moduli of the couplings
$g_{i}$ to the different channels have physical meaning, since they provide
the partial decay widths of a narrow resonance ParticleDataGroup:2022pth ,
$\Gamma_{i}={|g_{i}|^{2}}/({8\pi M_{R}^{2}})$. Then, the $S$-matrix phase
transformation only change the phases of the resonance couplings

$\displaystyle S_{\cal O}(s)$ $\displaystyle\equiv{\cal O}S(s){\cal
O}^{T}\leavevmode\nobreak\ ,$ (31) $\displaystyle{\cal O}$ $\displaystyle={\rm
diag}(e^{i\phi_{1}},\ldots,e^{i\phi_{n}})\leavevmode\nobreak\ ,$
$\displaystyle g_{i}^{2}$ $\displaystyle\to
g_{i}^{2}e^{2i\phi_{i}}\leavevmode\nobreak\ .$

As an example of this mechanism let us consider $\pi\pi-K\bar{K}$ scattering
in $J=0$ and $I=0$ ($I$ is the total isospin). As the author checked in the
calculations for Ref. Oller:1997ti , the coupling to $\pi\pi$ of the
$f_{0}(980)$ is a positive number times a phase-factor $i$, which stems from
the phase-shifts of $\pi\pi$ rescattering due to the $f_{0}(500)$ resonance at
the rise of the $f_{0}(980)$. This is clearly seen from Fig. 2.

Therefore, $X_{i}$, Eq. (27), only changes its phase under the phase
transformations of Eq. (31), and they can be chosen so that it becomes a real
positive number. Hence,

$\displaystyle X_{i}\to$ $\displaystyle|X_{i}|\geq 0\leavevmode\nobreak\ .$
(32)

Let us now generalize Eq. (32) for non-narrow resonances. The point is to
require the validity of the Laurent series expansion around the resonance pole
up to physical values of $s$ above threshold, so that the resonance couplings
still directly impact physical energies. This implies that $s_{{\rm th};n}<\Re
s_{R}<s_{{\rm th};n+1}$, where $s_{{\rm th};n}$ is the threshold of channel
$n$.

##  6 Effective range expansion

We assume now that a bound or resonance pole lies close enough to a relevant
threshold so that the well-known effective range expansion (ERE) can be
applied. We follow here the developments in Ref. Kang:2016ezb . Up to and
including the scattering length $a$ and the effective range $r$ the $S$-wave
PWA can be written as

$\displaystyle T(k)$
$\displaystyle=\frac{1}{-\frac{1}{a}+\frac{1}{2}rk^{2}+G(k)}\leavevmode\nobreak\
,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
G(k)=-ik\leavevmode\nobreak\ .$ (33)

Then, if $T(k)$ has a resonance pole at $E=E_{R}=M_{R}-i\Gamma/2$ (energy is
measured from $m_{\rm th}$), $a$ and $r$ can be given in terms of the mass and
width of the resonance as

$\displaystyle a$
$\displaystyle=-\frac{2k_{i}}{|k_{R}|^{2}}\leavevmode\nobreak\
\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\
k_{R}=\sqrt{2\mu E_{R}}=k_{r}-i\,k_{i}\leavevmode\nobreak\ ,$ (34)
$\displaystyle r$ $\displaystyle=-\frac{1}{k_{i}}\leavevmode\nobreak\
\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\ \text{so that
}\frac{r}{a}>2\leavevmode\nobreak\ .$

By applying Eq. (27) we can express the compositeness as

$\displaystyle X$
$\displaystyle=-\gamma^{2}\frac{dG}{ds}=-\gamma_{k}^{2}\frac{dG}{dk}=i\frac{k_{i}}{k_{r}}=i\tan\frac{\phi}{2}\leavevmode\nobreak\
,$ (35)

with $\gamma_{k}^{2}$ the residue at the pole position in the complex
$k$-plane. The simplicity of the ERE up to including $r$ perfectly illustrates
the condition on the mass of the resonance, discussed at the end of Sec. 5,
because

$\displaystyle|X|\leq 1\leftrightarrow\phi\in[0,\pi/2]\leftrightarrow
M_{R}\geq 0\leavevmode\nobreak\ ,$ (36)

with $|X|=1$ for $M_{R}=0$ and $\Gamma>0$. Furthermore, since $X$ is purely
imaginary in Eq. (35) if its real part is taken to end with a real
compositeness, as advocated in Ref. Aceti:2014ala , then from the ERE $X=0$.
Of course, this result is not meaningful in such generality. The expression
for $|X|$ can also be rewritten as $|X|=\left(\frac{2r}{a}-1\right)^{-1}$. In
the subsequent we drop the sign of modulus on $X$ and directly consider by
this symbol the compositeness calculated within the ERE.

##  7 CDD poles. Track of elementariness

While the ERE is a smooth expansion around $k=0$ of the combination
$t(E)^{-1}+ik$, which has no right-hand cut, more structure can be accounted
for by allowing poles in $t(E)^{-1}$. These are the so-called CDD poles
Castillejo:1955ed . In this way, a once-subtracted dispersion relation (DR)
for $t(E)^{-1}$ accounting for a pole in $t(E)^{-1}$ at $M_{Z}$ gives

$\displaystyle t(E)=\frac{1}{\frac{\lambda}{E-M_{Z}}+\beta-
ik}\leavevmode\nobreak\ ,$ (37)

with $\lambda$ the residue at the CDD pole and $\beta$ a subtraction constant.
A consequence of this pole is that the ERE or a Flatté parameterization break
down for $|k|\geqslant\sqrt{2\mu|M_{Z}|}$. A near-threshold CDD pole
contributes to $a$ and $r$ as

$\displaystyle\delta a$
$\displaystyle=\frac{M_{Z}}{\lambda}\leavevmode\nobreak\ ,\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \delta
r=-\frac{\lambda}{mM_{Z}^{2}}\leavevmode\nobreak\ .$ (38)

Therefore, $\delta a\to 0$ and $\delta r\to-\infty$ (unless $\lambda=0$) for
$M_{Z}\to 0$. As a result, a large negative value of $r$ is a hint for a
nearby CDD pole and for elementariness. Indeed, if we calculate $X$ from the
ERE formula $X=\left(\frac{2r}{a}-1\right)^{-1}\to 0$ for $M_{Z}\to 0$.
Nonetheless, one should keep in mind that having a near CDD pole to a
resonance pole is sufficient but not necessary for the resonance to be
elementary. For instance, as analyzed in Ref. Oller:1998zr , the CDD pole
associated to the $\rho(770)$ resonance is located at
$s=M_{\rho}/(1-g_{V}^{2})$ and tends to infinity because of the KSFR relation,
which requires $g_{V}^{2}=1$. This is so despite the resonance $\rho(770)$ is
a clear example of a $q\bar{q}$ resonance. From a hadronic point of view this
is clearly signaled by studying the dependence of the pole position with the
number of colors of QCD $N_{c}$ Pelaez:2006nj ; Guo:2012yt .

##  8 Determination of $X$ by making use of the decays of the resonance

Here we outline the method introduced in Ref. Meissner:2015mza to combine the
knowledge of the resonance pole position and the saturation of its width and
branching ratios (when they are available), and/or of the compositeness $X$.
This method has the advantage that it is a coupled-channel study, while the
one based in the ERE takes into account only one channel. If the channel 1 is
the lighter channel and 2 is the one near the mass of the resonance, the main
equations are $X=X_{1}+X_{2}$ and $\Gamma=\Gamma_{1}+\Gamma_{2}$, with

$\displaystyle\Gamma_{1}$
$\displaystyle=\frac{2X_{1}}{\mu}k(M_{R})|k_{R}|\leavevmode\nobreak\
,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\Gamma_{2}=\frac{X_{2}|k_{R}|M_{R}^{2}}{\pi\mu}\int_{M_{\rm
th}}^{+\infty}dW\frac{k(W)}{W^{2}}\frac{\Gamma}{(M_{R}-W)^{2}+\Gamma^{2}/4}\leavevmode\nobreak\
.$ (39)

From these equations one can then constrain the couplings or, equivalently,
the partial compositeness coefficients.

As a concluding remark, we stress that the different methods used in many
instances to study the nature of resonances have provided when used
simultaneously a consistent picture on the nature of these resonances. Several
examples are discussed in Ref. oller.theseproceedings . This is interesting
since each method has its own realm of applicability, and the fact that one
can achieve compatible conclusions from all of them reinforces the global
picture that emerges.

## Acknowledgements

This work has been supported in part by the MICINN AEI (Spain) Grant No.
PID2019â106080GB-C22/AEI/10.13039/501100011033 . I thank Zhi-Hui Guo for the
feedback on the manuscript.

## References

  * (1) S. Weinberg, Phys. Rev. 130, 776 (1963)
  * (2) S. Weinberg, Phys. Rev. 131, 440 (1963)
  * (3) S. Weinberg, Phys. Rev. 137, B672 (1965)
  * (4) J.A. Oller, Annals Phys. 396, 429 (2018), 1710.00991
  * (5) E.M. Henley, W. Thirring, _Elementary Quantum Field Theory_ (McGray-Hill Book Company Inc., 1962)
  * (6) D. Rodriguez Entem, R. Machleidt, Y. Nosyk, Front. in Phys. 8, 57 (2020)
  * (7) E. Epelbaum, H. Krebs, U.G. Meißner, Phys. Rev. Lett. 115, 122301 (2015), 1412.4623
  * (8) E. Hernandez, A. Mondragon, Phys. Rev. C 29, 722 (1984)
  * (9) S. Weinberg, _The Quantum Theory of Fields I_ (Cambridge University Press, 2005), ISBN 0521670535, 978-0521670531
  * (10) Z.H. Guo, J.A. Oller, Phys. Rev. D 93, 096001 (2016), 1508.06400
  * (11) J.A. Oller, _A Brief Introduction to Dispersion Relations_ , SpringerBriefs in Physics (Springer, 2019), ISBN 978-3-030-13581-2, 978-3-030-13582-9
  * (12) J.A. Oller, Prog. Part. Nucl. Phys. 110, 103728 (2020), 1909.00370
  * (13) T. Sekihara, T. Hyodo, D. Jido, PTEP 2015, 063D04 (2015), 1411.2308
  * (14) F. Aceti, E. Oset, Phys. Rev. D 86, 014012 (2012), 1202.4607
  * (15) J.A. Oller, E. Oset, Nucl. Phys. A 620, 438 (1997), [Erratum: Nucl.Phys.A 652, 407–409 (1999)], hep-ph/9702314
  * (16) R.L. Workman et al. (Particle Data Group), PTEP 2022, 083C01 (2022)
  * (17) X.W. Kang, Z.H. Guo, J.A. Oller, Phys. Rev. D 94, 014012 (2016), 1603.05546
  * (18) F. Aceti, L.R. Dai, L.S. Geng, E. Oset, Y. Zhang, Eur. Phys. J. A 50, 57 (2014), 1301.2554
  * (19) L. Castillejo, R.H. Dalitz, F.J. Dyson, Phys. Rev. 101, 453 (1956)
  * (20) J.A. Oller, E. Oset, Phys. Rev. D 60, 074023 (1999), hep-ph/9809337
  * (21) J.R. Pelaez, G. Rios, Phys. Rev. Lett. 97, 242002 (2006), hep-ph/0610397
  * (22) Z.H. Guo, J.A. Oller, J. Ruiz de Elvira, Phys. Rev. D 86, 054006 (2012), 1206.4163
  * (23) U.G. Meißner, J.A. Oller, Phys. Lett. B 751, 59 (2015), 1507.07478
  * (24) J.A. Oller, Z.H. Guo, Eur. Phys. J. C. These proceedings (2022)