# Multi-label Prediction in Time Series Data
using Deep Neural Networks

Wenyu Zhang1  Devesh K. Jha2  Emil Laftchiev2  and Daniel Nikovski2  1
<EMAIL_ADDRESS>2  {jha, laftchiev<EMAIL_ADDRESS>Department of
Statistics, Cornell University, Ithaca, NY, USA Mitsubishi Electric Research
Labs (MERL), Cambridge, MA, USA

###### Abstract

This paper addresses a multi-label predictive fault classification problem for
multidimensional time-series data. While fault (event) detection problems have
been thoroughly studied in literature, most of the state-of-the-art techniques
can’t reliably predict faults (events) over a desired future horizon. In the
most general setting of these types of problems, one or more samples of data
across multiple time series can be assigned several concurrent fault labels
from a finite, known set and the task is to predict the possibility of fault
occurrence over a desired time horizon. This type of problem is usually
accompanied by strong class imbalances where some classes are represented by
only a few samples. Importantly, in many applications of the problem such as
fault prediction and predictive maintenance, it is exactly these rare classes
that are of most interest. To address the problem, this paper proposes a
general approach that utilizes a multi-label recurrent neural network with a
new cost function that accentuates learning in the imbalanced classes. The
proposed algorithm is tested on two public benchmark datasets: an industrial
plant dataset from the PHM Society Data Challenge, and a human activity
recognition dataset. The results are compared with state-of-the-art techniques
for time-series classification and evaluation is performed using the F1-score,
precision and recall.

###### keywords:

keywords: Time-series analysis, Fault Detection, Fault Prediction.

##  1 Introduction

Time series analysis for rare events such as faults is generally a known and
difficult problem Yamanishi Takeuchi (2002). The problem is particularly
difficult in the multi-dimensional, or multi-variate setting, where the events
may be described by simultaneous occurrences on multiple time series. A key
confounding factor is the number of class labels, or the number of events that
must be discovered. In the worst case, the events are described by labels from
a known set, but may occur simultaneously. Naturally, this approach to
labeling leads to a number of classes that grows combinatorially with the
number of individual labels in the original set. The problem in this setting
is one of time series analysis on infrequent events with multi-label classes
that are characterized by severe class imbalance.

This is a critical problem in the field of prognostics and health management
where faults for an equipment occur rarely but often have serious impacts such
as service disruption, safety concerns for users and associated costs of
repair or replacement. In these fields, the detection of rare events (fault
detection, anomaly detection) in a timely fashion is critical to minimizing
the detrimental impacts of the events. An example of this type of application
was demonstrated in Holst et. al. Holst . (2012) where a fault detection tool
was used to on a fleet of trains leading to reduced long-term maintenance
costs by $5-10\%$.

While detecting faults is important Chandola . (2009); Romeres . (2019) and
many models have been used including deep learning Jha . (2016); Sarkar .
(2016), today the field of anomaly detection is moving towards predicting when
a fault will occur. This capability is commonly known as prognostics, or the
ability to predict impending faults. Prognostics is a critical technology in
the drive to reduce cost of operation. The key difference is that while
anomaly detection leads to corrective maintenance, fault prediction leads to
predictive maintenance which can be performed before failures occur or before
they develop into major defects. In practice this means that the condition of
the equipment is monitored, and the time to failure is continuously estimated.
Such an approach is able to reduce the number of repairs, allow for optimal
scheduling of maintenance workers and improve the safety of human operators.

This paper focuses on the problem of fault prediction in the multidimensional
time series case with multi-label classes some of which a have few examples.
We propose a recurrent neural network-based (RNN) approach that in addition to
the observed time series incorporates contextual information for past, present
and future time intervals. This contextual information describes the operation
of the plant/machine and could contain information about the target set-point,
type of control being used for the plant, expected output, speed of
plant/machine operation, etc. Using this approach, we are able to predict data
labels for future time segments and to localize the faults that could occur in
future with better accuracy than several other state-of-the-art algorithms for
time-series analysis.

Figure 1: Notation used in this paper. Example given in the setting of
industrial plant fault prediction with two fault types: binary labels denote
the presence of fault, observations correspond to sensor measurements, context
sequence are control system setpoints.

Figure 1 illustrates the problem setting and the paper notation. Here $\tau$
is the length of observed data, and $T-\tau$ is the prediction horizon. For a
sample $i$ of some monitored system,
$z^{(i)}_{1:\tau}=\left[z^{(i)}_{1},\dots,z^{(i)}_{\tau}\right]$ of
length-$\tau$ is a historical sequence of observations where
$z^{(i)}_{t}\in\mathbbm{R}^{d_{z}}$. During the historical period, there are
no labels that indicate the state of faults. For the future time segment,
fault labels are predicted at two levels of granularity through segment labels
and stepwise labels. Given $L$ possible fault labels,
$\tilde{y}^{(i)}\in\mathbbm{R}^{L}$ denotes the segment label where each
element $\tilde{y}^{(i)}[\ell]\in\\{0,1\\}$ indicates the absence or presence
of a label $\ell$ in the next $T-\tau$ steps. For example, if $L=2$, the set
of possible segment labels is $\left\\{[0,0],[1,0],[0,1],[1,1]\right\\}$. The
stepwise label is
$\tilde{o}^{(i)}_{\tau+1:T}=\left[\tilde{o}^{(i)}_{\tau+1},\dots,\tilde{o}^{(i)}_{T}\right]$
where each binary vector $\tilde{o}^{(i)}_{t}\in\mathbbm{R}^{L}$ indicates the
absence or presence of the $L$ labels at step $t\in\\{\tau+1,\dots,T\\}$. Here
$\tilde{o}^{(i)}_{\tau+1:T}$ provides label localization on top of
$\tilde{y}^{(i)}$. Mathematically,
$\tilde{y}^{(i)}[\ell]=\mathbbm{1}\\{\sum_{t=\tau+1}^{T}\tilde{o}^{(i)}_{t}[\ell]>0\\}$.
There is no restriction on the number of one’s in the vectors
$\tilde{y}^{(i)}$ and $\tilde{o}^{(i)}_{t}$, as is consistent with the
definition of a multi-label classification problem. In the fault prediction
context, this means that multiple types of faults are allowed to occur
simultaneously. The case where multiple components of a system are monitored
for failure is also included.

Contextual input $c^{(i)}_{1:T}=\left[c^{(i)}_{1},\dots,c^{(i)}_{T}\right]$
where $c^{(i)}_{t}\in\mathbbm{R}^{d_{c}}$ can include any independent variable
of the system that is fully known at time $\tau$ (ex. pre-specified setpoint
control sequences in industrial plants - Figure 1.) The presence of a label
$\ell$ is a function of both input $z$ and $c$ without further assumptions on
the structure within the inputs.

In this paper, we would like to estimate
$\text{P}\left(\tilde{y}[\ell]=1|z_{1:\tau},c_{1:T}\right)$, the probability
of given label in a future segment. To estimate this probability, the usual
approach is to first define two functions $F$ and $V^{\ell}$ on the input
variables $z_{1:\tau},c_{1:T}$. The function $V^{\ell}$ transforms the inputs
from the observed space into an embedded space, while the function $F$ maps
from the embedded space to the interval $[0,1]$. A simple map is chosen so
that there is a clear and interpretable relationship between embedding and the
output. Second, the optimal parameters of these functions are learned such
that $F\left(V^{\ell}\left(z_{1:\tau},c_{1:T}\right)\right)$ approximates
$\text{P}\left(\tilde{y}[\ell]=1|z_{1:\tau},c_{1:T}\right)$ as closely as
possible.

This paper represents a new neural network architecture which we call Multi-
label Predictive Network (MPN) using RNNs with a loss function that is
designed considering the class imbalance. The architecture is fashioned after
the sequence-to-sequence (Seq2Seq) model which has been successful in neural
machine translation Sutskever . (2014). This paper demonstrates that the model
can be adapted for a different objective of multi-label prediction for
temporal data.

We follow the insight that while prediction of multi-dimensional time-series
data is a complex task due to the high-dimensional temporal data and the
complex correlations which might be present between the different components
of the data. It is, however, easier to learn the dynamics of the faults
occurring in the system using a low-dimensional representation of the time-
series data in an abstract space.

Neural networks used for sensor data compression are shown to be capable of
capturing intramodal and intermodal correlations, and the compressed
representations are effective for prediction and missing data imputation tasks
Liu . (2017). In prognostics and health management, neural networks can learn
useful features to estimate remaining useful life and degradation Li . (2018).

For accurate prediction of faults over a certain prediction horizon, the
faults predicted by the network at every instant of time should be taken into
account as the faults may be correlated and thus prediction of a fault at a
time-instant ’$k$’ might affect of the probability of occurrence of the fault
at a future instant ’$k+1$’. Additionally, this information could also be
useful in the multi-fault setting as some faults could always occur together
or some faults never occur together and thus, this can simplify learning of
the fault dynamics. As a result of this motivation, the proposed MPN predicts
faults one step at a time, and then the predicted fault is fed back to the
network for making predictions on the faults occurring in future over a finite
window of time.

The loss function used to train MPN is comprised of sub-objectives that
predict the true segment labels and the stepwise labels, $\tilde{y}$ and
$\tilde{o}$. Both label types are multi-label, which means that the number of
possible labels scales with the number of discrete labels in the label set.
Samples can thus be labeled with any number of labels ranging from none
(healthy) to all possible labels in the label set. For each label class, the
loss is weighted to account for the imbalance in the training data set.
Classes that are rare are weighted more heavily. The loss function also
includes regularization components to (possibly) prevent overfitting in the
network. The proposed ideas for fault prediction using MPN are verified using
an industrial plant dataset from the 2015 Prognostics and Health Management
Society (PHM Society) Data Challenge which consists a total of 33 plants. The
network is also tested on a second dataset which is a human activity
recognition (HAR) dataset from the 2011 Opportunity Activity Recognition
Challenge, and we aim to predict low-level motions based on sensor readings
and high-level activities. Both datasets are publicly available. Comparisons
are conducted with several state-of-the-art techniques for time series
classification, and MPN is shown to have better performance in general in
terms of precision, recall and F1-score. Compared to the best performers in
the two data challenges which use hand-engineered features, MPN attains
comparable or better performance without the need for extensive feature
extraction.

##  2 Related Work

An important quantity for predictive maintenance is the remaining useful life
(RUL). RUL of an equipment is the length from the current time to the end of
its useful life, which is usually a unit of time, but can also be other
measurements such as revolutions for rotating machines, number of operations
for switchgear and load cycles for structural components Welte Wang (2014). If
a model of the performance degradation path can be built, RUL can also be
found by determining the time at which degradation exceeds the failure
threshold Hu . (2015). The rate of degradation is usually not known in real
world applications where physical simulation technology or accelerated life
testing is not possible. In general, physical and stochastic models may not
sufficiently capture the dynamics in the data from a complex system. RUL
estimation does not reflect the duration of a fault, while the proposed MPN
also attempts to localize the start and end times of faults in a target
window.

A conventional data-driven approach for label prediction is to directly
perform classification on the historical observations with standard shallow
classifiers Salfner . (2010). Ensemble-based fault detection methods
frequently use hand-engineered features Xiao (2016); Xie . (2016) that are
hard to identify and require domain knowledge. The label power set strategy
represents multi-label combinations as distinct classes Tsoumakas . (2011)
however this approach is limited by unobserved classes and exponential space
and computational complexity. The simplifying assumption of independent labels
allows for training a separate binary classifier for each label. Classifier
chains Read . (2009) can be used to preserve information on label dependence,
but chain order can significantly affect performance. A long short term memory
(LSTM) RNN on raw time series with replicated targets achieves state-of-the-
art performance in medical diagnoses Lipton . (2016). In this paper, LSTM
models are used in a predictive network to take advantage of their ability to
model long-term dependencies.

Alternative distance-based approaches include Shapelet Forests (S-F) Patri .
(2014) which extracts subsequences that are the most discriminative for each
class and calculates their distances to the time series for classification.
K-Nearest Neighbors (KNN) approaches use a majority vote of the neighbors to
classify a sample. Dynamic Time Warping (DTW) distance can quantify the
qualitative dissimilarity of sequences and is therefore often used for human
activity recognition Seto . (2015). Siamese LSTM networks learn a similarity
metric from time series by first embedding sample pairs in a new space and
then comparing the embedded representations using cosine similarity Pei .
(2016). All KNN methods depend on the diversity and quality of the training
samples which is a significant issue in the setting of severe class imbalance.

When the data is fully labeled, Markov models combined with an iterative
strategy can be used to do multi-step forward prediction Read . (2017). A
common approach is time series prediction followed by classification Molaei
Keyvanpour (2015). However such labeling is generally not available in real
applications.

The Seq2Seq model uses two LSTM networks to generate output sequences for
neural machine translation Sutskever . (2014). Extensions of RNNs have
benefited from including spatial and temporal information Liu . (2016). RNNs
have also seen success in RUL estimation, with the benefit that a neural
network setup can be configured to include contextual information such as
workloads, operating conditions and deterioration modes Zheng . (2017); Heimes
(2008) In this paper, the Seq2Seq model is modified to incorporate temporal
contextual input in the forecasting window and to output multi-label
predictions. Moreover, the approach here avoids sequential error sources by
directly predicting time series labels.

##  3 Proposed Approach

The objective of this work is to predict fault labels in a multidimensional
time series on a window of time in the future. This prediction is performed
using historical observations of the time series and contextual (application
specific) input. Given $L$ labels, the proposed approach has two steps. First,
using MPN, we learn an $L$-dimensional predictive representation of the data
in the future time segment. Then second, we use a linear classifier in each
dimension of the predictive representation to determine the presence or
absence of a given label. Importantly, each dimension $\ell$ of the
representation corresponds to a label $\ell$ in the total label set and all
dimensions are learned jointly. The linear classification is thus performed
per dimension and independently of the other dimensions. This independence in
the prediction means that the number of dimensions is equal to the number of
faults and not the size of the power set which is all possible combinations of
faults.

To reduce the complexity of the predictive problem, the MPN predicts each
fault separately. This is motivated by the realization that the network learns
the coupled, correlated dynamics between different faults during training.
This is because the network is trained to predict faults one step at a time
and these predictions are then used to predict faults in the future. In such a
scenario, the network learns the dependence of faults occurring in time– more
specifically, it can possibly learn which faults act as precursors to other
faults. As a result of this simplification, the dimension of the predicted
vector is reduced to the number of faults occurring in the system instead of
all the possible combinations of faults in the system. This simplification
also allows to alleviate the sample complexity of the neural network as it has
to learn fewer classes.

The predictive representation is modeled using a neural network because neural
networks have the advantage of being able to model irregular data shapes and
complex system dynamics from data. The network in this paper consists of two
connected LSTM networks: the encoder and decoder networks. Each network is
turned to capture the time series dynamics in either the historical data
(network 1) or the future segment data (network 2). The following equations
describe the update step of an LSTM network. Here at each step, given input
$x_{t}$ and network states at the previous time step, the current hidden state
$h_{t}$ is obtained as Understanding LSTMs (2015):

$\displaystyle f_{t}$
$\displaystyle=\sigma\left(W_{f}\cdot\left[h_{t-1},x_{t}\right]+b_{f}\right)$
$\displaystyle i_{t}$
$\displaystyle=\sigma\left(W_{i}\cdot\left[h_{t-1},x_{t}\right]+b_{i}\right)$
$\displaystyle\tilde{\xi}_{t}$
$\displaystyle=\tanh\left(W_{\xi}\cdot\left[h_{t-1},x_{t}\right]+b_{\xi}\right)$
$\displaystyle\xi_{t}$
$\displaystyle=f_{t}\odot\xi_{t-1}+i_{t}\odot\tilde{\xi}_{t}$ $\displaystyle
q_{t}$
$\displaystyle=\sigma\left(W_{q}\cdot\left[h_{t-1},x_{t}\right]+b_{q}\right)$
$\displaystyle h_{t}$ $\displaystyle=q_{t}\odot\tanh\left(\xi_{t}\right)$ (1)

where $\sigma$ is the sigmoid function, $\tanh$ is the hyperbolic tangent and
$\odot$ is the Hadamard product. The cell state is $\xi_{t}$. The forget gate
is denoted as, $f_{t}$, the input gate is denoted as $i_{t}$, and the output
gate is denoted as $q_{t}$. The hidden states undergo additional computations
to arrive at the final output of the network. These computations are
application specific. The weight matrix $W$ and bias vector $b$ corresponding
to each gate are learned to minimize the loss function of the network. For
$h_{t}\in\mathbbm{R}^{L}$ and $x_{t}\in\mathbbm{R}^{m}$, the total number of
trainable parameters for an LSTM network is $4\left(L^{2}+Lm+L\right)$. The
MPN encoder is an one-layer LSTM network with $L$ units, $x_{t}$ is a
concatenation of $z_{t}$ and $c_{t}$, and $m=d_{z}+d_{c}+L$. The MPN decoder
is another one-layer LSTM netwok with $L$ units, $x_{t}$ is $c_{t}$ and
$m=d_{c}+L$.

###  3.1 Predictive Data Representations

As previously noted, here we are interested in estimating the conditional
probability
$\text{P}\left(\tilde{y}^{(i)}[\ell]=1\left|z^{(i)}_{1:\tau},c^{(i)}_{1:T}\right.\right)$.
This estimate is preformed using MPN which is depicted in Figure 2. In this
figure, the blue blocks (square pattern) represent the encoder LSTM network.
This network encodes the historical data into the embedded space. The green
(stripped pattern) blocks represent the decoder LSTM network which decodes the
embedded representation into predictions of the labels on a future segment.
The flow of information both within the networks and between the two networks
is represented by the arrows. Interpreting the figure from left to right, the
encoder network synthesizes historical observations $z^{(i)}_{1:\tau}$ and
contextual variables $c^{(i)}_{1:\tau}$ to output
$h^{(i)}_{\tau}\in\mathbbm{R}^{L}$, an estimated representation of labels
present at $\tau$. This representation is input to the decoder network
together with the final hidden state of the encoder network. Given this
synthesized historical information from the encoder network and the future
context $c^{(i)}_{\tau+1:T}$, the decoder network directly learns
$h^{(i)}_{t}$ to predict labels for each future time step.

Figure 2: Multi-label Predictive Network (MPN) architecture to embed sample
$i$ as $g^{(i)}$. Encoding network (blue square pattern) encodes historical
observations $z^{(i)}_{1:\tau}$ and historical context $c^{(i)}_{1:\tau}$,
decoding network (green stripped pattern) predicts for labels with additional
contextual input $c^{(i)}_{\tau+1:T}$. Hidden units are denoted $h^{(i)}_{t}$.
MPN outputs multi-label predictions $y^{(i)}$ for the segment labels and
$o^{(i)}_{\tau+1:T}$ for the stepwise labels.

In the final step, the estimate of $y^{(i)}$ is produced by adding bias to the
sum of $h^{(i)}_{t}$ terms (hidden states) and taking the sigmoid
transformation on the total sum.

$\displaystyle g^{(i)}$
$\displaystyle=\sum\limits_{t=\tau+1}^{T}h^{(i)}_{t}+b_{g}$ (2) $\displaystyle
y^{(i)}$ $\displaystyle=\sigma\left(g^{(i)}\right)$ (3)

The final output, $y^{(i)}[\ell]$, is the estimate of the desired conditional
probability
$\text{P}\left(\tilde{y}^{(i)}[\ell]=1\left|z^{(i)}_{1:\tau},c^{(i)}_{1:T}\right.\right)$.
That is, using notations in Section 1, the function $V^{\ell}$ is the first
portion of the proposed MPN that maps to $\mathbbm{R}$ to produce the
embedding $g$, and the function $F$ is the sigmoid function.

####  3.1.1 Designing the Loss Function

The network described in the last section are trained using a loss function
considering two main objectives. Specifically, the loss function has two
components: the component which penalizes errors in labeling the whole future
time segment, and a component which penalizes errors in labeling each step in
the future segment.

The first component is the component which focuses on learning the labels on
the whole future segment. A variable in this component of the objective
function is $p_{\ell}$. $p_{\ell}$ is the probability of label occurrence in
the training set, and it is used as a weight in the cross-entropy loss,
$w_{\ell}=-\log\left(p_{\ell}\right)$ Jain . (2016). Using this weighting is
important because it accentuates rare classes and forces the network to learn
classes which have very few examples. This weighting is known to increase the
false positive rate in the rare classes. However, in this application this
increase is acceptable as compared to the relative importance of true positive
and the importance of preserving label diversity in the predictions.

We note that there are many different weight schemes Jain . (2016). The choice
in this paper is motivated by the empirical experiments which show that this
choice optimizes the performance of MPN. Edge cases where $p_{\ell}=0$ are
assigned a weight of 1 to indicate no preference in predicting label $\ell$ as
either present or absent in the test set. The label prediction loss is then
expressed as follows with true labels denoted by ~(tilde),

$\displaystyle
L^{(i)}_{Y}=-\sum\limits_{\ell=1}^{L}\left[w_{\ell}\tilde{y}^{(i)}[\ell]\log\left(y^{(i)}[\ell]\right)+\right.$
$\displaystyle\hskip
85.35826pt\left.\left(1-\tilde{y}^{(i)}[\ell]\right)\log\left(1-y^{(i)}[\ell]\right)\right]$
(4)

The second component which penalizes errors in labeling each step in the
future segment. This component of the objective function uses
$o^{(i)}_{t}=\sigma\left(h^{(i)}_{t}\right)\sigma\left(g^{(i)}\right)\in(0,1)$,
the step wise label vector which annotates the occurrence of labels in each of
future segment time steps. At each time instant in the future segment,
$o^{(i)}_{t}[\ell]$ approximates the probability
$\text{P}\left(\tilde{o}^{(i)}_{t}[\ell]=1\left|z^{(i)}_{1:\tau},c^{(i)}_{1:T}\right.\right)$
that the label is present or absent at $t$ for sample $i$.

When $\tilde{o}^{(i)}_{t}[\ell]=1$, both $\sigma\left(h^{(i)}_{t}\right)$ and
$\sigma\left(g^{(i)}\right)$ are $\approx 1$. When a given label is present at
$t$ and $h^{(i)}_{t}[\ell]$ is negative, this function penalizes the result.
Similarly, negative value of $g^{(i)}[\ell]$ are penalized when label $\ell$
is present in the forecasting window. Therefore $o^{(i)}_{t}[\ell]$ tends to
be positive for a present label and close to zero for an absent label.
$o^{(i)}_{t}[\ell]$ is used in a squared loss function that ensures training
stability.

$\displaystyle
L^{(i)}_{O}=\frac{1}{L(T-\tau)}\sum\limits_{t=\tau+1}^{T}\left[\tilde{o}^{(i)}_{t}\cdot\left(1-o^{(i)}_{t}\right)^{2}+\right.$
$\displaystyle\hskip
113.81102pt\left.\left(1-\tilde{o}^{(i)}_{t}\right)\cdot\left(o^{(i)}_{t}\right)^{2}\right]$
(5)

Note that the summand is similar to
$\left\lVert\tilde{o}^{(i)}_{t}-o^{(i)}_{t}\right\rVert^{2}_{2}$. The true
step wise label $\tilde{o}^{(i)}_{t}[\ell]\in\\{0,1\\}$. When
$\tilde{o}^{(i)}_{t}[\ell]=1$, the loss incurred is
$\left(1-o^{(i)}_{t}[\ell]\right)^{2}$, and when
$\tilde{o}^{(i)}_{t}[\ell]=0$, the loss incurred is
$\left(0-o^{(i)}_{t}[\ell]\right)^{2}$. We write $L_{O}^{(i)}$ as such so that
it has similar form as $L_{Y}^{(i)}$.

The objective function further has an $\ell_{2}$ regularization on the weights
of the network, collectively denoted as $W$, to prevent overfitting. Letting
$B$ denote the set of samples in a batch during training, the overall batch
loss functions with hyperparameter $\lambda$ are:

$\displaystyle L^{B}_{base}=\frac{1}{|B|}\sum\limits_{i\in
B}L^{(i)}_{Y}+\frac{\lambda}{2}\left\lVert W\right\rVert^{2}_{F}$ (6)
$\displaystyle L^{B}_{localize}=\frac{1}{|B|}\sum\limits_{i\in
B}\left[L^{(i)}_{Y}+L^{(i)}_{O}\right]+\frac{\lambda}{2}\left\lVert
W\right\rVert^{2}_{F}$ (7)

The loss functions $L^{B}_{base}$ and $L^{B}_{localize}$ correspond to network
configurations MPN-base and MPN-localize, respectively. MPN-base does not make
use of stepwise labels and hence does not incorporate the $L_{O}$ loss. It is
suitable for situations where stepwise labels are unavailable or label
localization is not required.

(a) Target distribution

(b) Learned distribution

(c) Original distribution

Figure 3: Embedded representations $g$ and original distribution for two
labels. $[n_{1}\hskip 2.0ptn_{2}]$ denotes the segment label to predict.

###  3.2 Multi-label Decision

Figure 3(c) shows the t-SNE representation of the original samples of the PHM
dataset for two fault types, which demonstrates that the samples are not
readily separable at their original state. Figure 3(a) shows the ideal
distribution of $g^{(i)}$ across $i$ learned for two labels. In this
representation, samples with identical label vectors are distinctly clustered
together, and a threshold at zero would clearly separate samples that disagree
on a given label $\ell$. Figure 3(b) plots the actual embeddings produced by
MPN, which are evidently more linearly separable than the original samples.

Cluster radius is typically governed by the data. Two observations can be
made. First, the more data points in a cluster, the larger the cluster’s
radius. Second, the larger the noise component observed in the data, the
larger the cluster radius. Both cases may lead to cluster overlap as shown in
Figure3(b). Thus in practice it may be difficult to immediately note the
correct classification boundary. However, standard classification algorithms
can be used to find this boundary. An example here is a linear one-vs-all SVM
classifier which was found to be robust to input data during our experiment.
Label localization, or prediction of labels per step in the future interval,
can be achieved using a linear SVM classifier.

##  4 Experiments using Real Data

The experiments for this study compare the performance of MPN to six state-of-
the-art multi-label classifiers in two applications: An ensemble of
classifiers on hand-engineered features and observations (Feature); a KNN
classifier using majority vote and the DTW distance (DTW); Shapelet Forests
(S-F) Patri . (2014) which require learning on a powerset of labels; a multi-
label LSTM network with replicated targets (ML-LSTM) Lipton . (2016); a
Siamese LSTM network learned metric McClure (2017) used in a KNN classifier
(Siamese-LSTM); and a SVM using the RBF kernel which does not explicitly model
the temporal input (SVMrbf). Details of these methods can be found in Section
2. As noted before, the inputs to these classifiers are $z^{\prime}_{1:\tau}$
and $c_{1:T}$, where $z^{\prime}_{1:\tau}$ is $z_{1:\tau}$ padded to the
length of $c$ with the feature mean. This is a common approach taken to remedy
the case of missing data from real life data collection. Because these
classifiers do not learn the stepwise labels $\tilde{o}$ they are compared
with MPN-base which does not have the loss component $L_{O}$.

In addition, different features and ensemble of classifiers are used for each
of the two applications due to differences of the two domains. Details of the
datasets are in Sections 4.1 and 4.2. The two methods are named Feature (PHM)
and Feature (HAR) according to the two domains. Feature (PHM) includes all
features used in the winning entry Xiao (2016) of the 2015 PHM Society Data
Challenge, namely month, weekday, hour, day, time, and historical
observations. The ensemble comprises gradient boosting machine, random forest
and penalized logistic regression, which are the best-performing methods for
the tasks in Xiao’s paper Xiao (2016). The ensemble in Feature (HAR) comprises
three best-performing methods in the 2011 Opportunity Activity Recognition
Challenge, namely SVM, 1-Nearest Neighbor and decision tree Chavarriaga .
(2011). The corresponding features (mean, variance, maximum, minimum and
historical observations) are included for Feature (HAR). Feature engineering
is a common preprocessing technique for human activity recognition Figo .
(2010). More recent methods utilize deep learning and are moving away from
being feature-based Ordóñ±ez Roggen (2016), but these methods are mainly
focused on segmentation and classification and not prediction.

Detailed experiments for this paper also studied the effects of different
multi-label decision classifiers and auxiliary losses in MPN. The performance
of three classifiers is tested. The default classifier on the embedded
representations is the linear SVM. Two other classifiers tested are hard
thresholding at zero and assuming the nearest class as measured by distance
from the empirical class mean. MPN-base and MPN-localize are network
configurations with loss functions $L^{B}_{base}$ and $L^{B}_{localize}$,
respectively. Siamese MPN-localize is a Siamese version of MPN-localize with
an auxiliary Siamese objective measuring the similarity
$\tilde{s}^{(i,j)}[\ell]$ of label $\ell$ for sample pair $(i,j)$ through
$s^{(i,j)}[\ell]=\exp{\left(-\left|g^{(i)}[\ell]-g^{(j)}[\ell]\right|\right)}$.
This version is included since the Siamese structure has shown success in
classification tasks with class imbalance Pei . (2016); Mueller Thyagarajan
(2016); Neculoiu . (2016). Siamese MPN-localize is trained with loss function:

$\displaystyle L^{B}_{siamese}=\frac{1}{|pair(B)|}\sum\limits_{(i,j)\in
pair(B)}$ $\displaystyle\hskip
71.13188pt\left[\beta\left(L^{(i)}_{Y}+L^{(j)}_{Y}+L^{(i)}_{O}+L^{(j)}_{O}\right)+\right.$
$\displaystyle\hskip
71.13188pt\left.(1-\beta)L^{(i,j)}_{S}\right]+\frac{\lambda}{2}\left\lVert
W\right\rVert^{2}_{F}$ (8)

where $pair(B)$ is a set of unique pairs of samples in $B$, and
$L^{(i,j)}_{S}$ is the squared loss for $s^{(i,j)}$:

$\displaystyle
L^{(i,j)}_{S}=\frac{1}{L}\left[\tilde{s}^{(i,j)}\cdot\left(1-s^{(i,j)}\right)^{2}+\right.$
$\displaystyle\hskip
85.35826pt\left.\left(1-\tilde{s}^{(i,j)}\right)\cdot\left(s^{(i,j)}\right)^{2}\right]$
(9)

For the experiments, we train all three network configurations using
$L^{B}_{siamese}$ with the appropriate loss components turned off, so that all
configurations have the same number of trainable parameters and are
comparable.

Performance is evaluated using the F1-score, precision and recall. Micro and
macro averages are presented because metric choice depends on the importance
of the rare labels to the user. Macro metrics average the performance of each
label type, and micro metrics average the performance across each sample.

For all methods, a hyperparameter grid search is performed on a held out
validation set to maximize the sum of micro and macro F1-scores. The
hyperparameters tuned and the search values are presented in Table 1. In all
experiments, there are 500, 100, and 400 training, validation, and test
samples, respectively.

Method |  Hyperparameter
---|---
Feature (PHM) |  Regularization parameter ($C$)
Feature (HAR) |  RBF kernel parameter ($\gamma$), regularization parameter ($C$)
DTW |  number of nearest neighbors ($k$), warping window ($w$)
SVMrbf |  RBF kernel parameter ($\gamma$), regularization parameter ($C$)
S-F |  None
ML-LSTM |  learning rate ($\eta$), intermediary target parameter ($\alpha$), dropout probability ($p$)
Siamese-LSTM |  number of nearest neighbors ($k$), learning rate ($\eta$), margin parameter ($m$), dropout probability ($p$)
MPN-base |  learning rate ($\eta$), regularization parameter ($\lambda$)
MPN-localize |  learning rate ($\eta$), regularization parameter ($\lambda$)
Siamese MPN-localize |  learning rate ($\ell$), weighing parameter ($\beta$), regularization parameter ($\lambda$)
(a) Hyperparameters tuned

Hyperparameter |  Values
---|---
$k$ |  $1,3,5,7,9$
$w$ |  $0,1,2$
$\gamma$, $C$ |  $0.001,0.01,0.1,1,10$
$\eta$ |  $0.001,0.01,0.1$
$\alpha$, $\beta$, $p$, $m$ |  $0.1,0.3,0.5,0.7,0.9$
$\lambda$ |  $0.01,0.1,1$
(b) Values tuned

Table 1: Hyperparameters tuned for each method

###  4.1 Fault Prediction for Industrial Plants

The PHM dataset contains plant data from 33 plants with labeled data for 6
types of failure events 2015 PHM Data Challenge (2015). The sampling interval
for all plants is 15 minutes. The dataset contains time series samples for
several years. For this reason fixed-length samples (for each plant) are
chosen using random sampling. Sample length for historical observations is 30
(7.5 hours), and the forecast length for future events is 10 (2.5 hours).
These durations are chosen because the dataset description stipulates that
faults are predictable at most three hours in advance. Observation input $z$
contains sensor measurements and contextual input $c$ contains control system
setpoints, and their combined dimension is 20 to 116 across plants. Because
each plant has a unique sensor set, testing and training is performed
separately for each plant.

Each plant has a particular number of monitored components, and each component
corresponds to four sensors and four control references. The components are
further grouped by zones, and there are two sensors providing environmental
readings per zone. The true component-to-zone mapping is not provided in the
dataset, hence no mapping is explicitly enforced through model input. Fault
instances are given in terms of a start and end time, and a code specifying
the fault type. This information is formatted according to our multi-label
objective as $\tilde{y}$ and $\tilde{o}_{\tau+1:T}$ for training and testing
the models. An example experiment setup for Plant 1 is shown in Figure 4.
Plant 1 has six components split over three zones.

Figure 4: PHM: Monitoring setup for Plant 1 with components (yellow circles)
split over zones. True component-to-zone mapping is not provided, and included
only for illustration purposes.

This dataset has extreme label imbalance with important events that happen
infrequently. This is demonstrated in the label distribution table in Table 2,
where fault label 4 is observed only 8 times in the 600 training and
validation samples of Plant 1. Furthermore, multiple concurrent faults can
create unique system states. Figure 5 shows two samples of measurements from
the first sensor and first control reference across the six components
monitored in Plant 1. Figure 5(a) comes from a healthy sample with no faults.
The sensor readings demonstrate an upward trend and fluctuations, possibly due
to frequent changes of the setpoints in the control reference. This indicates
the importance of including information of future context for prediction.
Figure 5(b) comes from a sample with co-occurring fault labels 1 and 6. Fault
6 starts in the middle and persists till the end of the observed time frame,
and Fault 1 starts and ends in the second half of the time frame. Multi-label
objectives are required due to the variety of label combinations.

Fault Code | Occurrences
---|---
Code 1 | 58
Code 2 | 43
Code 3 | 31
Code 4 | 8
Code 5 | 20
Code 6 | 155
Table 2: PHM: Number of Plant 1 training samples per fault label.

(a) Sample with no faults

(b) Sample with fault labels 1 and 6. Dotted and solid vertical line mark the
start and end of a fault, respectively. Black denotes fault label 1 and pink
denotes fault label 6.

Figure 5: PHM: Plant 1 sample measurements of the first sensor and first
control reference across six components.

####  4.1.1 Results and Discussion

Figure 6 shows the multi-label prediction test performance, where the
performance of MPN-base for each plant is subtracted from that of competing
methods for the corresponding plant. The illustration allows direct comparison
of the methods for each of the 33 plants. A negative value after subtraction
indicates a superior performance of MPN-base, which is mostly the case at both
micro and macro levels for precision, recall and F1-scores. The
hyperparameters most frequently selected for each model across the 33 plants
are presented in Table 3.

Method |  Selected hyperparameter
---|---
Feature (PHM) |  $C=0.1$
DTW |  $k=1$, $w=2$
SVMrbf |  $\gamma=0.001$, $C=10$
S-F |  None
ML-LSTM |  $\eta=0.001$, $\alpha=0.1$, $p=0.9$
Siamese-LSTM |  $k=1$, $\eta=0.001$, $m=0.1$, $p=0.9$
MPN-base |  $\eta=0.001$, $\lambda=1$
MPN-localize |  $\eta=0.01$, $\lambda=0.1$
Siamese MPN-localize |  $\eta=0.01$, $\beta=0.3$, $\lambda=0.1$
Table 3: PHM: Hyperparameters most frequently selected across 33 plants.

S-F performed the worst, because the limited number of samples corresponding
to each label combination does not allow sufficiently diverse shapelets to be
extracted for classification. The other five methods performed similarly with
SVMrbf being slightly worse overall since it does not attempt to capture the
dynamics of time series. Feature (PHM) has slightly higher micro-F1 scores but
lower macro-F1 scores, which implies that it has worse prediction performance
for rare labels. In general, the competing methods did not perform as well as
MPN-base, possibly because they are not constructed to make effective use of
future context. The additional future context results in an irregular input
data shape. Padding of the samples, a common remedy for missing data, allows
the inclusion of the extra information in these models, but padding with a
constant can cause a bias in learning since the missing data is not random and
the data is temporal in nature.

Figure 6: PHM: Label prediction test performance of methods subtracted by MPN-
base for each of 33 plants. Negative value indicates that MPN-base
outperformed in the metric for the plant.

Although some training labels are rare as the Plant 1 example shows in Table
2, the proposed MPN-base is able to learn the full variety of labels. Figure 7
shows the distribution of the true Plant 1 test labels versus the labels
predicted by MPN-base. The counts are similar across all labels, and MPN-base
is capable of predicting label 4 despite it being underrepresented in the
training set.

Figure 7: PHM: Number of true and estimated Plant 1 test samples per fault
label.

Effects of model design: Table 4 shows three different methods of
classification on the embedded representations: linear SVM; hard thresholding
at zero; and assuming the nearest class as measured by distance from the
empirical class mean. The SVM shows the best overall performance. Note the
poor performance of the third approach is because the variance of the
representations is different across the classes.

Classifier | Macro-F1 | Micro-F1
---|---|---
Mean | Var | Mean | Var
Linear SVM | 0.610 | 0.017 | 0.774 | 0.007
Threshold at 0 | 0.603 | 0.018 | 0.765 | 0.007
Nearest class | 0.599 | 0.019 | 0.590 | 0.044
Network | Macro-F1 | Micro-F1
Mean | Var | Mean | Var
MPN-base | 0.610 | 0.017 | 0.774 | 0.007
MPN-localize | 0.605 | 0.021 | 0.771 | 0.007
Siamese MPN-localize | 0.603 | 0.020 | 0.773 | 0.008
Table 4: PHM: Performance of MPN-base network given choice of classifier, and
performance of three versions of the MPN network. The mean values are averaged
across 33 plants.

Table 4 shows a comparison of MPN-base, MPN-localize and Siamese MPN-localize.
All three versions have similar F1-scores, showing that the auxiliary Siamese
targets added in this experiment do not improve learning. Separation of the
embedded representations across different labels is achieved sufficiently by
the loss functions $L^{B}_{base}$ and $L^{B}_{localize}$.

Fault localization: For testing the label localization performance of MPN-
localize, the baseline is the multi-label prediction output without temporal
localization, i.e. a fault label is either present/absent at all forecast time
steps. Figure 8 shows the localization performance subtracted from the
baseline performance for each of the 33 plants. As expected, the baseline has
higher recall but also more false positives. Overall, localization improves
accuracy and the F1-score.

Figure 8: PHM: Fault localization test performance of multi-label prediction
baseline subtracted by label localization for each of 33 plants.

###  4.2 Label Prediction for Human Activity Recognition

This application of human activity prediction is included to further
demonstrate the effectiveness of the proposed MPN in multi-label prediction,
and also its usage outside of PHM domains.

The Opportunity Dataset contains time series data used for HAR tasks Roggen .
(2010). Each of 4 subjects perform an activity of daily living (ADL) 5 times.
Sensor readings are collected from sensing systems deployed on the body of the
subjects, on objects and in the environment. The ADL consists of a sequence of
high-level activities, namely early morning, coffee time, sandwich time,
cleanup and relaxing. On a finer level, each action is described by the motion
of the arms and the objects the arms interact with. We are interested in
predicting the fine-level motions based on the sensor readings and high-level
activities.

In this experiment, fixed-length samples are uniformly sampled with observed
and forecast lengths as 75 and 25, respectively. The observation input $z$ is
the 243-dimensional measurements from GPS, motion sensors and other sensing
units, and the contextual input $c$ is the indicator matrix for high-level
activities which empirically can be inferred from the time of day and the
subject’s pattern of past schedules. Labels to be predicted are the low-level
activities of the right arm. This is a combination of its motion in 13
categories including open and unlock, and the object the arm is interacting
with which consists 23 options such as bottle and dishwasher. That is, $L=36$,
Label occurrences range from 1 to 180 times. This is a challenging task due to
label sparsity and the inherent difficulty of predicting human activities Ryoo
(2011).

####  4.2.1 Results and Discussion

Figure 9 shows the test performance of each competing method subtracted by
that of MPN-base. The proposed method has the best overall performance. This
is similar to the results in the PHM application in Section 4.1.1.

Figure 9: HAR: Label prediction test performance of methods subtracted by MPN-
base. Negative value indicates that MPN-base outperformed in the metric.

The hyperparameters selected for each model are presented in Table 5.

Method |  Selected hyperparameter
---|---
Feature (HAR) |  $\gamma=0.001$, $C=0.001$
DTW |  $k=1$, $w=2$
SVMrbf |  $\gamma=10$, $C=0.1$
S-F |  None
ML-LSTM |  $\eta=0.001$, $\alpha=0.1$, $p=0.9$
Siamese-LSTM |  $k=1$, $\eta=0.01$, $m=0.1$, $p=0.7$
MPN-base |  $\eta=0.001$, $\lambda=1$
MPN-localize |  $\eta=0.01$, $\lambda=1$
Siamese MPN-localize |  $\ell=0.01$, $\beta=0.3$, $\lambda=0.1$
Table 5: HAR: Hyperparameters selected.

Effects of model design: Table 6 shows similar performance for SVM and
thresholding, indicating that the learned representations can be classified by
the zero cutoff as designed. MPN-localize has the best F1-score, showing that
MPN benefits from the additional stepwise label targets in this application.

Classifier | Macro-F1 | Micro-F1
---|---|---
Linear SVM | 0.361 | 0.417
Threshold at 0 | 0.362 | 0.413
Nearest class | 0.343 | 0.298
Network | Macro-F1 | Micro-F1
MPN-base | 0.361 | 0.417
MPN-localize | 0.411 | 0.426
Siamese MPN-localize | 0.344 | 0.420
Table 6: HAR: Performance of MPN-base network given choice of classifier, and
performance of three versions of the MPN network.

##  5 Conclusion

This paper proposes a recurrent neural network-based approach for multi-label
prediction problem in high-dimensional time series data with severe class
imbalance. The proposed network, which we refer to as multi-label predictive
network (MPN), embeds time series samples into a target space where the data
is linearly separable. The loss function of MPN allows a variable number of
labels for each sample and appropriately weighs each class to compensate for
the rarity of some labels. Co-occurring labels and rare labels are common
characteristics in fault prediction applications. The proposed algorithm for
fault prediction using MPN performed well against state-of-the-art techniques
on two different benchmark datasets, and is applicable to prediction tasks
beyond PHM domains as well.

## References

  * 2015 PHM Data Challenge (2015) phm152015 PHM Data Challenge. 2015 PHM Data Challenge.  2015\.  https://www.phmsociety.org/events/conference/phm/15/data-challenge.  Accessed: 2018-09-20
  * Chandola . (2009) chandola2009anomalyChandola, V., Banerjee, A. Kumar, V.  2009\.  Anomaly detection: A survey Anomaly detection: A survey.  ACM computing surveys (CSUR)41315.
  * Chavarriaga . (2011) opportunity2011Chavarriaga, R., Sagha, H., Roggen, D. Ferscha, A.  2011\.  Opportunity Activity Recognition Challenge: Results and Conclusions. Opportunity Activity Recognition Challenge: Results and Conclusions.  http://www.opportunity-project.eu/system/files/docs/Workshop/2011_SMC_Challenge.pdf.  Accessed: 2019-08-27
  * Figo . (2010) figo2010Figo, D., Diniz, PC., Ferreira, DR. Cardoso, JMP.  2010Oct01.  Preprocessing techniques for context recognition from accelerometer data Preprocessing techniques for context recognition from accelerometer data.  Personal and Ubiquitous Computing147645–662.
  * Heimes (2008) heimes2008Heimes, FO.  2008Oct.  Recurrent neural networks for remaining useful life estimation Recurrent neural networks for remaining useful life estimation.  2008 International Conference on Prognostics and Health Management 2008 International Conference on Prognostics and Health Management ( 1-6).
  * Holst . (2012) holst2012Holst, A., Bohlin, M., Ekman, J., Sellin, O., Lindström, B. Larsen, S.  2012\.  Statistical anomaly detection for train fleets Statistical anomaly detection for train fleets.  AI Magazine3433-42.
  * Hu . (2015) hu2015Hu, C., Zhou, Z., Zhang, J. Si, X.  2015\.  A survey on life prediction of equipment A survey on life prediction of equipment.  Chinese Journal of Aeronautics28125 - 33.
  * Jain . (2016) jain2016Jain, H., Prabhu, Y. Varma, M.  2016\.  Extreme multi-label loss functions for recommendation, tagging, ranking, other missing label applications Extreme multi-label loss functions for recommendation, tagging, ranking, other missing label applications.  Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining ( 935–944).  New York, NY, USAACM.
  * Jha . (2016) jha2016temporalJha, DK., Srivastav, A. Ray, A.  2016\.  Temporal learning in video data using deep learning and Gaussian processes Temporal learning in video data using deep learning and gaussian processes.  Workshop on Machine Learning for Prognostics and Health Managament at 2016 KDD, San Francisco, CA. Workshop on Machine Learning for Prognostics and Health Managament at 2016 KDD, San Francisco, CA.
  * Li . (2018) li2018Li, X., Ding, Q. Sun, JQ.  2018\.  Remaining useful life estimation in prognostics using deep convolution neural networks Remaining useful life estimation in prognostics using deep convolution neural networks.  Reliability Engineering & System Safety1721 - 11\.
  * Lipton . (2016) lipton2015Lipton, ZC., Kale, DC., Elkan, C. Wetzel, RC.  2016\.  Learning to diagnose with LSTM recurrent neural networks Learning to diagnose with LSTM recurrent neural networks.  International Conference on Learning Representations.
  * Liu . (2016) liu2016Liu, Q., Wu, S., Wang, L. Tan, T.  2016\.  Predicting the next location: A recurrent model with spatial and temporal contexts Predicting the next location: A recurrent model with spatial and temporal contexts.  AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence.
  * Liu . (2017) liu2017Liu, Z., Zhang, W., Lin, S. Quek, TQS.  2017April.  Heterogeneous Sensor Data Fusion By Deep Multimodal Encoding Heterogeneous sensor data fusion by deep multimodal encoding.  IEEE Journal of Selected Topics in Signal Processing113479-491.
  * McClure (2017) mcclure2017McClure, N.  2017\.  TensorFlow machine learning cookbook Tensorflow machine learning cookbook.  Packt Publishing.
  * Molaei Keyvanpour (2015) molaei2015Molaei, SM. Keyvanpour, MR.  2015March.  An analytical review for event prediction system on time series An analytical review for event prediction system on time series.  2015 2nd International Conference on Pattern Recognition and Image Analysis (IPRIA) 2015 2nd International Conference on Pattern Recognition and Image Analysis (IPRIA) ( 1-6).
  * Mueller Thyagarajan (2016) mueller2016Mueller, J. Thyagarajan, A.  2016\.  Siamese recurrent architectures for learning sentence similarity Siamese recurrent architectures for learning sentence similarity.  Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence ( 2786–2792).  AAAI Press.
  * Neculoiu . (2016) neculoiu2016Neculoiu, P., Versteegh, M. Rotaru, M.  2016\.  Learning text similarity with Siamese recurrent networks Learning text similarity with Siamese recurrent networks.  Rep4NLP@ACL. Rep4NLP@ACL.
  * Ordóñ±ez Roggen (2016) ordonez2016Ordóñ±ez, FJ. Roggen, D.  2016\.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition.  Sensors161.
  * Patri . (2014) patri2014Patri, OP., Sharma, AB., Chen, H., Jiang, G., Panangadan, AV. Prasanna, VK.  2014Oct.  Extracting discriminative shapelets from heterogeneous sensor data Extracting discriminative shapelets from heterogeneous sensor data.  2014 IEEE International Conference on Big Data 2014 IEEE International Conference on Big Data ( 1095-1104).
  * Pei . (2016) pei2016Pei, W., Tax, DMJ. van der Maaten, L.  2016\.  Modeling time series similarity with Siamese recurrent networks Modeling time series similarity with Siamese recurrent networks.  CoRRabs/1603.04713.
  * Read . (2017) read2017Read, J., Martino, L. Hollmén, J.  2017\.  Multi-label methods for prediction with sequential data Multi-label methods for prediction with sequential data.  Pattern Recognition6345 - 55.
  * Read . (2009) read09Read, J., Pfahringer, B., Holmes, G. Frank, E.  2009\.  Classifier chains for multi-label classification Classifier chains for multi-label classification.  Proceedings of the 2009th European Conference on Machine Learning and Knowledge Discovery in Databases - Volume Part II Proceedings of the 2009th European Conference on Machine Learning and Knowledge Discovery in Databases - Volume Part II ( 254–269).  Berlin, HeidelbergSpringer-Verlag.
  * Roggen . (2010) roggen1020Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Förster, K., Tröster, G.d. R. Millàn, J.  2010June.  Collecting complex activity datasets in highly rich networked sensor environments Collecting complex activity datasets in highly rich networked sensor environments.  2010 Seventh International Conference on Networked Sensing Systems (INSS) 2010 Seventh International Conference on Networked Sensing Systems (INSS) ( 233-240).
  * Romeres . (2019) 8795698Romeres, D., Jha, DK., Yerazunis, W., Nikovski, D. Dau, HA.  2019June.  Anomaly Detection for Insertion Tasks in Robotic Assembly Using Gaussian Process Models Anomaly detection for insertion tasks in robotic assembly using Gaussian process models.  2019 18th European Control Conference (ECC) 2019 18th European Control Conference (ECC) ( 1017-1022).  10.23919/ECC.2019.8795698
  * Ryoo (2011) Ryoo2011Ryoo, MS.  2011Nov.  Human activity prediction: Early recognition of ongoing activities from streaming videos Human activity prediction: Early recognition of ongoing activities from streaming videos.  2011 International Conference on Computer Vision 2011 International Conference on Computer Vision ( 1036-1043).
  * Salfner . (2010) salfner2010Salfner, F., Lenk, M. Malek, M.  201003\.  A survey of online failure prediction methods A survey of online failure prediction methods.  ACM Comput. Surv.42310:1–10:42.
  * Sarkar . (2016) sarkar2016multimodalSarkar, S., Jha, DK., Lore, KG., Sarkar, S. Ray, A.  2016\.  Multimodal spatiotemporal information fusion using neural-symbolic modeling for early detection of combustion instabilities Multimodal spatiotemporal information fusion using neural-symbolic modeling for early detection of combustion instabilities.  2016 American Control Conference (ACC) 2016 American Control Conference (ACC) ( 4918–4923).
  * Seto . (2015) seto2015Seto, S., Zhang, W. Zhou, Y.  2015Dec.  Multivariate time series classification using dynamic time warping template selection for human activity recognition Multivariate time series classification using dynamic time warping template selection for human activity recognition.  2015 IEEE Symposium Series on Computational Intelligence 2015 IEEE Symposium Series on Computational Intelligence ( 1399-1406).
  * Sutskever . (2014) sutskever2014Sutskever, I., Vinyals, O. Le, QV.  2014\.  Sequence to sequence learning with neural networks Sequence to sequence learning with neural networks.  Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 ( 3104–3112).  Cambridge, MA, USAMIT Press.
  * Tsoumakas . (2011) tsoumakas2011Tsoumakas, G., Katakis, I. Vlahavas, I.  2011July.  Random k-labelsets for multilabel classification Random k-labelsets for multilabel classification.  IEEE Transactions on Knowledge and Data Engineering2371079-1089.
  * Understanding LSTMs (2015) lstm2015Understanding LSTMs. Understanding LSTMs.  2015\.  http://colah.github.io/posts/2015-08-Understanding-LSTMs/.  Accessed: 2018-09-20
  * Welte Wang (2014) welte2014Welte, TM. Wang, K.  2014Mar01.  Models for lifetime estimation: An overview with focus on applications to wind turbines Models for lifetime estimation: An overview with focus on applications to wind turbines.  Advances in Manufacturing2179–87.
  * Xiao (2016) xiao2016Xiao, W.  201603\.  A probabilistic machine learning approach to detect industrial plant faults A probabilistic machine learning approach to detect industrial plant faults.  International Journal of Prognostics and Health Management711.
  * Xie . (2016) xie2016Xie, C., Yang, D., Huang, Y. Sun, D.  2016\.  Feature extraction and ensemble decision tree classifier in plant failure detection Feature extraction and ensemble decision tree classifier in plant failure detection.  International Journal of Prognostics and Health Management7.
  * Yamanishi Takeuchi (2002) yamanishi2002unifyingYamanishi, K. Takeuchi, Ji.  2002\.  A unifying framework for detecting outliers and change points from non-stationary time series data A unifying framework for detecting outliers and change points from non-stationary time series data.  Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining ( 676–681).
  * Zheng . (2017) zheng2017Zheng, S., Ristovski, K., Farahat, A. Gupta, C.  2017June.  Long Short-Term Memory network for remaining useful life estimation Long Short-Term Memory network for remaining useful life estimation.  2017 IEEE International Conference on Prognostics and Health Management (ICPHM) 2017 IEEE International Conference on Prognostics and Health Management (ICPHM) ( 88-95).


# Explaining Tree Model Decisions in Natural Language for Network Intrusion
Detection

Noah Ziems, Gang Liu, John Flanagan, Meng Jiang
University of Notre Dame
{nziems2, gliu7, jflanag5<EMAIL_ADDRESS>
###### Abstract

Network intrusion detection (NID) systems which leverage machine learning have
been shown to have strong performance in practice when used to detect
malicious network traffic. Decision trees in particular offer a strong balance
between performance and simplicity, but require users of NID systems to have
background knowledge in machine learning to interpret. In addition, they are
unable to provide additional outside information as to why certain features
may be important for classification.

In this work, we explore the use of large language models (LLMs) to provide
explanations and additional background knowledge for decision tree NID
systems. Further, we introduce a new human evaluation framework for decision
tree explanations, which leverages automatically generated quiz questions that
measure human evaluators’ understanding of decision tree inference. Finally,
we show LLM generated decision tree explanations correlate highly with human
ratings of readability, quality, and use of background knowledge while
simultaneously providing better understanding of decision boundaries.

##  1 Introduction

Network intrusion detection (NID) systems monitor incoming and outgoing
network traffic to detect potentially malicious activity. When suspicious
traffic is detected, a system administrator is alerted for further
investigation. Early NID systems gained popularity by analysing traffic with
simple rule-based approaches [1]. However, over time new types of attacks such
as malware, phishing, worms, sql injection, scan attacks, and distributed
denial-of-service (DDoS) attacks have significantly increased the vulnerable
attack surface of networks. As the kinds of attacks increase, so too does the
complexity of detecting them using rule-based methods alone. In response,
machine learning (ML) and data mining based NID systems which learn to detect
malicious traffic from large amounts of real data began to emerge [2].
However, due to their black-box nature, understanding why and how a model has
come to a particular conclusion remains a significant limitation of ML based
NID systems in real-world applications. While decision trees offer a strong
balance between simplicity and performance, they still require a user familiar
with their underlying mechanics to interpret why a particular prediction was
made. Efforts to assist in explaining decision-tree based NID systems often
rely on providing additional information such as quantifying the importance of
features for a given prediction [3, 4, 5]. How one can create decision tree
explanations for NID systems that can be easily understood by users with no
background knowledge of machine learning remains an open area of research.
Further, how to measure the quality of these explanations also remains an open
question.

The recent success of large language models (LLMs) has enabled high quality
open-ended text generation for a wide variety of tasks such as long-form
question answering, summarization, arithmetic reasoning, and more. In this
work, we explore whether large language models can be used to generate high
quality explanations from decision tree inference, and we explore new
approaches for measuring the quality of decision tree explanations.

Our main contributions are as follows:

  1. 1.

We introduce Large Language Model Decision Tree Explanations (LLM-DTEs), an
approach for generating decision tree explanations targeted at users with no
background knowledge of machine learning.

  2. 2.

We introduce a new framework of human evaluation of decision tree explanations
which relies on an evaluator’s ability to answer counterfactual quiz
questions.

  3. 3.

We apply LLM-DTE to network intrusion detection (NID) and show it scores
highly with human evaluators.

Figure 1: The overall framework for our proposed approach (LLM-DTE). Given
network traffic data, a decision tree evaluates the data and classifies it as
either Benign or Threat. Path and structure data from the decision tree are
then provided to a large language model to generate an explanation. At the
same time, quiz questions are generated using rule-based templates (not LLMs).
Finally, a human evaluator is provided with the generated explanation and
tasked with answering quiz questions to test their understanding.

##  2 Related Work

###  2.1 AI Explainability

Recent years have seen a near exponential growth in research activity
surrounding AI explainability [6]. Where interpretability tends to focus on
understanding a prediction based on the inherent mechanics of a model [7],
explainability often focuses instead on post-hoc understanding targeting
particular audiences [6]. Numerous studies have explored methods like LIME,
SHAP, and TreeExplainer to enhance model interpretability by identifying and
simplifying influential features and structures within complex models [8, 9,
10].

Prior work has explored the use of prompting large language models to generate
explanations to assist in performance on reasoning tasks [11, 12, 13].
However, while some work discards these explanations as unwanted generation
artifacts, there is limited work exploring these explanations in practice.
Most similar to our work, Gavriilidis, et al. (2022) explore combining
distilled decision trees with Natural Language Explanations [14] to provide
autonomous underwater vehicle (AUV) operators with rule-based natural language
explanations [15].

###  2.2 Network Intrusion Detection

Network Intrusion Detection (NID) systems which aim to detect malicious
behavior in real-world network traffic in real time have been extensively
studied [1, 16]. Early NID systems relied on data mining techniques and
machine learning approaches due to their strong performance [2, 17]. More
recently, deep learning based approaches have been also been applied to NID
[18]. However, both traditional and deep learning approaches struggle in
practice due to their lack of explainability. Previous work has explored using
XAI approaches to assist with explainability. Local Explanation Method using
Nonlinear Approximation (LEMNA) was introduced to help find decision
boundaries in security applications such as binary code analysis [3]. A number
of previous works have also created frameworks which apply SHAP to NID [4, 5].

Despite existing work in NID explainability, nearly all prior work aims to
provide additional metrics to assist practitioners in understanding the
important features or decision boundaries. Relatively little attention has
been paid to generating natural language explanations which do not rely on any
prior knowledge of machine learning. Further, little work has been done to
provide background information as to why particular features may be more
relevant than others.

##  3 Proposed Method

In this section we introduce both our proposed approach of generating decision
tree explanations as well as our proposed framework for human evaluation of
decision tree explanations.

###  3.1 LLM-DTE

Given a well-trained decision tree, $\mathcal{T}$, and a vector of input
features $x$, our goal is to generate a natural language explanation that
describes how $\mathcal{T}$ used $x$ to generate a prediction $\hat{y}$.

The set of all nodes in $\mathcal{T}$ can be denoted as $\mathcal{N}$ where
each node $n\in\mathcal{N}$ has an associated decision function $d(n)$ that
maps input feature vector $x$ to either True or False for that particular
node, even if it is not a leaf node. In the case of NID, these are often
Benign and Threat instead. Then, $\mathcal{E}$ is defined as the set of edges
in $\mathcal{T}$ where each edge $e\in\mathcal{E}$ connects two nodes in
$\mathcal{N}$. A path $p$ is a sequence of nodes $n_{1},n_{2},\dots,n_{k}$ and
their associated edges $e_{1},e_{2},\dots,e_{k-1}$ such that $n_{1}$ is the
root node of $\mathcal{T}$ and $e_{i}$ is the edge connecting $n_{i}$ and
$n_{i+1}$ where $i\in[1,k-1]$. The decision function $d(n)$ can then be used
to map the last node $n_{k}\in p$ to final prediction
$\hat{y}\in\\{\textit{Benign},\textit{Threat}\\}$.

Generating an explanation in this setting can then be defined as finding a
function which generates a natural language explanation given $\mathcal{T}$,
$p(x)$, and $d(n)$.

We experiment with a variety of implementations of this function which can
largely be split into two categories: Rule-Based and LLM-Based.

####  3.1.1 Rule-Based Explanations

Rule-based decision tree explanations rely on simple text-based templates for
generating explanations. For instance, one can traverse each node in path $p$
then generate a sentence about why that node was traversed instead of its
sibling, such as "Petal Width was greater than 0.8cm, so the right path was
chosen, which favors, Iris virginia." While Rule-based decision tree
explanations are simple to implement, they have many downsides. Often, they
still require some background knowledge of decision trees by the user, which
can lead to poor understanding if the user is unfamiliar with machine
learning. Further, they are not able to incorporate background knowledge to
explain how the various features and classes may interact with each other. For
example, they are not capable of indicating why Petal Width may be important
for distinguishing iris flowers.

####  3.1.2 LLM-Based Explanations

LLM-based decision tree explanations (LLM-DTE) seek to remedy the issues with
Rule-based explanations by abstracting away the underlying mechanics of the
decision tree as well as providing background knowledge to explain the
features and classes. To do this, they leverage LLMs which have been
autoregressively pretrained on large corpora spanning a wide range of topics.
Similar to Rule-based explanation approaches, templates are used to convert
important data from the path and the decision tree into a text format. Unlike
Rule-based explanations though, the filled-in template is provided as a prompt
to a LLM which follows the prompt instructions to generate a natural language
decision tree explanation.

We explore a variety of different prompt templates in an attempt to provide
high-quality, relevant background knowledge for each explanation. Following
the example of classifying Iris flowers, at each node the language model can
be prompted to explain why the decision tree favors some classes more than
others when a given feature is beyond a particular threshold. For example,
given input features with high Petal Width and high Sepal Width which the
decision tree classified as Iris-virginica, the prompted LLM is able to
generate an explanation that describes how Iris-virginica tends to have longer
and wider petals and sepals compared to the other two species.

Further, additional information can be provided to the language model to
assist in generating more nuanced explanations. For instance, by calculating
the nodes along the path that have the highest information gain and providing
that information in the prompt template, the LLM is able to emphasize key
points along the path where certain classes became heavily favored over
others. This is particularly useful for larger decision trees where not all
nodes along a path may include information worth mentioning in the
explanation.

LLM-based explanations also allow for tuning the explanation to a target user.
If a user has limited knowledge of network intrusion detection, the model can
be prompted to generate an explanation tailored to their level of knowledge.

###  3.2 Human Evaluation Framework

For evaluation, we propose a new framework to test the understanding of human
evaluators. To do this, we construct a set of counterfactual quiz questions
based on $\mathcal{T}$. Human annotators are then given a natural language
explanation and quizzed with the questions to evaluate their understanding.
Different explanations can then be directly compared based on the proportion
of questions correctly answered by human evaluators, as the better
explanations will lead to better understanding which will in turn lead to more
questions correctly answered.

To create the set of quiz questions, we first construct a set of constraints
$\mathcal{C}_{p}$ for path $p$ which contains the ranges of each input feature
that would traverse $p$. The constraints $\mathcal{C}_{p}$ for input $x$ to
follow path $p$ can be defined as:

$C_{p}(x)=\bigcap_{i=1}^{k-1}c_{i}(x_{j}),\text{\quad where
}c_{i}(x_{j})=\begin{cases}x_{j}\leq t_{i}&\text{if }e_{i}\text{ corresponds
to }x_{j}\leq t_{i}\\\ x_{j}>t_{i}&\text{if }e_{i}\text{ corresponds to
}x_{j}>t_{i}\end{cases}$

Where $t_{i}$ is the threshold for $n_{i}$. The set of all input vectors $x$
that satisfy all the constraints in $C_{p}$ will traverse the path $p$ in the
decision tree.

We then take $\mathcal{C}_{p}$ to construct a series of counterfactual
questions to test a human annotator’s understanding of the decision tree
inference. Specifically, we take each constraint from $\mathcal{C}_{p}$ and
generate a counterfactual question about the true value of an input feature
$x_{i}$ and a hypothetical value of the same feature.

As a simple case study, we use the Iris dataset [19] which is comprised of 150
samples each containing 4 features (Sepal Length, Sepal Width, Petal Length,
and Petal Width) along with a corresponding species label. Suppose a decision
tree is constructed with the and a sample is taken from the Iris dataset. We
first traverse the decision tree with the sample features to construct a path
through the tree. Given the path, suppose we are able to generate constraints
for each feature. Suppose one of the features, Petal Width is constrained to
be between 0.8cm and 1.75cm, and the actual true input value of that feature
for that sample is 0.9cm. Using this information, we can then generate a
series of counterfactual true/false/unsure questions that test an evaluator’s
understanding of the decision boundary around this feature. For instance, one
question would be "If the petal width had been significantly smaller (such as
0.5cm) would the outcome have been the same?" If the evaluator has a clear
understanding of how the decision was made, they would give the correct answer
of False. However, if the evaluator does not have a good understanding of how
the decision was made, such as when they are given a poor explanation of the
decision, they may give an incorrect answer of True or Unsure.

This approach has numerous advantages. First and foremost, it measures not
whether the actual explanation is correct, but instead whether the evaluator
who read it understands the nuances of the inference itself. Second, this
approach does not rely on gold-labeled explanations, as the quiz questions can
be automatically generated for any given decision tree, and the quality of
explanation is measured simply by the proportion of correct answers an
evaluator gives. Finally, this approach is entirely agnostic to the
explanation approach, as it relies solely on the evaluators’ ability to
interpret explanations and answer questions.

##  4 Experiments

In this section, we conduct our experiments to demonstrate the effectiveness
of LLM-DTE for generating high quality decision tree explanations for network
intrusion detection.

##### Large Language Model

The large language model we use to generate explanations for our experiments
is GPT-4 due to its high quality generation capabilities shown by other long
form generation tasks. Due to their relatively low quality explanations in
initial experiments, results of other models such as GPT-3, GPT-3.5, LLaMA,
and LLaMA2 are not reported.

##### Dataset

For our network intrusion detection dataset, we use NF-BoT [20], an industry
standard dataset containing 600k samples. Each sample contains a wide range of
metadata features to assist in intrusion detection including packet source and
destination ports, tcp flags, flow duration, source and destination ip
addresses, number of packets exchanged, protocol, and others. Each sample is
also labeled as either Benign or Threat.

##### Decision Tree

We train a simple decision tree on the NF-BoT dataset using Scikit-learn [21].
The max tree depth is limited to 4 to limit the complexity of the tree and
resulting explanations. The trained decision tree achieves a test set accuracy
of 98.7%. It is worth noting that the goal of this paper is not to improve the
performance of decision trees, but instead to help generate high quality
explanations of their predictions.

###  4.1 Human Evaluation

For evaluation, we use the human evaluation metric proposed in §3.2. More
specifically, we randomly select 10 samples from the test set, which are given
as input to the decision tree. For each decision tree inference, we
programmatically generate a series of quiz questions to test a human
evaluator’s understanding of the decision tree prediction. Varying numbers of
questions are asked depending on the path the decision tree. The human
evaluators are automatically scored based on whether they correctly identify
when the counterfactual inputs would have resulted in different outcomes or
not. The individual human evaluation score is then found by calculating the
number of quiz questions correctly answered for a given explanation, and the
overall score for the explanation is given by the mean and standard deviation
across the evaluators’ scores.

Each evaluator is given the same set of questions. We use 10 evaluators in
total. Half of the evaluators are selected for their background in machine
learning while the other half have a background in security.

###  4.2 Explanation Approaches

We experiment with a wide variety of different implementations to generate
decision tree explanations which can largely be split into two categories:
Rule-Based and LLM-Based.

##### Rule Based Explanations

Following §3.1.1, we implement a rule-based explanation approach which simply
traverses each node in path $p$ then constructs a sentence about why that node
was traversed instead of its sibling. Details about what the individual
features represent are provided.

##### LLM-Based Explanations

Following §3.1.2, we also implement an LLM-based explanation approach. To
construct our LLM prompt, we first describe the task of network intrusion
detection and provide feature descriptions along with a string-based
representation of the trained decision tree. Similar to our rule-based
explanation, we add a sentence for each node traversed in path $p$, providing
data such as the feature being considered, the threshold of the node, the
split between classes in the training set, and the value of the feature in
question. The predicted label from the decision tree is provided and
emphasized. Finally, the prompt is ended with instructions to describe in
simple terms why the decision tree came to its conclusion.

After providing the prompt to the LLM, the generated explanation is recorded.

###  4.3 Results

Table 1: Human Evaluators’ Quiz Scores

Explanation Approach Quiz Score Rule-based 12.4 $\pm$ 6.5 (out of 25) LLM-DTE
(Ours) 17.3 $\pm$ 5.6 (out of 25) Overall 30.4 $\pm$ 12.1 (out of 50)

The results from the human evaluators’ quiz scores are shown in Table 1. The
evaluators scored much higher on the questions when given an LLM-based
explanation than when given a rule-based explanation, indicating that the LLM-
based explanation approach significantly outperforms the rule-based approach.
Specifically, LLM explanations led to 39.5% higher quiz scores than rule-based
approaches.

Table 2: Human Evaluators’ Qualitative Preferences of Rule-based and LLM-based
explanations. Results are presented as percentages averaged over evaluators
and questions in each category.

Readability Quality Background Knowledge Approach Low Medium High Low Medium
High Low Medium High Rule-based 30 $\pm$ 12.2 40 $\pm$ 7.1 30 $\pm$ 10 22
$\pm$ 4.5 48 $\pm$ 4.5 30 $\pm$ 7.1 16 $\pm$ 5.5 54 $\pm$ 13.4 30 $\pm$ 10
LLM-DTE (Ours) 12 $\pm$ 10.9 46 $\pm$ 11.4 42 $\pm$ 4.5 6 $\pm$ 5.5 42 $\pm$
16.4 52 $\pm$ 13.0 2 $\pm$ 4.5 46 $\pm$ 11.4 52 $\pm$ 13.0

Human evaluators were also tasked with qualitatively scoring explanations
based on their readability, overall quality, and use of background knowledge
on a scale of Low, Medium, and High. The qualitative preferences of human
evaluators are shown in in Table 2. Percentages are averaged over all
questions.

Evaluators preferred LLM-based explanations to Rule-based ones for
readability, quality, and background knowledge. Specifically, LLM-based
explanations were rated highly on readability 15% more often than rule-based
explanations. Similarly, LLM-based explanations were rated highly nearly twice
as often for quality, and 70% more often for use of background knowledge.

These results indicate not only that human evaluators strongly preferred LLM-
based explanations to rule-based ones, but also that our proposed human
evaluation framework correlates strongly with human evaluators’ preferences.

##  5 Conclusion and Future Work

In this paper, we explore whether large language models can be used to
generate explanations for decision trees used for network intrusion detection.
We find LLM generated explanations of decision trees are rated highly by human
evaluators in terms of readability, quality, and use of background knowledge.
Further, we introduce a new framework for human evaluation of decision tree
explanations that relies on automatically generated quiz questions to test the
understanding of human evaluators.

There are a number of exciting directions for future work. While effort has
been put into generating decision tree explanations in the closed book
setting, it remains to be seen whether retrieving relevant documents similar
to dense passage retrieval (DPR) [22] would further improve explanations.

## Limitations

There are a number of limitations of LLM-based decision tree explanations. Due
to their usage of large language models and the length of text being
generated, they are relatively slow and expensive to generate. Further, as the
decision tree depth increases, the prompt provided to the language model
lengthens and explanation quality begins to degrade.

## Acknowledgements

This work was supported by ONR N00014-22-1-2507.

## References

  * [1] Biswanath Mukherjee, L Todd Heberlein, and Karl N Levitt.  Network intrusion detection.  IEEE network, 8(3):26–41, 1994.
  * [2] Wenke Lee, Salvatore J Stolfo, and Kui W Mok.  A data mining framework for building intrusion detection models.  In Proceedings of the 1999 IEEE Symposium on Security and Privacy (Cat. No. 99CB36344), pages 120–132. IEEE, 1999.
  * [3] Wenbo Guo, Dongliang Mu, Jun Xu, Purui Su, Gang Wang, and Xinyu Xing.  Lemna: Explaining deep learning based security applications.  In proceedings of the 2018 ACM SIGSAC conference on computer and communications security, pages 364–379, 2018.
  * [4] Pieter Barnard, Nicola Marchetti, and Luiz A DaSilva.  Robust network intrusion detection through explainable artificial intelligence (xai).  IEEE Networking Letters, 4(3):167–171, 2022.
  * [5] Maonan Wang, Kangfeng Zheng, Yanqing Yang, and Xiujuan Wang.  An explainable machine learning framework for intrusion detection systems.  IEEE Access, 8:73127–73141, 2020.
  * [6] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  Information Fusion, 58:82–115, 2020.
  * [7] Gang Liu, Tong Zhao, Jiaxin Xu, Tengfei Luo, and Meng Jiang.  Graph rationalization with environment-based augmentations.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1069–1078, 2022.
  * [8] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  "why should I trust you?": Explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135–1144, 2016.
  * [9] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765–4774. Curran Associates, Inc., 2017.
  * [10] Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee.  From local explanations to global understanding with explainable ai for trees.  Nature Machine Intelligence, 2(1):2522–5839, 2020.
  * [11] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  * [12] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  arXiv preprint arXiv:2203.11171, 2022.
  * [13] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models.  arXiv preprint arXiv:2305.10601, 2023.
  * [14] Francisco Javier Chiyah Garcia, David A Robb, Xingkun Liu, Atanas Laskov, Pedro Patron, and Helen Hastie.  Explainable autonomy: A study of explanation styles for building clear mental models.  In 11th International Conference of Natural Language Generation 2018, pages 99–108. Association for Computational Linguistics, 2018.
  * [15] Konstantinos Gavriilidis, Andrea Munafo, Helen Hastie, Conlan Cesar, Michael DeFilippo, and Michael R Benjamin.  Towards explaining autonomy with verbalised decision tree states.  arXiv preprint arXiv:2209.13985, 2022.
  * [16] Vern Paxson.  Bro: a system for detecting network intruders in real-time.  Computer networks, 31(23-24):2435–2463, 1999.
  * [17] Anna L Buczak and Erhan Guven.  A survey of data mining and machine learning methods for cyber security intrusion detection.  IEEE Communications surveys & tutorials, 18(2):1153–1176, 2015\.
  * [18] Ravi Vinayakumar, Mamoun Alazab, KP Soman, Prabaharan Poornachandran, Ameer Al-Nemrat, and Sitalakshmi Venkatraman.  Deep learning approach for intelligent intrusion detection system.  Ieee Access, 7:41525–41550, 2019.
  * [19] R. A. Fisher.  Iris.  UCI Machine Learning Repository, 1988.  DOI: https://doi.org/10.24432/C56C76.
  * [20] Mohanad Sarhan, Siamak Layeghy, Nour Moustafa, and Marius Portmann.  Netflow datasets for machine learning-based network intrusion detection systems.  In Big Data Technologies and Applications: 10th EAI International Conference, BDTA 2020, and 13th EAI International Conference on Wireless Internet, WiCON 2020, Virtual Event, December 11, 2020, Proceedings 10, pages 117–135. Springer, 2021.
  * [21] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [22] Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  arXiv preprint arXiv:2004.04906, 2020.