11institutetext: 1 Dept of Computer Science & Engineering, Chittagong
University of Engineering & Technology, Chittagong-4349, Bangladesh.
2Jahangirnagar University, Dhaka, Bangladesh.
3 Canberra Institute of Technology, Canberra, Australia.
4 Department of Information Science, Kuwait University, Shadadiya, Kuwait.
$*$Correspondence<EMAIL_ADDRESS>
# An Isolation Forest Learning Based Outlier Detection Approach for
Effectively Classifying Cyber Anomalies

Rony Chowdhury Ripan1 Iqbal H. Sarker1,∗ Md Musfique Anwar2 Md. Hasan Furhad3
Fazle Rahat1 Mohammed Moshiul Hoque1 Muhammad Sarfraz4

###### Abstract

Cybersecurity has recently gained considerable interest in today’s security
issues because of the popularity of the Internet-of-Things (IoT), the
considerable growth of mobile networks, and many related apps. Therefore,
detecting numerous cyber-attacks in a network and creating an effective
intrusion detection system plays a vital role in today’s security. However, it
is difficult to accurately model cyber threats since modern security databases
contain large number of security features that could include Outliers. In this
paper, we present an Isolation Forest Learning-Based Outlier Detection Model
for effectively classifying cyber anomalies. In order to evaluate the efficacy
of the resulting Outlier Detection model, we also use several conventional
machine learning approaches, such as Logistic Regression (LR), Support Vector
Machine (SVM), AdaBoost Classifier (ABC), Naive Bayes (NB), and K-Nearest
Neighbor (KNN). The effectiveness of our propsoed Outlier Detection model is
evaluated by conducting experiments on Network Intrusion Dataset with
evaluation metrics such as precision, recall, F1-score, and accuracy.
Experimental results show that the classification accuracy of cyber anomalies
has been improved after removing outliers.

###### Keywords:

cybersecurity, Outlier detection, Network intrusion detection system, Machine
learning, Cyber data analytics

##  1 Introduction

The modern digital world is overwhelmed by an unprecedented amount of data in
various domains [12]. Network intrusion is related to cybersecurity data [13],
which refers to any attempt to undermine the host and network’s
confidentiality, credibility, or availability [4], and is one of cyberspace’s
most common threats. An intrusion detection system (IDS) is a network security
device that tracks real-time network traffic and can warn or take corrective
action when unauthorized transmissions are discovered [8]. The fundamental
motive for detecting the intrusion is to develop classifier accuracy that can
effectively recognize the invasive behavior [21]. Unfortunately, new security
threats are quickly emerging since the attackers become more advanced than
before and as a result, the possibility of damaging vital infrastructures
increases dramatically in recent time. Thus, it is very important for IDS to
detect and deal with novel attacks.

Supervised classification techniques can learn well from a security dataset
that can play a major role in IDS [13, 15]. However, modeling cyber-attacks
effectively is problematic due to the high dimensions of security features and
the presence of outliers in today’s security datasets [13]. Outliers are the
data that differs significantly from normal behavior, also known as noisy
instances in data [9] [13]. Outliers found in a security model may also cause
various problems, such as over-fitting problems in a classification model. The
computing time and cost can be very high to train such model due to the
scarcity of model generalization. As a result, the performance of security
model degrades in terms of classification accuracy.

In this paper, we propose a model based on the Isolation Forest algorithm. At
first, we perform necessary preprocessing steps like categorical feature
encoding, feature scaling [11] to extract fifteen essential features to fit
into the proposed model. Finally, we applied five popular classification
algorithms [14] such as Logistic Regression (LR), Support Vector Machine
(SVM), AdaBoost Classifier (ABC), Naive Bayes (NB), and K-Nearest Neighbor
(KNN) to evaluate the performance of our system.

The rest of the paper is organized as follows: Section 2 reviews related works
of the outlier detection system and network intrusion detection system. In
Section 3, we present our proposed outlier detection model. Our experimental
design is described in Section 4, including a brief discussion. Finally,
Section 5 concludes the paper and underlines the future work.

##  2 Related Work

A considerable amount of research has been devoted to the problem of outlier
detection. For example, Mascaro et al. [6] proposed an anomaly detection
system by combining dynamic and static Bayesian network models to improve
anomaly detection performance. The authors in [7] proposed a method using
standard SVM to classify the sensor node data into different categories, such
as a local outlier or cluster outlier, or network outlier. Sun et al. [19]
presented an anomalous user behavior detection framework that applies an
extended version of the Isolation Forest algorithm to an enterprise dataset to
isolate anomalous instances. Stripling et al. [18] represented the iForestCAD
approach that computes conditional anomaly scores to detect fraud. Ben-Gal et
al. [2] presented multiple approaches for outlier identification that can be
divided between univariate vs. multivariate techniques as well as parametric
vs. nonparametric procedures. Aggarwal et al. [1] proposed a method which is
capable of finding lower-dimensional projections that are locally sparse. As a
result, their method is suitable for outlier detection in high dimensional
data.

There are numerous studies on network intrusion detection. For Instance, Song
et al. [17] represented a deep convolutional neural network based intrusion
detection system. Shapoorifard et al. [16] proposed a Hybrid IDS model to
improve the KNN classifier in the remaining intrusion detection task, which
combines K-MEANS clustering and KNN classification. Yan et al. [20] proposed a
new network intrusion detection method that builds an improved transductive
SVM by introducing simulated annealing to degenerate the optimization problem
and then apply Support Vector Classifier. However, all these foregoing
approaches did not consider the impact of outlier instances on the
classification results. Our proposed framework considers this factor and can
improve the classification accuracy of cyber anomalies in a network intrusion
dataset.

##  3 Methodology

In order to provide complete coverage of the system, we model an approach that
has different stages, as presented in Fig. 1. Firstly, the pre-processing is
performed on raw data to normalize the values of the features. Secondly, we
select important features using Recursive Feature Elimination (RFE) [10].
Next, we apply the Isolation forest algorithm to train the instances of the
dataset. Finally, we analyze the effectiveness of the proposed outlier
detection model.

Figure 1: The Proposed Methodology

###  3.1 Dataset & Data Preprocessing

To develop an efficient outlier detection model, it is necessary to understand
the characteristics of network data. In this study, we used a network
intrusion dataset that is publicly available in Kaggle [3]. This dataset
contains a total of 41 features of which, 3 of those features are Nominal, 15
of those features are Float, and the remaining 23 features are Integer. The
target feature set consists of two (normal and anomaly) types of class
variables. The dataset was created by simulating a standard local area network
of the US Air Force, which is exposed to numerous cyber attacks known as
intrusions [3]. In terms of data distribution, all the features are not
identical and differ one from another. So, the pre-processing of this raw data
is needed to build the proposed Isolation Forest Learning-based outlier
detection model. Data pre-processing requires both encodings and scaling on
the features based on the specified network intrusion dataset’s
characteristics. Although most of the features in this intrusion dataset are
quantitative, it has 3 qualitative features such as: protocol_type, flag, and
service. These three features need to be encoded to build the Isolation Forest
Learning-based outlier detection model. In this study, we used “Label
Encoding” instead of “One Hot Encoding” to avoid the feature increase problem
of “One Hot Encoding” [10]. As we described earlier, the security features’
values vary from feature to feature in different ranges. So, We used a
Standard Scaler to normalize the safety features with an average value of 0
and a standard deviation of 1.

###  3.2 Important Features Extraction

This network intrusion dataset has 41 features that are used in our modeling
purpose. Having too many irrelevant features can increase overfitting,
training time and decrease the accuracy. Therefore, selecting the essential
features that contribute most to the prediction variable or output in a
specific dataset is necessary [10]. In this work, we apply the Recursive
Feature Elimination (RFE) method for this purpose. RFE is a wrapper method
with two important configuration options: the choice in the number of features
to select and the choice of the algorithm that helps to choose those features.
We choose 15 features that have been selected using Extra Tree Classifier with
the help of the RFE technique.

###  3.3 Outlier Detection with Isolation Forest

Isolation forest [5] is an unsupervised outlier detection algorithm that works
rather than profiling normal points on the notion of isolating outliers. The
most common outlier detection techniques are focused on building a profile of
“normal” instances, after which the outliers are recorded as those that do not
adhere to the normal profile in the dataset. Isolation Forest explicitly
isolates outlier instances in the dataset. Two quantitative properties of
outlier data points in a sample are the basis for Isolation Forest. They are
fewer and have feature values that are very different from those of “normal”
instances. They are the minority.

Figure 2: Example of an isolation tree (iTree) on Network Intrusion data. The
red triangle represents outlier data point

Isolation Forest builds an ensemble of “iTrees” (Isolation Trees) on network
intrusion dataset similar to the example in Fig. 2. The algorithm takes n
random samples of size m from a given dataset. For each random sample, the
“iTree” is built by splitting the sub-sample instances over a split value of a
randomly selected feature so that the instances whose corresponding feature
value is smaller than the split value go left, and the others go right, and
the process continues recursively until the tree is entirely built. The split
value is selected at random between the minimum and maximum values of the
selected feature. In addition, the “iTree” comprises two groups of nodes;
internal nodes and external nodes. Internal nodes are non-leaf, containing the
split value, split features, and pointing to two child subtrees. External
nodes are leaf nodes that can not be further separated and reside at the
bottom of the tree and carry the unbuilt subtree’s size to measure the outlier
value. Finally, outliers are the points that have shorter average path lengths
on the “iTrees.”

The outlier score of an instance is calculated based on the observation that
the structure of “iTrees” is similar to that of Binary Search Trees (BST): the
termination of the external node “iTree” corresponds to the failure of the BST
search. Consequently, the estimate of the average h(x) for the termination of
the external node is the same as an unsuccessful search in BST. That is,

${\displaystyle c(m)={\begin{cases}2H(m-1)-{\frac{2(m-1)}{m}}&{\text{for
}}m>2\\\ 1&{\text{for }}m=2\\\ 0&{\text{otherwise}}\end{cases}}}$ (1)

where $n$ is the testing data size, $m$ is the size of the sample set, and $H$
is the harmonic number, which can be estimated by

${\displaystyle H(i)=ln(i)+0.5772156649\text{ (Euler-Mascheroni constant.)}}$
(2)

The value of $c(m)$ in Equation 1 is the average $h(x)$ for a given $m$. Then
it is used to normalize $h(x)$ to get an estimate of outlier score for a given
instance $x$ as:

${\displaystyle s(x,m)=2^{\frac{-E(h(x))}{c(m)}}}$ (3)

where $E(h(x))$ is the average value of $h(x)$ from a collection of “iTrees”.
Finally, instance $x$ is assigned to outlier if the value of $s$ is close to
1, otherwise x is likely to be considered normal instance if the value of $s$
is smaller than 0.5. After detecting outliers using the Isolation Forest
algorithm, we remove all the outliers to evaluate the system’s improved
performance in terms of classification accuracy. To achieve this goal, we
employ five popular machine learning classification techniques [14], such as
Logistic Regression (LR), Support Vector Machine (SVM), AdaBoost Classifier
(ABC), Naive Bayes (NB), and K-Nearest Neighbor (KNN) to classify cyber
anomalies in the Network Intrusion dataset.

##  4 Experimental Evaluation

In this section, we describe all the performance metrics used in this work to
test our proposed model. In order to test our model, we use Precision, Recall,
Accuracy, and F1-score.

$\displaystyle Precision=\frac{TP}{TP+FP}$ (4) $\displaystyle
Recall=\frac{TP}{TP+FN}$ (5) $\displaystyle
Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$ (6) $\displaystyle
F1-score=2\times\frac{Precision\times Recall}{Precision+Recall}$ (7)

###  4.1 Experimental Setup

All tests are performed on an 8 GB RAM Intel Core i5 2.50GHz CPU. The proposed
Isolation Forest Learning-Based Outlier Detection model is implemented in
Python with packages scikit-learn under OS Windows 10.

###  4.2 Implementation & Performance Evaluation

As we stated earlier in Section 3, first we preprocess raw dataset for fitting
into the Isolation Forest model. At first, all the categorical data are
transformed into numerical value using Label Encoder. Next, all the features
are normalized using Standard Scaler. Then, Recursive Feature Elimination
(RFE) is used for feature selection. In this work, we select 15 feature out of
41 features. Next, data is fitted into the Isolation Forest Model that assigns
a value of 1 to an instance if it is a normal instance, otherwise assigns -1
if it is an outlier. A scatter plot of normal and outlier data points of
network intrusion data is represented in Fig. 3(a).

(a) Original Dataset

(b) Without Outlier

Figure 3: Scatter plot of Network Intrusion Data

After that, all the outliers have been removed to improve the classification
accuracy of cyber anomalies. A scatter plot of network intrusion data after
removing outliers is represented in Fig. 3(b). For measuring classification
accuracy, we apply five popular classification algorithms; Logistic Regression
(LR), Support Vector Machine (SVM), AdaBoost Classifier (ABC), Naive Bayes
(NB), and K-Nearest Neighbor (KNN). We observe that the classification
accuracy of cyber anomalies has been improved after removing outliers except
for SVM. The proposed model also demonstrates improved performance in terms of
Precision, Recall, F1-score values. Table 1 shows that the classifiers KNN,
SVM, NB, LR and ABC have accuracy of 99%, 99%, 90%, 96% and 99%, respectively.
After removing outliers, KNN, SVM, NB, LR, and ABC classifiers have an
accuracy of 100%, 99%, 95%, 98%, and 100%, respectively. For a better view, a
graphical representation of accuracy comparison has shown in Fig 4.

Figure 4: Bar Chart of Classification Accuracy

(a) Original Dataset

(b) Without Outlier

Figure 5: ROC curve of various machine learning classification models Table 1: comparison of Precision, Recall, Accuracy, F1-score Dataset | classification | Accuracy | Precision | Recall | F1-score
---|---|---|---|---|---
| models | (%) | (%) | (%) | (%)
Original dataset | KNN | 99 | 99 | 99 | 99
SVM | 99 | 99 | 99 | 99
NB | 90 | 90 | 90 | 90
LR | 96 | 96 | 96 | 96
ABC | 99 | 99 | 99 | 99
Without Outlier | KNN | 100 | 100 | 100 | 100
SVM | 99 | 99 | 99 | 99
NB | 95 | 95 | 94 | 95
LR | 98 | 98 | 98 | 98
ABC | 100 | 100 | 100 | 100

Moreover, to evaluate the performance of our Isolation Forest learning-based
outlier detection model, the Receiver Operating Characteristic (ROC) curve of
the five classifiers for the dataset with and without outliers have been shown
in Fig. 5(a) and Fig. 5(b). From these two figures, it is observed that after
removing outliers, LR and NB have a better area under the ROC curve (AUC) of
0.996 and 0.997, respectively, than the original dataset.

##  5 Conclusion

In this paper, we present an Isolation Forest Learning-Based outlier detection
approach to classify Cyber anomalies effectively. In our approach, we have
first considered feature selection according to their importance and then
applied the Isolation Forest to detect outliers. After that, we remove all the
outliers to make the network intrusion detection model more effective
regarding classification accuracy. Finally, by performing several tests on the
Network Intrusion Dataset, the efficacy of the proposed outlier detection
model is tested. Experimental results show that the classification accuracy of
cyber anomalies has been improved after the removal of outliers. For future
work, the efficacy of our propsoed Isolation Forest Learning-based Outlier
Detection model can be evaluated by gathering large data sets with more
security feature dimensions in IoT security services and also evaluate their
efficiency at the application level in the cybersecurity domain.

## References

  * [1] Aggarwal, C.C., Yu, P.S.: Outlier detection for high dimensional data. In: Proceedings of the 2001 ACM SIGMOD international conference on Management of data. pp. 37–46 (2001)
  * [2] Ben-Gal, I.: Outlier detection. In: Data mining and knowledge discovery handbook, pp. 131–146. Springer (2005)
  * [3] Bhosale, S.: Network intrusion detection (Oct 2018), https://www.kaggle.com/sampadab17/network-intrusion-detection
  * [4] Buczak, A.L., Guven, E.: A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Communications surveys & tutorials 18(2), 1153–1176 (2015)
  * [5] Liu, F.T., Ting, K.M., Zhou, Z.H.: Isolation forest. In: 2008 Eighth IEEE International Conference on Data Mining. pp. 413–422. IEEE (2008)
  * [6] Mascaro, S., Nicholso, A.E., Korb, K.B.: Anomaly detection in vessel tracks using bayesian networks. International Journal of Approximate Reasoning 55(1), 84–98 (2014)
  * [7] Mohamed, M.S., Kavitha, T.: Outlier detection using support vector machine in wireless sensor network real time data. Int J Soft Comput Eng 1(2) (2011)
  * [8] Samrin, R., Vasumathi, D.: Review on anomaly based network intrusion detection system. In: 2017 International Conference on Electrical, Electronics, Communication, Computer, and Optimization Techniques (ICEECCOT). pp. 141–147. IEEE (2017)
  * [9] Sarker, I.H.: A machine learning based robust prediction model for real-life mobile phone data. Internet of Things 5, 180–193 (2019)
  * [10] Sarker, I.H., Abushark, Y.B., Alsolami, F., Khan, A.I.: Intrudtree: A machine learning based cyber security intrusion detection model. Symmetry 12(5), 754 (2020)
  * [11] Sarker, I.H., Alqahtani, H., Alsolami, F., Khan, A.I., Abushark, Y.B., Siddiqui, M.K.: Context pre-modeling: an empirical analysis for classification based user-centric context-aware predictive modeling. Journal of Big Data 7(1), 1–23 (2020)
  * [12] Sarker, I.H., Hoque, M.M., Uddin, M.K., Alsanoosy, T.: Mobile data science and intelligent apps: Concepts, ai-based modeling and research directions. Mobile Networks and Applications pp. 1–19 (2020)
  * [13] Sarker, I.H., Kayes, A., Badsha, S., Alqahtani, H., Watters, P., Ng, A.: Cybersecurity data science: an overview from machine learning perspective. Journal of Big Data 7(1), 1–29 (2020)
  * [14] Sarker, I.H., Kayes, A., Watters, P.: Effectiveness analysis of machine learning classification models for predicting personalized context-aware smartphone usage. Journal of Big Data 6(1), 57 (2019)
  * [15] Seufert, S., O’Brien, D.: Machine learning for automatic defence against distributed denial of service attacks. In: 2007 IEEE International Conference on Communications. pp. 1217–1222. IEEE (2007)
  * [16] Shapoorifard, H., Shamsinejad, P.: Intrusion detection using a novel hybrid method incorporating an improved knn. Int. J. Comput. Appl 173(1), 5–9 (2017)
  * [17] Song, H.M., Woo, J., Kim, H.K.: In-vehicle network intrusion detection using deep convolutional neural network. Vehicular Communications 21, 100198 (2020)
  * [18] Stripling, E., Baesens, B., Chizi, B., vanden Broucke, S.: Isolation-based conditional anomaly detection on mixed-attribute data to uncover workers’ compensation fraud. Decision Support Systems 111, 13–26 (2018)
  * [19] Sun, L., Versteeg, S., Boztas, S., Rao, A.: Detecting anomalous user behavior using an extended isolation forest algorithm: an enterprise case study. arXiv preprint arXiv:1609.06676 (2016)
  * [20] Yan, M., Liu, Z.: A new method of transductive svm-based network intrusion detection. In: International Conference on Computer and Computing Technologies in Agriculture. pp. 87–95. Springer (2010)
  * [21] Yin, C., Zhu, Y., Fei, J., He, X.: A deep learning approach for intrusion detection using recurrent neural networks. Ieee Access 5, 21954–21961 (2017)