ISSN 2278-3091

Volume 10, No.4, July - August 2021

Hassan Adegbola Afolabi et al., International Journal of Advanced Trends in Computer Science and Engineering, 10(4), July – August 2021, 2752 – 2761
International Journal of Advanced Trends in Computer Science and Engineering
Available Online at http://www.warse.org/IJATCSE/static/pdf/file/ijatcse161042021.pdf
https://doi.org/10.30534/ijatcse/2021/161042021

Comparison of Single and Ensemble Intrusion Detection

Techniques using Multiple Datasets
1 2
Hassan Adegbola Afolabi , Abdurazzag Aburas
1
School of Electrical, Electronic and Computer Engineering
University of Kwazulu-Natal South Africa, [email protected]
2
School of Electrical, Electronic and Computer Engineering
University of Kwazulu-Natal South Africa,[email protected]

ABSTRACT employed in order to handle intrusions more

effectively due to high speed and enormous volume
The advancement of Internet of Things (IoT) of data, the rapid development of sophisticated
technology raises numerous security concerns, as attacks and zero-day vulnerabilities on computer
new threats emerge every day. Prior to preventing networks. Intrusion detection has become a major
these threats, they must be detected. This makes area of concern. They were developed to exclusively
intrusion detection a major priority. However, monitor computer networks, detect intrusions,
datasets play a significant role in intrusion detection. attacks, unauthorized or any other malicious activities
The dataset used to evaluate machine learning-based [1-3]. This will enhance security and compliment the
solutions has an effect on their accuracy. Most of the shortcomings of other traditional security
time, these datasets do not accurately reflect real techniques.Intrusion detection systems can be
network traffic and contains lots of redundant and classified into three types based on their detection
irrelevant features that undermine Intrusion Detection method namely: anomaly-baseddetection,
System (IDS) efficiency. Motivated by the above, our knowledge-based detection, and hybrid detection.
work focuses on extracting the most relevant features The knowledge based which is also known as
from four datasets namely CICIDS2017, IoTID20, signature-based technique rely on a database that
NSL-KDD and N-BaIoT datasets using information contains signatures of existing attacks to look for a
gain approach. Then we evaluated and compared defined pattern. To keep up with emerging attacks,
some single and ensemble classifiers based four the database must be updated frequently. Therefore,
important performance metrics. Finally, these only well-known attacks can be detected by this
algorithms were combined in an ensemble learner to technique. On the other hand, anomaly-based IDSs
see how well they performed. Our findings are also known as behavior-based because they monitor
considered to be relevant in the combination of the system's, users', and network's normal behavior
strong classification algorithms in the development of and warn the administrator if any deviation occurs.
IDS systems and experimental results indicates that The ability of anomaly-based IDSs to detect novel
feature selection can yield better accuracy. threats is due to this feature. The hybrid-based
detection system refers to a system that combines
Key words: Ensemble techniques, Feature selection, anomaly-based and knowledge-based intrusion
Internet of things, Intrusion detection systems, detection.
Machine learning. An IDS's performance is greatly dependent on the
datasets used to test and analyze it. In order to
1. INTRODUCTION evaluate and test new approaches, appropriate and
valid datasets are needed. Many researchers find this
The increased use of Internet has greatly increased difficult which makes it a major task. While the
the data growth rate from different devices and majority of their tested with obsolete datasets [4],
created numerous security concerns. Various current network traffic data should be used to test
technologies like user authentication, data encryption IDS in order to make detection system more resilient
and firewall have been used to address these security [5]. However, implementing an efficient intrusion
concerns. Though these countermeasures may detection system could be a difficult task, given the
prevent many kinds of attacks, they cannot quickly abundance of redundant and irrelevant features in the
detect intrusion nor perform a thorough analysis of dataset. It is tough to monitor all the features in the
packets. Big data analysis and techniques are dataset, this could cause computational complexity

2752
Hassan Adegbola Afolabi et al., International Journal of Advanced Trends in Computer Science and Engineering, 10(4), July – August 2021, 2752 – 2761

and decreased efficiency. As a result of this, in order network abnormality. Additionally, the authors
to improve IDS detection accuracy, selected features proposed that this issue could be avoided by the
from a dataset should be extracted prior to using any combination of real-world traffic and simulated
detection approach. A preprocessing technique dataset. Later, an approach using random forest for
known as feature selection has been shown to be a misuse-based, anomaly-based and hybrid IDSs was
suitable solution for an IDS [6,7]. It discovers highly presented by authors in [10]. Numerous machine
important features and removes unnecessary ones. learning techniques with improved accuracy have
Motivated by the discussion above, our study will been developed over the past few years, a hybrid
focus on the performance of various machine approach suggested in [11] which combines K-means
learning techniques used in detection classification clustering and the radial basis function (RBF) kernel
systems when applied to four publicly available of a support vector machine (SVM) is an example of
recent datasets. such evolution. In addition to these advancements,
The contributions of this work are as follows: various performance comparisons of these intrusion
 We present an overview of intrusion detection systems have been conducted. Belavagi et
detection systems that employ machine al. [12] used the NSL-KDD dataset to evaluate
learning techniques. Logistic Regression, Gaussian Naive Bayes, Support
 A feature extraction technique known as Vector Machine, and Random Forest techniques.
Information gain to extract the best feature According to the author, Random Forest Classifier
was employed to manage large amounts of outperforms the other three algorithms. See table I
irrelevant features in the datasets. below.
 Five algorithms were evaluated, the majority Table 1
of which fall under the category of ALGORITHMS PRECISION ACCURACY
individual and ensemble classifiers. (%) (%)
 We suggested a novel approach for intrusion Gaussian Naïve 79 79
Bayes
detection that combines the benefits of
Logistic 83 84
feature selection, single and ensemble Regression
classifiers. Random Forest 76 75
 We studied the performance of our approach
Support Vector 99 99
and each analyzed classifier using four real
Machine
traffic datasets.A Comprehensive
comparison was done.
Additionally, Almseidin et al. [13] studied Random
The rest of the paper is organized as follows: In
Forest, Random Tree, Bayes Network, Naïve Bayes,
Section II, we focus on some of the major related
Decision Table, MLP and J48 machine learning
works in the area of intrusion detection. Section III
describes the experimentation procedure, tools and algorithms in 2017. However, on the KDD dataset,
methodology used in different steps of the evaluation. decision tree has the lowest false negative value
Our ensemble model is described in Section IV and (0.002), but random forest outperforms in terms of
accuracy. See table II below.
Section V discusses the results of the experiments.
The conclusion and future work are presented in
Table 2
sections VI and VII respectively.
ALGORITHMS PRECISION ACCURACY
(%) (%)
2. RELATED RESEARCH WORK Bayes Network 99.2 90.7
Decision Table 94.4 92.4
Intruders update themselves and the tools they use to J48 98.9 93.1
develop new cyber-attacks on a daily basis. Due to MLP 97.8 91.9
this, Intrusion detection techniques are being Naïve Bayes 98.8 91.2
designed at a rapid pace to ensure that network Random Forest 99.1 93.7
systems are effectively secured against newly Random Tree 99.2 90.5
developed malware. Numerous researches have been
conducted for this reason, and new ones are Likewise, Zaman et al. [14] conducted experiments to
conducted daily to improve the efficacy of IDS compare the precision, accuracy, and recall of Fuzzy
systems. Research findings in a study conducted in C-Means, Radial Basis Function, k-Nearest
[8] concludes that datasets representing exact Neighbors, Support Vector Machine, k-Means, Nave
network systems are now becoming more important Bayes, and an ensemble technique combining all six
to evaluate intrusion detection algorithms. algorithms. Kyoto+ dataset was used to evaluate
As a result, Mahoney et al. [9] studied and discovered these algorithms, and it was determined that Radial
that the DARPA/MIT Lincoln laboratory evaluation Basis Function outperformed the others. See the table
dataset results in an overly optimistic detection of below.

2753
Hassan Adegbola Afolabi et al., International Journal of Advanced Trends in Computer Science and Engineering, 10(4), July – August 2021, 2752 – 2761

[19] presented a novel framework for intrusion

Table 3 detection using ensemble approach. This architecture
ALGORITHMS PRECISION ACCURACY evaluated various supervised and unsupervised
(%) (%) machine learning algorithms on CICIDS-17 dataset.
Ensemble 88.4 96.7 Experimentshows that ensemble approach provides
Fuzzy C-Means 75 83.6 better performance.
K means 75 83.6
K-Nearest Neighbors 95.6 97.5 3. EXPERIMENTATION
Naïve Bayes 91.6 96.7
Radial Basis 92 97.5
This section briefly discusses fundamental concepts
Function
such as dataset description, pre-processing of the
Support Vector 86.9 94.2
Machine dataset, feature extraction procedure, machine and
deep learning methods used, and model design. We
examined two network datasets which reflect actual
Also, in 2018, Aljawarneh et al. [15] presented a
real-world network traffic namely: NSL-KDD and
hybrid intrusion detection model using a voting
CICIDS 2017 and Two IoT datasets namely N-BaIoT
scheme that combined Naive Bayes, J48, Random
and IoTID20. In order to obtain reliable results, the
Tree, AdaBoostM1, Decision Tree, Decision Stump,
following steps was followed before the analysis of
and Meta Bagging. As a result, 99.81 percent
the algorithms:
detection accuracy was achieved.
Hajisalem et al [1] used an Artificial Bee Colony  Import the datasets that will be used for
(ABC) and an Artificial Fish Swarm (AFS) to design algorithm training and testing into the Google
a hybrid classification approach. They used Fuzzy C- Colaboratory environment.
Means Clustering (FCM) and Correlation-based  Preprocess the data to select relevant attributes.
Feature Selection (CFS) approaches to select Mutual Information Feature selection
features. This approach was applied to UNSW-NB15 (Information Gain) was applied on all four
and NSL-KDD datasets. 99% accuracy rate was selected datasets to extract 20 most relevant
obtained. The CSE-CIC-IDS-2017 dataset was attributes.
developed by Sharafaldin et al. [4] since the current  Applied Feature engineering to create columns
datasets did not fulfill today's demand for intrusion for each attack class to enable us obtain results in
detection. A test environment was created with multiclass.
network attackers and victims to generate this  Analyzed the selected algorithms individually
dataset. Attacks such as distributed denial of service, with the preprocessed data attributes and obtain
denial of service, Infiltration attack, Web attack, results to check individual performance.
brute force, botnet and heart bleed were organized in  Combination of 3 other algorithms with DNN as
the test environment. Additionally, machine learning our base classifier (algorithms combined are
approaches were used to evaluate system (SGD, LGBM, XGBOOST AND DNN).
performance. Ferrag et al [16], studied some deep  The output of the four base classifiers was used
learning algorithms namely deep neural networks to train the ensemble model (Meta classifier
(DNN), recurrent neural networks (RNN), convoluted which uses LR algorithm).
neural networks (CNN), deep autoencoders (DA),  Test was done with a different portion of the
deep belief networks (DBN), deep Boltzmann dataset on the meta classifier to obtain
machines (DBM), and restricted Boltzmann machines experimental results based on the following
(RBM) when implemented on CSE-CIC-IDS2018 metrics (Section III.D): true positive, false
and Bot-IoT datasets. The classification success of positive, precision, recall and F- measure.
deep learning is then compared to the classification Note:
time for these data sets. Hassan et al. [17] evaluated a) A comparison was done with the performance of
ML classifiers such as Artificial Neural Network each individual algorithm so as to find the best
(ANN), Support Vector Machine (SVM), Gaussian possible algorithm to combine with the DNN to
Naïve Bayes (NB), Decision Tree (DT) and Random build the stacking model.
Forest (RF) with both KDD’99 and ISCX1DS2012 b) The choice of using Logistic Regression
datasets. The SVM outperforms other algorithms for algorithm as the metaclassifier is because it’s the
both datasets and NB algorithm was the least weakest as per our results with all other
accurate for both datasets. Additionally, a standard algorithms.
deep neural network (DNN) approach was presented
in [18] This approach was based on back propagation The experiment is performed on Google
algorithm and trained using 3 hidden layers. When Colaboratory under python 3 using TensorFlow and
evaluated using an unlabeled CICIDS2017 dataset, an Graphics Processing Unit (GPU). Installed on Mac
average accuracy of 84.5% was obtained.Authors in OS X.

2754
Hassan Adegbola Afolabi et al., International Journal of Advanced Trends in Computer Science and Engineering, 10(4), July – August 2021, 2752 – 2761

3.1 Dataset Description network. The dataset consists of 42 raw network

As discussed in the previous sections, this study packet files (pcap) collected using the wireless
considered four datasets for evaluation purpose, network adapter's monitor mode at various time
CICIDS2017, NSL-KDD, N-BaIoT 2018 and points. The new dataset, on the other hand, contains a
IoTID20 datasets. more extensive network and flow-based features.
These flow-based features could be used in the
A. CICIDS2017 analysis and evaluation of a flow-based Intrusion
The Canadian Institute for Cybersecurity generated detection system. The packet files description in this
this dataset in 2017. The dataset was created by dataset include ‘benign’ as normal traffic, and 'attack
setting up a victim and an attacker network traffic' for attack classes such as MITM ARP
laboratory environment. The CICIDS2017 is a spoofing, DoS, Scan and mirai.
publicly accessible dataset that resembles real-world
IDS network traffic [20]. Contained in this dataset are 3.2 Selection of Features
benign or ‘normal’ traffic and seven common and The selection of features (attributes/variables) is an
most recent attacks that simulates real-world data [4]. important step to develop an intrusion detection
The attacks contained in this dataset include Brute model that is effective. Given the huge number of
Force, Distributed Denial of Service (DDoS), Web, irrelevant features in network data, it is important to
Infiltration, Botnet, and Port Scan attacks. Over 2 extract only the necessary attributes to minimize
million records and 78 features are contained in processing time and achieve a higher detection rate
CICIDS2017. and accuracy. The method of feature (attribute)
selection is an important data preprocessing approach
B. NSL-KDD that is used to extract a subset of relevant features
NSL-KDD is a freely available dataset that was (variables/attributes) in order to improve the
created to address the shortcomings in the KDD 99 performance of learning algorithms. Additionally,
dataset [21–23]. A benefit of this dataset is that it has this process reduces the amount of storage
no insignificant records in the train set, which means required.Feature selection methods can be classified
that the classifiers will not be biased towards more into three categories such as embedded, filter, and
repeated records. According to [8], this dataset still wrapper methods. The Filter and Wrapper methods
lacks public network data.The NSL-KDD dataset are the most frequently used [28]. In this study, we
consists of 42 attributes. This dataset contains denial employed the mutual information gain feature
of service (DoS), remote to local (R2L), user to root selection approach.Information Gain is a single-
(U2R), and probe attacks. attribute evaluator that is used in relation with the
Ranker search method to score all attributes based on
C. N-BaIoT their information gain. This is used to evaluate the
N-BaIoT Dataset was created in 2018 to resolve the value of each attribute by calculating the information
inadequacy of publicly available botnet datasets, gain in relative to the class. The score is determined
particularly for IoT. Created using real traffic data by how much information about the classes is
from nine commercial IoT devices infected by obtained when that feature is used.The Information
authentic botnets from two families, Mirai and Gain equation is shown in Eq. 1,
BASHLITE, which are two of the most prevalent
IoT-based botnets that have already demonstrated IG(X)=H(Y)-H(Y|X) (1)
their malicious capabilities. [24-25]. The aim was to
use anomaly detection algorithms to differentiate where H(Y) and H(Y|X) are the entropy of Y and the
between benign and malicious traffic data. However, conditional entropy of Y for given X, respectively
the dataset could also be useful for multi-class [29]. For this research, only 20 attributes were
classification because the malicious data is classified selected from each dataset when considering
into 10 attacks classes carried out by two botnets, threshold values 0.29, 0.40, 0.73 and 0.42 for datasets
plus one "benign" attack class. The N-BaIoT dataset NSL-KDD, CIC2017, N-BaIoT and IoTID20 datasets
contains 115 independent features in each file, as respectively.Attributes whose information gain value
well as a class label generated from the respective file is below the considered threshold value are removed
name (e.g., "benign" or "TCP attack"). from the dataset. Figure 1- 4 below shows the
description of the selected attributes.
D. IoTID20
IoTID20 dataset is a new dataset proposed by [26].
Originally created using two basic smart home
devices, the SKT NUGU (NU100) and the EZVIZ
Wi-Fi Camera (C2C Mini O Plus 1080P) [27]. All
devices involved was connected to the same wireless

2755
Hassan Adegbola Afolabi et al., International Journal of Advanced Trends in Computer Science and Engineering, 10(4), July – August 2021, 2752 – 2761

Information Gain Values Of Attributes For CICIDS2017 Dataset 3.3 Evaluation of Algorithms
0 .6
For evaluation purpose, this project has considered
0 .5
LR, SGD, LGBM, XGBOOST, and DNN algorithms.
INFORMATION GAIN 0 .4

0 .3
A. Logistic Regression
0 .2
Logistic Regression is a type of supervised machine
0 .1
learning method that is used to classify data. It can be
0
used with categorical dependent variables. This

th
s

s/ s
rt
ax

ax
n

td

a rd

n
i ze

ce

i ze

ts

ax
a rd
s

et
y te

yte
ea

ea

Ma

io
Po
ke

ng
hS

hM

TM

M
ia n

ck

tS

te
tS

ra t
kw
rw
M

hM

dB
algorithm has gained importance in recent years and
dB

ac

Le

AT
Pa

By
gt

en

g th

ion
ar
ke

IA
gt

ac

Du
th

Fo

dP

t
en

Bw

Fw
gt
V

dI
ac

w
gm

at
en

e
d

en
_B

w
s_
ng

ck
th

Bw

en

F lo
Fw
L

st in
eP

Fw
F lo
tL

tL
ow

yte
Le

Se

t es

F lo
ow

Pa
et

ng

tL

ke
ag

of

of

ke
ck

De
By
et

b fl

B
wd
Le

bfl

ax
ke
er

ac
in_
Pa

th

th

ac
ck

Su
its application has grown tremendously. The

in_
et

eB

M
Su

ac
Av

dP
ng

ng
Pa

dP
W
ck

dP

_W
ag
Le

Le
it_

Bw
Pa

Fw
Bw

er

In it
ta l

ta l
In
Av
To

To
ATTRIBUTES objective of the logistic regression algorithm is to
Figure1: CIC2017 attributes assign data to their appropriate classes based on their
correlation.For a mathematical expression of logistic
Information Gain Attribute Values for NSL-KDD Dataset regression, let us look at a simple linear regression
1
0.9
equation below:
INFORMATION GAIN

0.8
0.7
0.6 = + ∗ (2)
0.5
0.4 apply sigmoid function to the above equation will
0.3
0.2
give:
0.1
0
= (3)
s t t t t s s e t t
te ice Fl
ag te te un un un un te te te te te er _in Rat un un
te
By rv Ra Ra Co Co By Ra Ra Ra Ra rn Ra
c_ se v_ v_ Co Co t_ r_ r_ r_ t_ ea
ed st
_ Co C o or _
Sr Sr Sr v_ v_ v_ ro ro ro or gg v_ t_
ff_ e_ _ Sr _Sr Sr Ds er er er _P m
_L Lo _Ho Sr os er
r
i _ S S S c f H S
D
Sa
m
st_
Di
ff
_S
e
am _Ho
st
s t
Ho
st_
t_
Sr
v_
m
e_
Sr Nu
rv
_D
if
D
st_ Srv
_
Logistic regression formula can be derived by
Ho st D st_ os Sa _S
st_ Ho D _H t_ st
D
DSt_ D st
t_
H os
Ds
t_ Ho substituting eq. 2 in eq. 3 to give.
Ds
ATTRIBUTES

Figure 2: NSL-KDD attributes ln( ) = + +⋯ (4)

Information Gain Values of Attributes For IoT Traffic 2020 Dataset

1
It has a value between 0 and 1
0.9 , … . are the regression parameters,
0.8
, … . are the predictor values.
INFORMATION GAIN

0.7

0.6

0.5 B. Stochastic Gradient Descent

0.4 Stochastic gradient descent (SGD) is a machine
0.3
learning optimization approach that is frequently
0.2

0.1
used to calculate the model parameters that best fit
0 the expected and actual outputs. They are a variant of
rt rt ts n /s ts/ s n n ax ax Len Pkt ts/ s yt
s g n n in ean Avg
Sr
c_
Po _ Po _By atio kts
t r P
Ds Win _Du wd_ ow_ IAT d le _IA
ea ea
P k _M _M T _M le_M er_
Id ead _Bw low
d_ _ By d_B ze_ _M _ M
w Si
Av
le
ea
n
ea _M
IA
T
n_
M ize_
S
gradient descent techniques that’s solves the issues of
d _ lo w B Fl _ I w n _B kt_ Id t_Le w_ t_Le eg_
it _ Bw F Flo
w Flo
B wd_ H
TOTL
e F

Su
bflo
w P
Pk Flo
d _ Pk d _S
Bw
computational time. In SGD, the gradient of a
In Bw

ATTRIBUTES randomly selected subset of the observations rather

Figure3: IoTID20 attributes
than all of them is calculated [30].

∇ ( , , )(5)
Information Gain Values for Attributes of IoT Traffic 2018
2
= ℎ , = given data instances,
IN FO R M AT IO N G A IN

1.8
1.6
1.4
1.2
1 = , ∇ =true gradient.
0.8
0.6
0.4
0.2 C. Light Gradient Boosting Machine
0
LGBM is a high-performance gradient boosting
E
0 .1 1 _W D E

H H H H_ G N I T
0 .0 0 .0 1 D E

_ jit 3 _W DE
. 0 1 GH T

_W HT

H_ _ W T
HT

_W T
HT
N

N
I_D L3 _W HT

T
_W HT
I_D .0 1_ IG HT

W T
T

H H M AG EAN
H
I_D _ L1 IG H

0 .1 E IG H
UD
H H _L 0. E IG H
L 1 I GH

. 01 G H
EA

H H MA E A

U
_M E IG

H H jit_L I TU
G

EI G

EI G

system based on the decision tree approach that may

TU
E IG

0 .1 IG

H_ _ M

3 _ N IT

H H H_ L0 N IT
I

1_ _ M
H _ W EI

L0 W E I

M
IR _ W E
E
M IR _ L W E

N
W

_ L 01 _
G
G
_

_
_

MA

A
L5

L5
L3

be used for ranking, classification, and a variety of

.
L0

.
L0
IR _
IR _

H_

L
5_
0

_L
IR

H
H_
IR_

_
_L
I_D
I_D

H
_L

other machine learning tasks. LGBM splits the tree

_L
I_D
M
M
M

M
M

ATTRIBUTES leaf wise based on the best fit. Thus, when growing
on the same leaf in LGBM, the leaf-wise approach
Figure 4: IoTID18 attributes
can reduce loss significantly more than other existing
boosting techniques.A diagrammatic explanation is
given in the figure below.

2756
Hassan Adegbola Afolabi et al., International Journal of Advanced Trends in Computer Science and Engineering, 10(4), July – August 2021, 2752 – 2761

This is the harmonic mean between recall and

precision. It clearly shows how efficient an algorithm
is. F-Score ranges between [0,1] and tries to find the
balance between precision and recall.
= 2∗ (10)
Figure 5: Leaf Wise Tree Growth in LGBM

D. Extreme Gradient Boosting Let’s assume TRUE/FALSE to be 1/0, there are 4

The XGBoost algorithm is a regression tree model important terms to note from the above metrics.
classifier [31]. It provides parallel tree boosting (i.e.
GBDT, GBM) that addresses a wide variety of data TP= True positive which simply means that both the
science issues quickly and accurately. prediction and actual output is YES (1)
TN= True negative which simply means that both the
E. Deep Neural Network prediction and actual output is NO (0)
A deep neural network (DNN) is a type of artificial FN = False negative, means that the prediction NO
neural network (ANN) that contains one or more (0) is different from the actual output YES (1).
layers between the input and output layers. A DNN is FP= False Positive, means that the prediction YES
composed of the following basic components like (1) is different from the actual output NO (0)
neurons, synapses, weights, biases, and functions. It
is composed of sequential linear functions and 4. PROPOSED ENSEMBLE APPROACH
nonlinear activation functions and can be
mathematically expressed as below: Based on an extended experiment conducted in the
= ( + ) (6) previous sections, we present a stacking-based
ensemble learning technique for an intrusion
Where y, W, x and b are outputs, weights, inputs and detection system. This model uses LR as a meta
biases respectively. (. )is known as the classifier and combines SGD, LGBM, XGBOOST,
activationfunction and DNN algorithms as base classifiers.Our proposed
approach comprises two main stages namely:
3.4 Performance Metrics Stage 1: involves the training of the base classifiers
We obtained the mean value for the following on each input dataset
metrics during the performance analysis of all Stage 2: Involves the training of the meta-classifier
algorithms evaluated in the project: Accuracy, on the outputs of each individual base classifiers in
Precision, Recall, and F-score Accuracy, Precision, the ensemble.
Recall, and F-score. The framework of our proposed technique is shown
in figure 6 below.
A. Accuracy
In classification problems, accuracy refers to the
amount of accurate predictions divided all possible
predictions. The mean accuracy rate is the average
accuracy rate for each attack class in a given dataset.

= (7)

B. Precision
It is the ratio of accurate positive results to the
number of predicted positive results by the algorithm.

= (8)
C. Recall
It is calculated by dividing the number of accurate Figure 6:Framework of the Proposed Stacking Ensemble
positive results by the total number of relevant
samples. An advantage of using this approach is that meta-
= (9) classifier in the second stage can rectify the
shortcomings of any or all of the base classifiers in
the first stage. Since the objective is to obtain
D. F-score significantly better outcomes, our ensemble

2757
Hassan Adegbola Afolabi et al., International Journal of Advanced Trends in Computer Science and Engineering, 10(4), July – August 2021, 2752 – 2761

technique must outperform the results of the best

base classifier in the overall model.

Figure 7(b):Outcomes of Stochastic Gradient Descent on

the Experimental Datasets

Algorithm 1: Stacking Ensemble Pseudo Code

5. RESULTS AND ANALYSIS

Figure 7(c):Outcomes of Deep Neural Network on the
In this work, we compared the classification Experimental Datasets
performance of a few machine learning techniques
namely Logistic Regression, Stochastic Gradient
Descent, Light Gradient Boosting Machine, and
Extreme Gradient Boosting. Accuracy, precision,
recall, and F-score are the metrics used in this work.
The experiment was carried out on Google
Colaboratory using Python 3, TensorFlow, and a
Graphics Processing Unit (GPU) installed on a Mac
OS X 2.8 GHz Intel Core i7 CPU, 16.00 GB RAM
2133MHz LPDDR3.The experiments are in phases.
In the first phase: Information gain feature selection
was applied to all the datasets considered for this Figure 7(d):Outcomes of Light Gradient Boosting
work. 20 best attributes were extracted and the results Machine on the Experimental Datasets
were shown in figure 1-4.
In the second phase. A classification technique is
applied on all datasets involved. Figures: 7(a-e)
shows the overall performance of each technique
relative to the corresponding datasets.

Figure 7(e):Outcomes of Extreme Gradient Boosting on

the Experimental Datasets

From the results above, it shows that LGBM and

XGBoost are the strongest learners in terms of
Figure 7(a):Outcomes of Logistic Regression on the accuracy and F-score. Since They give the best
Experimental Datasets performance when applied to N-BaIoT and IoTID20

2758
Hassan Adegbola Afolabi et al., International Journal of Advanced Trends in Computer Science and Engineering, 10(4), July – August 2021, 2752 – 2761

datasets, we chose to combine them with SGD and observed that LGBM and XGBoost had the best
DNN as base classifiers to construct an ensemble performance having 99.5% and 99.8% accuracy,
stacking classifier and Logistic regression as meta 98.5% and 99.5% precision, 97.7% and 99.0% recall,
classifiers. In order to evaluate the performance of 98.0% and 99.2% F-score respectively. In contrast,
our proposed stacking ensemble method, we compare SGD performed relatively poor in terms of accuracy
all algorithms used in this work in terms of accuracy and precision having 89.1% and 55.8% respectively.
and F-score metrics as figures 8(a)-(b) below: DNN had the least performance in terms of recall
with 49.1% and LR yielded the least F-score of
53.3%.

Table 5

Figure 8(a):Performance comparison across all datasets

based on accuracy The next the experiment was conducted using
CICIDS 2017 dataset. Table 5 above clearly shows
the results obtained. Stochastic Gradient descent
yielded the lowest performance with 97.3%,11.2%,
15.1% and 12% for accuracy, recall, precision and F-
score respectively. Whereas, LGBM outperformed all
the other algorithms, with accuracy, precision, recall
and F-score obtained as 99.9%, 70.8%, 68.8% and
69.4% respectively.
Table 6

Figure 8(b):Performance comparison across all datasets

based on F-score

The chart above shows that the LR algorithm when

used as a meta classifier yields better accuracy and F-
score than LR as a single classifier and amongst the
other algorithms. Table 7
We observed that the performance of these
algorithms depends on the kind of datasets employed
because algorithms like LGBM, XGBoost and DNN
gave better results with N-BaIoT and IoTID20
datasets. Regardless, our proposed method
outperformed all other algorithms as shown in the
tables below.

Table 4

Results obtained from the experiments using NSL-

KDD and N-BaIoT are recorded in tables 6 and 7
above. We observed that SGD had the lowest
performance with both dataset except for Logistic
regression that the lowest recall of 54.1% with N-
BaIoT dataset. Extreme Gradient Boosting algorithm
yielded the highest results with (98.4%, 39.3%,
The table 4 above shows the recorded values of 37.2%, 37.4%) and (98.4%, 90.6%, 86.6%, 87.9%)
accuracy, precision, recall and F-score when all for accuracy, recall, precision and F-score
algorithms are tested with IoTID20 dataset. It was respectively.

2759
Hassan Adegbola Afolabi et al., International Journal of Advanced Trends in Computer Science and Engineering, 10(4), July – August 2021, 2752 – 2761

Table: 8 for misuse and anomaly detection, Comput.

Netw. 136 (2018) 37–50,
https://doi.org/10.1016/j.comnet.2018.02.028.
[2]. Z. Inayat, A. Gani, N.B. Anuar, M.K. Khan, S.
Anwar, Intrusion response systems:
foundations, design, and challenges, J. Netw.
Comput. Appl. 62 (2016),53–74,
https://doi.org/10.1016/j.jnca.2015.12.006.
[3]. A.S. Ashoor, S. Gore, Difference between
Considering our proposed stacking ensemble model, intrusion detection system (IDS) and intrusion
when it was evaluated using IoTID20 dataset, it prevention system (IPS), Commun. Comput.
achieved the highest F-score of 99.3% and accuracy, Inf. Sci. (2011) 497–501,
recall and precision similar to XGboost Thus, https://doi.org/10.1007/978-3-642-22540-6_48.
showing that our model was able to achieve the best [4]. I. Sharafaldin, A.H. Lashkari, A.A. Ghorbani,
results similar to the best algorithms for IoTID20 toward generating a new intrusion detection
dataset as shown in Table 8. However, the proposed dataset and intrusion traffic characterization, in:
stacking ensemble model performed better than other ICISSP 2018 - Proc. 4th Int. Conf. Inf. Syst.
algorithms considered for analysis in this work when Secur. Priv., 2018, pp. 108–116,
evaluated with N-BaIoT dataset. Thus, it shows that https://doi.org/10.5220/ 0006639801080116.
the new technique can be used to deliver better [5]. M. Ahmed, A. N. Mahmood and J. Hu, "A
results for Intrusion detection system in an IoT survey of network anomaly detection
platform. techniques," Journal of Network and Computer
Applications, vol. 60, pp. 19-31, 2016.
6. CONCLUSION AND FUTURE WORKS
[6]. H. Hota, A.K. Shrivas, Decision Tree
Techniques Applied on Nsl-kdd Data and Its
While several machine learning approaches have
Comparison with Various Feature Selection
been presented to improve the effectiveness of IDSs,
Techniques, in: Advanced Computing,
Identification of relevant features in a dataset that has
Networking and Informatics-Volume 1,
a substantial impact on IDS performance is a major
Springer, 2014, pp. 205–211, doi:10.1007/978-
challenge. Hence, with better feature selection, an
3-319-07353-8_24.
efficient IDS can be designed. This work investigated
[7]. C. Khammassi, S. Krichen, A ga-lr wrapper
the efficiency of the following techniques: LR, SGD,
approach for feature selection in network
LGBM, XGBoost, and DNN. Evaluation was done
intrusion detection, Comput. Secur. 70 (2017)
using four datasets namely: CICIDS2017, IoTID20,
255–277, doi:10.1016/j.cose.2017.06.005.
NSL-KDD and N-BaIoT’18 datasets. 20 most
[8]. J. McHugh, "Testing Intrusion detection
relevant features from each dataset were extracted
systems: a critique of the 1998 and 1999
using the information gain feature extraction
DARPA intrusion detection system
approach, and the techniques were compared based
evaluations as performed by Lincoln
on four metrics: accuracy, precision, recall, and f-
Laboratory," in ACM Transactions on
score. According to the experimental results, feature
Information and System Security, 2000.
selection can enhance detection accuracy. Also, we
[9]. M. Mahoney and P. Chan, "An analysis of the
presented an ensemble learning strategy based on the
1999 DARPA / Lincoln laboratory evaluation
stacking of the analyzed algorithms. The experiment
data for network anomaly detection," in
demonstrated that our ensemble technique
Recent Advances in Intrusion Detection, 6th
outperforms other single classifiers across all datasets
International Symposium, RAID 2003,
examined especially IoT traffic datasets.This
Pittsburgh, PA, USA, 2003.
demonstrates that our work has proven to be fairly
[10]. J. Zhang, M. Zulkernine and A. Haque,
significant in getting a better understanding of how to
"Random-forests-based network intrusion
develop security solutions for IoT and our technique
detection systems," IEEE Transactions on
can be used for practical application of IDSs.Further
Systems, Man, and Cybernetics, Part C
research into more diverse base learners and
(Applications and Reviews), vol. 38, no. 5, pp.
alternative combination methods to improve these
649 - 659, 2008.
outcomes, comparing our technique with existing
[11]. U. Ravale, N. Marathe and P. Padiya, "Feature
state of the art techniques is envisioned.
selection-based hybrid anomaly intrusion
REFERENCES detection system using K means and RBF
kernel function," in Proceeding of
[1]. V. Hajisalem, S. Babaie, A hybrid intrusion International Conference on Advanced
detection system based on ABC-AFS algorithm Computing, 2015.

2760
Hassan Adegbola Afolabi et al., International Journal of Advanced Trends in Computer Science and Engineering, 10(4), July – August 2021, 2752 – 2761

[12]. M. C. Belavagi and B. Muniyal, "Performance [22]. M. Tavallaee, E. Bagheri, W. Lu, A.A.
Evaluation of Supervised Machine Learning Ghorbani, A detailed analysis of the KDD
Algorithms for Intrusion Detection," Procedia CUP 99 data set, in: IEEE Symp. Comput.
Computer Science, vol. 89, pp. 117-123, 2016. Intell. Secur. Def. Appl. CISDA, 2009, p.
[13]. M.Almseidin, M. Alzubi, S. Kovacs and M. 2009,
Alkasassbeh, "Evaluation of Machine https://doi.org/10.1109/CISDA.2009.5356528
Learning Algorithms for Intrusion Detection [23]. S. Revathi and A. Malathi, "A Detailed
System," in 15th International Symposium on Analysis on NSL-KDD Dataset Using Various
Intelligent Systems and Informatics, 2017. Machine Learning Techniques for Intrusion
[14]. M. Zaman and C. H. Lung, "Evaluation of Detection," International Journal of
Machine Learning Techniques for Network Engineering Research & Technology, vol. 2,
Intrusion Detection," in IEEE/IFIP Network no. 12, 2013.
Operations and Management Symposium, [24]. Y. Meidan, M. Bohadana, Y. Mathov, Y.
2018. Mirsky, D. Breitenbacher, A. Shabtai, and Y.
[15]. S. Aljawarneh, M. Aldwairi and M. B. Elovici "N-BaIoT: Network-based Detection
Yassein, "Anomaly-based intrusion detection of IoT Botnet Attacks Using Deep
system through feature selection analysis and Autoencoders", IEEE Pervasive Computing,
building hybrid efficient model," Journal of Special Issue - Securing the IoT (July/Sep
Computational Science, vol. 25, pp. 152-160, 2018).
2018. [25]. Y. Mirsky, T. Doitshman, Y. Elovici& A.
[16]. M.A. Ferrag, L. Maglaras, S. Moschoyiannis, Shabtai 2018, "Kitsune: An Ensemble of
H. Janicke, Deep learning for cyber security Autoencoders for Online Network Intrusion
intrusion detection: approaches, datasets, and Detection", in Network and Distributed
comparative study, J. Inf. Secur. Appl. (2020) System Security (NDSS) Symposium, San
50, https://doi.org/10.1016/j.jisa.2019.102419. Diego, CA, USA.
[17]. H.A. Afolabi, A.A. Aburas, “An Evaluation of [26]. Ullah I., Mahmoud Q.H. (2020) A Scheme for
Machine Learning Classifiers for Prediction of Generating a Dataset for Anomalous Activity
Attacks to Secure Green IoT Detection in IoT Networks. In: Goutte C., Zhu
Infrastructure”International Journal of X. (eds) Advances in Artificial Intelligence.
Emerging Trends in Engineering Research, Canadian AI 2020. Lecture Notes in Computer
9(5), May 2021,549–557, Science, vol 12109. Springer, Cham.
https://doi.org/10.30534/ijeter/2021/03952021 https://doi.org/10.1007/978-3-030-47358-
. 7_52
[18]. H.A. Afolabi, A.A. Aburas, “Proposed Back [27]. Hyunjae Kang, Dong Hyun Ahn, Gyung Min
Propagation Deep Neural Network for Lee, Jeong Do Yoo, Kyung Ho Park, Huy
Intrusion Detection in Internet of Things Fog Kang Kim, September 27, 2019, "IoT network
Computing” International Journal of intrusion dataset", IEEE Dataport, doi:
Emerging Trends in Engineering Research, https://dx.doi.org/10.21227/q70p-q449.
9(4), April 2021, 464 – [28]. Sheena, Krishan Kumar, Gulshan Kumar:
469,https://doi.org/10.30534/ijeter/2021/2394 Analysis of Feature selection Techniques: A
2021. Data Mining Approach, International Journal
[19]. S. R. Khonde, V. Ulagamuthalvi, “Hybrid of Computer Applictions, ICAET 2016,
Framework for Intrusion Detection System IJCA2016 (1):17–21.
using Ensemble Approach” International [29]. Zahra Karimi, Mohammad Mansour and Ali
Journal of Advanced Trends in Computer Harpunabadi: Feature Ranking in Intrusion
Science and Engineering, 9(4), July – August Detection Dataset using Combination of
2020, 4881 – Filtering Methods, IJCA, Vol 78, No 4,
4890,https://doi.org/10.30534/ijatcse/2020/99 September 2013.
942020 [30]. Ruder, S. (2016). An overview of gradient
[20]. Intrusion Detection Evaluation Dataset descent optimization algorithms. arXiv
(CICIDS2017),"[Online]. Available: preprint arXiv:1609.04747.
https://www.unb.ca/cic/datasets/ids- [31]. Chen, T., &Guestrin, C. (2016, August).
2017.html. [Accessed 08 04 2019]. Xgboost: A scalable tree boosting system. In
[21]. NSL-KDD dataset," [Online]. Available: Proceedings of the 22nd acmsigkdd
https://www.unb.ca/cic/datasets/nsl.html. international conference on knowledge
[Accessed 08 04 2019]. discovery and data mining (pp. 785-794)

2761