See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/341125974

Intrusion Detection in Cyber Security: Role of Machine Learning and Data

Mining in Cyber Security

Article · January 2020

DOI: 10.25046/aj050310

CITATIONS READS
23 1,064

4 authors, including:

Gillala Rekha Shaveta Malik

K L University 31 PUBLICATIONS 215 CITATIONS
43 PUBLICATIONS 296 CITATIONS
SEE PROFILE
SEE PROFILE

Amit Tyagi
Pondicherry Engineering College
244 PUBLICATIONS 1,942 CITATIONS

SEE PROFILE

All content following this page was uploaded by Amit Tyagi on 12 April 2022.

The user has requested enhancement of the downloaded file.

 Advances in Science, Technology and Engineering Systems Journal Vol. 5, No. 3, 72-81 (2020)
ASTESJ
www.astesj.com
ISSN: 2415-6698

Intrusion Detection in Cyber Security: Role of Machine Learning and Data Mining in Cyber Security
Gillala Rekha1, Shaveta Malik2, Amit Kumar Tyagi3,*, Meghna Manoj Nair3
1
Koneru Lakshmaiah Education Foundation, Department of Computer Science and Engineering, Hyderabad, India – 522502
2
Terna Engineering College, Department of CSE, Navi Mumbai, Maharashtra, India.
3
Vellore Institute of Technology, School of Computer Science and Engineering, Chennai Campus, Chennai, 600127, Tamilnadu, India.

ARTICLE INFO ABSTRACT

Article history: In recent years, cyber security has been received interest from several research communities
Received: 10 August, 2019 with respect to Intrusion Detection System (IDS). Cyber security is “a fast-growing field
Accepted: 04 March, 2020 demanding a great deal of attention because of remarkable progresses in social networks,
Online: 03 May, 2020 cloud and web technologies, online banking, mobile environment, smart grid, etc.” An IDS
is a software that monitors a single or a network of computers from malicious activities
Keywords:
(attacks). Detecting an intrusion or prevention (due to increase the usage of internet), is
Cyber Security
becoming a critical issue. In past, several techniques have been proposed to overcome or
Intrusion Detection System
detect intrusion in a network. But most of the techniques (used now days in detecting IDS)
Machine Learning
are not able to overcome this problem (in efficient manner).Together this, Machine Learning
Data Mining
(ML) also has been adopted in various applications (due to providing good accuracy results
(in respective domain)). Hence, this work discusses “How machine learning anddata mining
can be used to detect IDS in a network” in near future.ML use efficient methods like
classification, regression, etc., with efficient results like high detection rates, lower false
alarm rates and less communication costs. This work also provides a detail comparison with
metrics in table 1-3 (with their performance/ algorithms/ dataset or metrics used).

1. Introduction (though they become hard to use in the case of naïve outbreaks).
It is to be pointed out that this method can’t be used for the
Cyber security involves the practice of preventing the exposure of identification of novel (or zero day) catastrophes. The second
computers, programs, etc. from attacks, unauthorized usage, classification (i.e., based on anomaly) replicates the behavioristic
modifications, destructions, etc. It’s a common practice to find approach by developing an activity profile, hence differentiating
every Cyber Security system to have a firewall, antivirus the ambiguity from the normal attitude. This method can be used
techniques and Intrusion Detection System (IDS). IDS are a for the detection of novel-attacks and hence are deeply
crucial component as they help in spotting any undesirable and encouraged. Furthermore, it customizes the normal activity
unwanted changes in the system [1]. Intruders are mainly routine for every instance, ensuring that the intruders are unable
categorized as External Intrusions/Intruders (i.e., attack by the to comprehend which of the activities can be performed incognito.
people who don’t belong to the organization) and Internal But just like how every coin has two sides, this technique too has
Intrusions/Intruders (i.e., attack by the people from within the its own disadvantage – it is likely for False Alarm Rates (FARs).
same establishment). However, cyber analytics can be separated The last categorization involves the combination of the first two
on the following bases: i) on the basis of misuse or signatures ii) methods – misuse and anomaly detection. They are mainly
on the basis of anomalous encryptions iii) on the basis of hybrid implemented to raise the rate of detection of common attacks and
nature. reduce the False Positive (FP) rate for the minor attacks. IDS’s
The first form of classification is created to represent attacks can also be divided based on network or host. An IDS which
following an ordered pattern to spot and prevent a similar attack depends on the network identifies attacks by keeping an eye on
in the further years along with the detection of famous attacks the traffic through the network devices. A host-based IDS screens
*
Corresponding Author: Amit Kumar Tyagi, [email protected] all processes and file activities related to the software with a host.
www.astesj.com 72
https://dx.doi.org/10.25046/aj050310
 G. Rekha et al. / Advances in Science, Technology and Engineering Systems Journal Vol. 5, No. 3, 72-81 (2020)
a) Host-based IDS (HIDS): It mainly focuses on analyzing the Destination Address (IP address of a destination) f) ICMP type
internal functioning of a computing system. It might detect (like echo requests, null, etc.) g) ICMP Code h) Raw Data Length
activities like which program is trying to access which (length of the data packets) i) Raw Data.
particular resource and are there any attempts on illegitimate
access. For example, a word processor which spontaneously The neural network model was trained using a back-propagation
alters the system password database. algorithm for 10,000 iterations of the selected training data. Out
b) Network-based IDS (NIDS): It focuses on analyzing and of 9.462 records, 1000 were randomly selected for testing and the
filtering the traffic among network device. It’s commonly remaining was used to train the system. The neural network model
found that intrusions occur as ambiguous patters. These are required 26.13 hours to complete. The results reveal that on
mainly caused by the attacks launched by external intruders training data the root mean square error is 0.058298 and on Test
who wish to access the network to gamble the network and data root mean square error is 0.069929. Finally, an accuracy of
destroy it.
93% can be considered based on RMS, where each data packet
Hence, the article is organized into a number of sections. Section was classified as either a normal or an attack set.
2 discusses several classifications (like signature and anomaly)
with respect to cyber security. Further, section 3 discusses several In [6], the authors proposed Online Analytical Processing (OLAP)
cyber data sets available for making a comparison and later on the Mining and Classification based IDS, (OMC-IDS). OMC_IDS
significance of machine and data mining in detection of intrusion handle any intrusion detection data using historical data analysis
detection in cyber security/ applications (in near future) has been from heterogeneous sources and summarization them by filtering
discussed in section 4. Further Section 5 discusses “how machine the data by removing the irrelevant data. Apart, a data cube is
learning and artificial intelligence can be more useful for cyber constructed and integrate OLAP techniques. They applied
security professionals for detecting vulnerabilities or preventing association rule mining to extract the interesting patterns and
attacks”. Finally, this work is concluded with some future classify each connection as normal or any attack. They proposed
enhancements in brief in section 6. association rules to find the correlation between TCP/IP
parameters and the types of attack on DARPA 1998 data set. They
2. Cyber Security’s Classifications generated rules and less constraint is retained. After the rules are
generated, a C4.5 classifier is applied for new connection records.
The three types of Intrusion detection in support of cyber security The experiments were carried out on DARPA19985 dataset. The
are [2]-[4]: Misuse-based or Signature based, Anomaly-based, training data and test data are generated in the first seven weeks
and Hybrid. Here, each one can be discussed in detail as: and in the next two weeks respectively. The results show that total
2.1. Misuse-Based or Signature Based of detection rates as 99%, 97%, 86% and 74%, respectively. The
main drawback of association rule mining is that the generated
There are multiple ways to replicate an attack. The attack can be rules may express correlation, but the approach is promising for
a pattern, or a signature used to identify the deviation. They are attack signature building.
bound to detect a majority or most of the common attack
techniques. However, they come to be of little use in the case of Further in [7], authors proposed an algorithm to use the existing
minor or unidentified attack patterns. These systems try to spot signature data and find the signature of the related attack in less
and differentiate on the principle of “bad” behavior. The prime time. They compared their approach with algorithm based on
obstacle to overcome is on how to create a signature that combines Apriori called Signature Apriori (SA) and found that it takes less
all the varieties of a consistent attack. A plethora of Machine processing time. Such algorithms can be used to generate new
Learning methodologies have been put into use for the detection signatures, i.e., used into misuse detection systems such as Snort.
of misuse in these systems. These detections prove to be useful to The proposed method finds newly attack signature based on the
identify the outbreaks on networks by associating the routine known signature. Scan Reduction method is also used for the
activities with that of the expected actions of an intruder. reduction of time consumed for scanning of databases. This
method involves the determination of a new attacking signature in
In [5], the author proposed a framework to identify and classify an efficient way when compared to the Signature Apriori
network activities based on Artificial Neural Network (ANN). algorithm. Authors have implemented the data mining approach
The data sources are based on various formats, i.e., limited, to complement the signature discovery in IDS based on network
incomplete, and nonlinear in nature. They implemented data [8]. This not only generates signatures for the detection of misuses
detection that utilizes the analytical strengths of neural networks. dependent on transfer protocols, but also for those based on
A multi-layer classification prototype using MLP is used to detect content of traffic. The Signature Apriori (SA) is based on the
the misuse by developing the architecture containing four fully typical association rules algorithm – Apriori algorithm [8]. The
connected layers. The neural architecture consists of 9 nodes as experiments have two parts to it: a) Speed testing of SA algorithm
input and 2 output nodes. The data pre-processing were conducted b) Accuracy testing of the signatures being mined. This evolves
at three different levels includes a) Protocol Identifier (PID) – the 70% support and the time consumed is extremely less (one is less
rules and regulations pertaining to an event (TCP = 0, UDP = 1, than 50111s the other is 330 ms). On the whole, the techniques
ICMP = 2, and Unknown = 3) b) Source Port c) Destination d) which are applied to tackle the cyber-attacks have been active
Source Address ( IP address corresponding to a source) e)
www.astesj.com 73
 G. Rekha et al. / Advances in Science, Technology and Engineering Systems Journal Vol. 5, No. 3, 72-81 (2020)
predominantly as they emphasize on screening the traffic in the experiments are carried out using KDD data sets using different
network, identification of anomalies and traffic sequences of attack percentages (0%, 1%, 5%, 10%, 25%, 50%, and 80%) and
cyber-attack. Apart from this, the misuse detection can be the author reported impressive results without prior knowledge of
enforced for the detection of these outbreaks prior to them actually any attacks in the KDD datasets. Further, Kruegel at el. [12]
being a part of the attack. Some authors have spotted the installed an intrusion detection signature using clustering
command and control traffic (C2C) in Internet Relay Chat using algorithms to derive decision tree for intrusion detection. It was a
the technique of machine learning to adhere to the botnet placement with Snort. With the help of a decision tree, we are able
existence, for which TCP level data sets have been put into use. to choose the features which highly distinguish the characteristics
Wireless traffic sniffers were used extensively to gather complete of the rule set, permitting parallel evaluation for every unique
TCP/IP headers from around 18 locations around the campus. feature. It provides a better performance with respect to Snort. In
This was divided into two major stages: (i) The initial stage [12], the author make use of the tcpdump files as the necessary
involved the distinction between IRC and non-IRC traffic, (ii) dataset for the ten days of test data when considering the
after which, there was a distinction between botnet and real IRC evaluation of 1999 DARPA intrusion detection. On comparing
traffic. For the initial stage, the comparison of performance is and contrasting the rate of processing of Snort and the decision
done between J48, naive Bayes, and Bayesian network classifiers tree for the above data, it was observed that real performance gain
to identify IRC and non-IRC traffic damages by attaining an vary drastically depending on the basis of the comprehended
excellent overall classification accuracy. Only the naïve Bayes traffic. 103% was found to be the maximum speed, while 5%
classifiers were capable of achieving reduced false negative rate. turned out to be the minimum. The decision trees performed better
as they result in an average speed of 40.3%. The second task was
The naive Bayes classifiers accurately classified 35 out of the 38 also conducted with increased number of protocols right from 150
botnet IRC (which flows correctly and achievesFalse Negative up to 1581. The results proved that the approach of the decision
Rate (FNR) of 7.89%) [9]. In Stage (ii), by applying classification tree works efficiently, especially with respect to large rule sets.
they accurately labelling IRC traffic as botnet and non-botnet were This approach notifies that the clustering action based on decision
more challenging. In [10], author proposed an adaptive intrusion trees will definitely reduce the operating time, thus enhancing the
detection system which is considered as a framework for detecting processing speed. Furthermore, it portrays a generic solution to
intrusion detection using Naïve Bayesian network. The DARPA many of the other IDSs like host and network-based, and firewall
KDD99 dataset with 38 attacks are used to find the new intrusion and packet filters.
signature like DoS, r21,u2r and probe.The dataset consists of 9
features in the inference network such as Protocol type, Service, Zhang et al. [13] study proposed a complete intrusion detection
Land, Wrong fragment, Numerous failed login, Logged in, Root framework containing a detector used for signature-based attack
shell, Is guest login. In the first stage, a junction tree inference prediction and a database to identify outlier. All the anomaly
technique is used to identify the normal or attack data with patterns identified by the system or user either manually or
performance detection rate 87.68% on normal and 88.64% on automatically are stored in the database. Because of the extremely
intrusion. In the second stage, the dataset is classified into 4 quick nature of its implementation, it’s often used as an online
classes: DoS, Probing, R2L and U2R.The performance determine solution. Gharibian et al. [14] has put forth a comparative study
a detection rate of 88.64% for DoS, 99.15% for Probing, 20.88% with the help of probabilistic and futuristic ML methods and
for R2L, 6.66% for U2R and 66.51% for other classes. processes for detection of intruders and their malicious acts
namely, Naïve Bayes and Gaussian along with those of Decision
In [11], authors used reliable signatures generated based on Tree and Random Forests. A lot of the training data sets which
supervised clustering algorithm and updating them in real-time have been constructed from KDD99 are being deployed for
using unsupervised clustering technique. The signature updating effective functioning today and each of the methods have been
is done to change attack methods while retaining the signatures used for categories of attack like DoS, Probe, R2L and U2R with
useful information. They used a simple density-based clustering a proper analytical study of their results. Normalization used in
algorithm, called Simple Logfile Clustering Tool (SLCT) to create the formation of these datasets, complementing the argument that
clusters of regular and anomaly traffic. The study made use of a the features in KDD are not similar to those of the others and they
new user stricture, M, in SLCT which mentions the percentage of possess high variance scales. The executional capability of
fixed attributes to be spotted out of all the attributes that a Decision Trees (DT) and Random Forests (RF) portray valid
potential cluster is expected to have. If the value of M equates to results and operations in the identification of DoS. On the contrary,
0, it then allows the formation of clusters irrespective of the Gaussian and Naïve Bayes results shows much better in few of
number of fixed attributes. By equating the value of M to greater the varied attack domains like Probe, R2L and U2R. Based on the
values they recapitulate the intruder ones, thus classifying the results, the author stated that the probabilistic techniques are more
original data. This is inferred to with the help of parameter M as robustness in nature than predictive techniques for intrusion
SLCT attack. Both the clustering techniques are implemented for detection.
the detection of normal or attack traffic and for identification of
the usual traffic in a supervised manner accordingly. In [11], the Mukkamala et al. [15] considered the performance of ANN, SVM
author treated anomalous centroid of cluster as a signature. The and Multivariate Adaptive Regression Splines (MARS) and
www.astesj.com 74
 G. Rekha et al. / Advances in Science, Technology and Engineering Systems Journal Vol. 5, No. 3, 72-81 (2020)
proved that ensembles of ANNs, SVMs and MARS is of top 2.2. Anomaly and Hybrid Detection
priority for individualized perspectives for the detection of these Lippmann et al. [24] proposed an IDS system on transcripts of
attacks with respect to precision of division. The five class telnet sessions. The combination of training data and new
classification experiments were performed on 11,982 records. keywords were used to find the common attacks using neural
They applied 3 classification algorithms like SVMs, MARS and network model. The system achieves 80% of high detection rate.
ANNs. The ensemble of SVMs, MARS and ANNs approach out Palagiri et al. [25] proposed a model for learning the normal traffic
performs with accuracies of 99.71% for Normal, 99.85% for patterns from TCP/IP port. They applied preprocessing
Probe or Scan, 99.97% for DoS, 76% for U2R, and 100% for R2L techniques then perform clustering on normal traffic and final
are reported respectively. The accuracy of four classes are 99% trained using Artificial Neural Network (ANN). The study
using SVM, RP, SCG, OSS algorithms and the accuracy on the reported a 100% normal behavior.
U2R class is much less with 76%. In this paper [16] the author
used genetic algorithms to generate simple rules for network Apiletti et al. [26] proposed NETMINE framework which
traffic. classifies the traffic data using data mining/ machine learning
techniques. The framework performs data stream processing,
These rules are used to differentiate normal network connections refinement analysis by using general association rule extraction
from anomalous connections and these anomalous connections for profile data, anomaly detection, and identifying recurrent
refer to events with probability of intrusions. Abraham et al. [17] patterns.
applied genetic programming algorithms such as Linear Genetic
Programming (LGP), Multi Expression Programming (MEP) and Intrusion Detection Systems (IDS) mainly intend towards
Gene Expression Programming (GEP) in attack classification. In protection of computerized systems and helps in spotting
[18], Hansen et al. used GP with homologous crossover for vulnerabilities and other attack exposures. A novice structural
performing intrusion detection. Arnes et al. [19] proposed a novel outline which has its’ roots based on data mining methods have
approach to network risk assessment. The approach considers the been put forth [27] for the creation of an IDS. This framework
risk level of a network as the composition of the risks of individual proposes Association Based Classification (BC) which is
hosts. It is probabilistic and uses Hidden Markov models (HMMs) dependent on rules linked to fuzzy logic for the development of
to represent the likelihood of transitions between security states. classifiers and this helps in categorization of normal and un-
They tightly integrate the risk assessment tool with an existing normal records. Compatibility threshold is the central parameter
framework for distributed, large scale intrusion detection, and in this application. The approximate value for this depends on the
apply the results of the risk analysis to prioritize the alerts ROC curve of the system which is produced by carrying out lots
generated by the intrusion detection sensors. of tests on datasets, with varied threshold values. Therefore, 0.06
becomes the compatibility threshold which is to be dealt with in
An HMM is denoted by (P, Q, Π). Lee et al. [20] developed a the detection of anomalous behavior. The FP error produced can
systematic framework using data mining techniques for be reduced to the level of that of misuse detection situation and
automated IDS.In [21] the author trained Naïve Bayes classifier there’s a huge decrease in the detection rate of existing attacks. In
on KDD 1999. The data is partitioned into training set and test set the case of unforeseen intrusions, the ambiguous case outshines
and the data was grouped into four attacks (1. probe or scan, 2. the misuse perspective, and this is the key advantage of anomaly-
DoS, 3.U2R, and 4. R2L). The author stated an accuracy of 96%, based approaches.
99%, 90% and 90% for the respective attacks. Hu et al. [22]
proposed a framework for malicious transactions. An cyber-attack Luo et al. [28] has combined the association rule along with the
detection model is needed as a prerequisite for fast damage frequency episodes with that of fuzzy logic to determine the
recovery. The framework employed a sequential mining sequence in the data. This produces short and flexible variations
algorithm for finding the dependencies in database and presented for intrusion detection as a lot of quantifying features come into
as classification rules. The data captured from database logs play. To ensure that data instances don’t outshine the contribution
including (Tname) transaction name, (TID) transaction ID, begin of that of the others, normalization is carried out before retrieving
and end time, etc. They applied the framework for identifying the fuzzy association rules. The required simulations have been
U2R attacks as part of cyber security. The result presented 91% conducted by customized programs and the results have proved
of TP (True Positive) rate and 29% of FP (False Positive) rate. the necessity of fuzzy rules and its’ frequency occurrences in
intrusion detection. Kruege et al. [29] implemented an intrusion
In [23], the author presented an IDS model with high accuracy detection system for identifying attacks against Operating System
and efficiency using machine learning algorithms including K- (OS), they analyzed OS calls to detect attacks against daemon
means, Support Vector Machine (SVM). They also employed applications and set uid programs. Also implemented on machines
feature reduction methods to eliminate the unwanted features. running with Linux or Solaris with individual system calls. A
Table 1 shows the algorithm, data set, metric used for misuse- feature vector is represented which captures information specific
based intrusion detection. to each system call such as the system call number, its return code,
and its arguments. They applied Bayesian network to classify
events during open and executive OS calls.

www.astesj.com 75
 G. Rekha et al. / Advances in Science, Technology and Engineering Systems Journal Vol. 5, No. 3, 72-81 (2020)
Table 1: The algorithm, data set, metric used for misuse-based intrusion detection.

Paper
Algorithm Used Data Set Used Metric Used
Citation

RealSecure network monitor (Internet

[5] Artificial Neural Network Accuracy
Security Systems)

OMC-IDS (OLAP and Association rule

[6] DARPA 1998 Accuracy
mining)

[7] Signature Apriori (SA) Signature based data Accuracy

[8] Apriori algorithm SigSniffer architecture Accuracy

Dartmouth's wireless campus network

[9] J48, Naïve Bayes and Bayesian network Accuracy
(TCP level)

[10] Bayesian network DARPA KDD Accuracy

Density-based clustering algorithm

[11] KDD Accuracy
(SLCT)

[12] Decision Tree DARPA Accuracy

[13] Random Forest KDD Accuracy

[14] Random Forest (Predictive techniques) KDD Accuracy

[15] ANN, SVM and MARS DARPA Accuracy

[16] Genetic algorithms DARPA Accuracy

[17] Genetic algorithms DARPA Accuracy

[18] Genetic algorithms KDD Accuracy

[19] Hidden Markov Network KDD Accuracy

[20] RIPPER DARPA Accuracy

[21] Naïve Bayes KDD Accuracy

Sequence patterns of log files from

[22] Apriori algorithm database are examined to find database Performance
intrusions.

[23] Ant Colony Optimization (ACO) KDD Accuracy

Table 2: The algorithm, data set, metric used for anomaly and hybrid-based intrusion detection.

Paper Citation Algorithm Used Data Set Used Metric Used

Accuracy and False

[24] Artificial Neural Network (ANN) Transcripts of telnet sessions
alarm
[25] Artificial Neural Network DARPA -------
Network capture tools are used to
capture the network traffic
[26] NETMINE framework Support
packets and it was developed at
Politecnico di Torino
www.astesj.com 76
 G. Rekha et al. / Advances in Science, Technology and Engineering Systems Journal Vol. 5, No. 3, 72-81 (2020)
Fuzzy Association Based Classification
[27] KDD Accuracy and FP rate
(ABC)

[28] Fuzzy Logic Tcpdump Accuracy

Accuracy and False

[29] Bayesian network DARPA
Alarm Rate (FAR)

[30] Naïve Bayes algorithm DARPA ------

User command level (shell Accuracy and False

[31] sequence matching algorithms
commands) Alarm Rate (FAR)

EXPOSURE (C4.5 Decision Tree Accuracy and False

[32] DSN
algorithm) Alarm Rate (FAR)

EXPOSURE (C4.5 Decision Tree Accuracy and False

[33] Real-World Network
algorithm) Alarm Rate (FAR)

Accuracy and False

[34] Genetic algorithms KDD
Alarm Rate (FAR)

ROC (Receiver's
Operating Curve) and
[35] Genetic Programming DARPA
False Alarm Rate
(FAR)

(False Positive) FP rate

[36] Hidden Markov Network KDD and (False Negative)
FN rate

[37] RIPPER DARPA (False Alarm Rate) FAR

Accuracy and False

[38] Bayesian network KDD
Alarm Rate (FAR)

[39] Apriori algorithm DARPA Support

Accuracy and False

[40] Robust Support Vector Machines DARPA
Alarm Rate (FAR)

Accuracy and False

[41] Support Vector Machine NetFlow data (Flame tool)
Positive (FP) rate

Self-Organizing Feature Map (SOFM), Accuracy, (False

Genetic Algorithms (GA), and Positive) FP rate and
[42] DARPA 1999
(False Negative) FN
Support Vector Machine (SVM) rate

The DARPA 1999 data set is used to excite the OS kernel by real users. The study stated a detection rate as high as 80.3% and
TCP/IP packets. These features are fed to Bayesian network a false positive rate as low as 15.3%. Table 2 shows the algorithm,
model and if the output is close to zero it indicates normal or data set, metric used for anomaly and hybrid-based intrusion
anomaly state. detection. Now, next section will discuss availability of cyber
security dataset (in current) globally.
In [30], the author proposed alert correlation method based on
naïve bayes algorithm. 2000 DARPA dataset with their intrusion Bilge et al. [32] introduced EXPOSURE, a system that employs
objective are used to train Bayesian network. In [31], the author large-scale, passive DNS analysis techniques to detect domains
proposed a model for differentiating masquerader’s users from that are involved in malicious activity. Bilge et al. [33] presented
www.astesj.com 77
 G. Rekha et al. / Advances in Science, Technology and Engineering Systems Journal Vol. 5, No. 3, 72-81 (2020)
DISCLOSURE, a large-scale, wide-area botnet detection system b. KDD 1999 cup datasets: The most popular and widely used
that incorporates a combination of novel techniques to overcome datasets for intrusion detection are KDD 1999 datasets
the challenges imposed by the use of NetFlow data. In [34] the created by KDD cup challenge. This dataset is based on
author broadly demonstrates how information of the network DARPA 1998 dataset with 4 million records. The KDD 1999
connection can be replicated as genes and how the parameters in datasets consist of normal and 22 attacks categorized into 5
GA can be define in this respect. Lu et al. [35] presented a rule main components. Dos (Denial of Service attacks), R2L
evolution approach based on Genetic Programming (GP) for (Root to Local attacks), Probe (Probing attacks), U2R (User
detecting novel attacks on networks. Joshi et al. [36] classify the to Root attack) and normal. There exist 41 number of
TCP network traffic as an attack or normal using HMM and to attributes containing features related to basic, content and
build an anomaly detection system. Fan et al. [37] proposed an traffic.
algorithm to generate artificial anomalies to coerce the inductive
learner into discovering an accurate boundary between known Table 3: List of the Complete Basic Features of TCPconnection
classes of normal connections and known intrusions, and
anomalies. Amor et al. [38] uses a simple form of a Bayesian Basic Type Represented Description
Features
network that can be considered a Nave Bayes classifier in
intrusion detection. Li et al. [39] applied AprioriAll, an algorithm Duration Continuous Integer Time duration of
for mining frequent sequential pattern in Data mining field, to connection
discovery multistage attack behavior patterns. Hu et al. [40] Protocol, Symbolic Nominal Type of the
presented a new approach, based on Robust Support Vector type protocol (TCP,
Machines (RSVMs) for anomaly detection. Wanger et al. [41] UDP and ICMP)
proposed an approach for evaluating Netflow records by referring
to a method of temporal aggregation applied to Machine Learning Service Symbolic Nominal HTTP, Telnet, FTP,
techniques. In paper [42], they proposed a new SVM approach, SMTP and others
named Enhanced SVM, which combines soft-margin SVM and Flag Symbolic Nominal Connection status
one class SVM methods
Src bytes Continuous Integer Number of bytes
3. Cyber-Security Datasets sent per connection
Dst bytes Continuous Integer Number of bytes
Data plays an important role for ML and DM models. Today data
received per
is new oil for digital world (or for industries), i.e., based on
connection
collecting data, competitors can launch affordable services in
market. For example, based on collecting requirements/ demands Land Symbolic Binary Value=1 if port
of particular things in an area, companies can shift towards to sell numbers and src/
their product in that specified area/ region. The necessary dst IP address are
elements for the efficient conduction of research related to cyber same
security includes the right choice of data and its’ proper utilization. Wrong Continuous Integer Total of bad
To comprehend the ML and DM algorithms, put forth by a fragment checksum packets
number of authors, requires a better understanding of data sets.
Urgent Continuous Integer Sum of urgent
We can achieve cyber security of data with the help of different
packets
gatherings like Win Dump or Wireshark tool to acquire the
network data packets. It can also be done using the current public
datasets. Hence, this section discusses current cyber security datasets in
a. DARPA: DARPA (Defense Advanced Research Projects detail. Now next section will discuss a brief introduction of data
Agency) intrusion detection datasets was collected and mining and machine learning and necessary uses in detecting
published by the Cyber Systems and Technology Group vulnerabilities or intrusion over cyber – network (cyber space).
MIT/LL (Massachusetts Institute of Technology Lincoln
Laboratory. The data was generated using network 4. Introduction to Data Mining (DL) and Machine
simulation and compiled based on TCP/IP network data. The Learning (ML) for Cyber Security
datasets can be downloaded from the website and it primarily
includes: DARPA 1998, 1999, 2000. DARPA 1998 consists The terms Machine Learning (ML), Data Mining (DM), and
of data collected for 9 weeks, which includes training data Knowledge Discovery in Databases (KDD) are often used
(seven weeks) and of test data (two weeks). Similarly, interchangeably. As per research, KDD process is represented as
DARPA 1999 consists of data collection for five weeks whole and deals with extracting valuable, earlier unknown
wherein training data is for three weeks and the last two knowledge/information from data. Fayyad et al. [43], has clearly
weeks is test data. DARPA 2000 includes scenario-specific mentioned and explained the process of DM as a specific step in
datasets. Table 3 lists the complete basic features of TCP KDD which handles the implementation of algorithms for
connection. retrieval of sequences from data. It can hence, be observed that
www.astesj.com 78
 G. Rekha et al. / Advances in Science, Technology and Engineering Systems Journal Vol. 5, No. 3, 72-81 (2020)
they possess common characteristics between ML and DM. The assumed that the effect of a feature value of a given class doesn’t
steps involved in KDD process are as follows: data selection, data depend on the values from other features and is called conditional
cleaning and pre-processing, data transformation, application of independence. One of the most efficient, robust and best methods
DM algorithms, result interpretation/ evaluation. DM is one step to prevent noisy data is by making use of Naïve – Bayes classifiers.
among all and used for extracting patterns from data by applying The highlight feature being that it calls for only a small amount of
algorithms. It’s to be pointed out that there is a plethora of training data to approximate the strictures needed for
publications [e.g., Cross Industry Standard Process for Data categorization.
Mining (CRISP-DM) [44] along with industry participants who
consider the process DM. • K-Nearest-Neighbor

These two terms are commonly discussed together and are applied K-Nearest-Neighbor (k-NN) is a classification which is one of the
interchangeably. According to Arthur Samuel Creator of Machine simplest and fundamental ones, working well even in the presence
Learning (ML) defined “ML as a field of study that makes the of little or absolutely no prior knowledge regarding the data
computers to learn by itself without being explicitly programmed”. distribution and it’s based on the process of learning by
The machine learning algorithms mainly focus on classification equivalence. ‘m’ dimensional numerical attributes are used for
and prediction techniques. The ML algorithms learn from the describing the training samples with each sample replicating a
training/ past data and finds the insights for future/unknown certain point in the m-dimensional space. Hence, we can see that
conditions. The various classification algorithms in general all the points are stored in an m-dimensional pattern space. In the
applied to cyber security are discussed as below. case of an unknown data sample, a k-nearest neighbor classifier
checks out the pattern space for the k training data modules which
• Decision Trees are quite close to that of an unknown sample. ‘Closeness’ refers
to Euclidean distance. The new and unknown sample is
Decision trees are the important and popular techniques used for designated with the most common class from it is nearest k
classification. A decision tree is nothing but a simple flowchart neighbors.
similar to that of the structure of a tree which has every internal
node denoting a test with respect to an attribute such that each • Support Vector Machine
branch indicates the outcome of the test and each leaf node
acquired a class label.ID3 (Iterative Dichotomiser) is a decision It mainly plots the input vector into a space of very high
tree algorithm which was developed by Ross Quinlan. He then dimensions and helps in the construction of a hyper plane. The
represented the successor of ID3 – C4.5 which has turned out to hyper plane has the capacity to separate the data points into
be a benchmark for comprehending algorithms different classes. A great level of distinction is obtained by hyper
planes which has the greatest distance to the closest training data
• C4.5 Algorithm point of any class which is called as the functional margin. It’s
This model forms its basis from ID3 algorithm along with observed that with increase in margin, there’s a lower
additional characteristics to acknowledge the issues faced by that generalization error for the classifier. The hyperplane is a decision
of ID3. It’s considered to have a greedy approach and it is said to boundary for the two classes. In reality, the persistence of a
possess a top-down recursive divide and conquer method.Given a decision boundary ensures the detection of a misclassification
data samples S, C4.5 applies divide and conquer algorithm for tree which is created by a particular method. Classification, regression,
generation and the process is stated as follows: and other jobs are implemented with SVM.

a) If S is small or all the data samples in S belong to the same • Repeated Incremental Pruning to Produce Error Reduction
class, then the leaf node is labeled with the most frequent (RIPPER)
class in S.
b) Or else, the process of selecting attributed is made use of to RIPPER, is a generic methodology used for effectively applying
control the criterion of the splitting process. The criterion separate-and -conquer rule learning. It helps in increasing the
for the process of splitting indicates which attribute is to be precision of protocols by replacing or re-enforcing the individual
tested at node S by identifying the most efficient way to norms. Reduce Error Pruning was implemented to create the rule
distinguish the tuples into separate classes. and the created rules are often restricted to a smaller number. It
ensures the pruning of each rule right after the creation and
The process continues recursively to form a decision tree. removal of data samples. Reduced error pruning facilitates the
handling of huge training sets, thus improving the precision. The
• Naive Bayes Algorithm:
below mentioned steps are carried out: Spot the characters/
The Naive Bayes algorithm (NB) employs a simplified version of features from the training data and identifies the split of all
Bayesian learning method. It involves statistical classifiers. The attributes essential for categorization (i.e., feature/dimensionality
probabilities of membership can be determined with the help of reduction). Comprehend models using the training data and use
these classifiers and it has its foundation on Bayes theorem. It’s the trained model to segregate the unknown data. In the initial

www.astesj.com 79
 G. Rekha et al. / Advances in Science, Technology and Engineering Systems Journal Vol. 5, No. 3, 72-81 (2020)
stage pf training, each feature with a corresponding class is more useful in solving these issues/such problems using
acquired by using suitable algorithms from the training set. The regression, prediction, and classification techniques. In this smart
perspectives of ML/DM are mainly categorized into three classes era, we have large amount of data (generated from internet/ web
supervised, unsupervised and semi-supervised. The different browsing) and shortage of talented employees in cyber-security
machine learning and data mining methods applied for cyber domain/area. So, Machine Learning is the only solution to provide
security is mentioned in Table 1-2. efficient results in minimum time. Hence, in order to understand
importance of ML techniques for solving the IDS problems,
5. Role of Machine Learning and Artificial Intelligence which focus on the design of the single, hybrid and ensemble
towards Cyber Security classifier models (with discussing several algorithms, used
Today cyber security has put everything on risk, due to attracting datasets). This work also discussed “How Machine earning, and
billions of online users over internet and storage of data over data mining can be useful in identifying/ detecting intrusion, in
internet (at cloud side). Everyday every country is facing critical section 4”?
attacks by enemy nations on their computer labs, systems or Hence, we found that uses of different classifier/ ML techniques
network, which can create a situation of third world war. Till in IDS a promising study in cyber security and artificial
today, we are detecting cyber attackers or hackers through human intelligence. It will make attraction of young scientists from
work-force, for that we require a huge number of skilled research communities for a long time. For future work, this work
workforce to look over or prevent against any cyber threats. But has identified some valid points which are: removal of data
in near future, there is a possibility that intrusion or vulnerabilities redundancy and irrelevant features for the training phase (have
detection can be done by using machine learning and artificial important role in system performance), i.e., consideration of best
intelligence. Also, it will provide several benefits to society and feature selection algorithm will play an important role in the
avoid the problem of weaker security, lower efficiency, leaking of classification techniques in near future. Also, multiple or different
personal information by Internet of Things, increasing selection of algorithms for featured selection will provide best
vulnerabilities on cyber and physical space or cyber physical possible solutions in various scenarios/ intrusion detection in a
systems. Note that recently many critical attacks have been network. Last, but not the least, cyber security and intrusion
measured by several countries on their nuclear programs/ sites detection systems works well and shows a better performance
[45]-[48] by their enemy nations. On other side, Artificial with ensemble classification algorithms when compared to single
Intelligence (AI) will reduce required workforce (requirement of classification algorithms.
cyber security professionals), speed of detection of intrusion, etc. Authors’ Contributions
AI can help in living life longer and better through its emerging
innovations. Such benefits of AI are listed in following ways. Gillala Rekha drafted this manuscript, whereas Shaveta Malik and
Meghna Manoj Nair have put this article’s content in correct order.
• Handling huge volumes of security data In last, Amit Kumar Tyagi has approved this manuscript.
• Picking out threat needles in cyber haystacks Acknowledgement
• Acceleration of detection and response times
• Keeping up in the Artificial Intelligence arms race This research is funded by the Anumit Academy’s Research and
• Breathing space for human cyber security teams. Innovation Network (AARIN), India. The author would like to
thank AARIN India, an education foundation body and a research
Hence, data mining, machine learning and artificial intelligence network for supporting the project through its financial assistance.
are necessary components for 21stcenturygeneration. So, we will
see the tremendous uses of Machine learning, Artificial Conflict of interest
intelligence in next 20-30 years, which will do many/ everyday
task and will serve humanity better and better. The authors declare that they do not have any conflict of interest
with respect to publication of this research work.
6. Conclusion and Future Enhancements
Scope of the Work
In the recent/ several decades, several attacks have been
measured/ noticed. Due to this reason, cyber security and This work has been written through collecting articles from
intrusion detection has been coined in this smart era. Due to several international journals like ACM, IEEE, Springer, Wiley,
enormous internet usage (in the past decade), the vulnerabilities etc. This work will be useful for future researchers who are
of network security (in a network) need to be overcome. working towards computer vision/ the use of machinelearning or
Overcoming such issue has become an important issue today. In artificial intelligence towards cyber security.
general terms, Intrusion detection system is used to identify the
flaws in the system such as unauthorized access and unusual References
attacks over the secured networks. Hence, to solve this issue, [1] S. Mukkamala, A. Sung, A. Abraham, Cyber security challenges: Designing
several authors had discussed many studies. In that, we found that efficient intrusion detection systems and antivirus tools, Vemuri, V. Rao,
Enhancing Computer Security with Smart Technology. (Auerbach, 2006)
(from literature, refer section 2 and 3) machine learning can be
(2005) 125–163.
www.astesj.com 80
 G. Rekha et al. / Advances in Science, Technology and Engineering Systems Journal Vol. 5, No. 3, 72-81 (2020)
[2] A. Sundaram, An introduction to intrusion detectionCrossroads 2 (4) (1996) [25] C. Palagiri, Network-based intrusion detection using neural networks,
3–7. department of Computer Science Rensselaer Polytechnic Institute Troy, New
[3] V. Chandola, A. Banerjee, V. Kumar, Anomaly detection: A survey, ACM York (2002) 12180–3590.
computing surveys (CSUR) 41 (3) (2009) 15. [26] D. Apiletti, E. Baralis, T. Cerquitelli, V. DElia, Characterizing network
[4] B.-C. Park, Y. J. Won, M.-S. Kim, J. W. Hong, Towards automated traffic by means of the netmine framework, Computer Networks 53 (6) (2009)
application signature generation for traffic identification, in: Network 774–789.
Operations and Management Symposium, 2008. NOMS 2008. IEEE, IEEE, [27] A. Tajbakhsh, M. Rahmati, A. Mirzaei, Intrusion detection using fuzzy
2008, pp. 160–167. association rules, Applied Soft Computing 9 (2) (2009) 462–469.
[5] J. Cannady, Artificial neural networks for misuse detection, in: National [28] A. Ahmed, et al., “Modeling and Simulation of Office Desk Illumination
information systems security conference, Vol. 26, Baltimore, 1998. Using ZEMAX,” in 2019 International Conference on Electrical,
[6] H. Brahmi, I. Brahmi, S. B. Yahia, Omc-ids: at the cross-roads of OLAP Communication, and Computer Engineering (ICECCE), 2019, pp. 1–6.
mining and intrusion detection, in: Pacific-Asia Conference on Knowledge [29] C. Kruegel, D. Mutz, W. Robertson, F. Valeur, Bayesian event classification
Discovery and Data Mining, Springer, 2012, pp. 13–24. for intrusion detection, in: Computer Security Applications Conference,
[7] H. Zhengbing, L. Zhitang, W. Junqi, A Novel Network Intrusion Detection 2003. Proceedings. 19th Annual, IEEE, 2003, pp. 14–23.
System (NIDS) based on signatures search of data mining, in: Proceedings [30] S. Benferhat, T. Kenaza, A. Mokhtari, A naive bayes approach for detecting
of the 1stinternational Conference on Forensic Applications and Techniques coordinated attacks, in: Computer Software and Applications, 2008.
in Telecommunications, information, and Multimedia and Workshop, ICST, COMPSAC’08. 32nd Annual IEEE International, IEEE, 2008, pp. 704–709.
2008, p. 45. [31] K. Sequeira, M. Zaki, Admit: anomaly-based data mining for intrusions, in:
[8] H. Han, X.-L. Lu, L.-Y. Ren, Using data mining to discover signatures in Proceedings of the eighth ACM SIGKDD international conference on
network-based intrusion detection, in: Machine Learning and Cybernetics, Knowledge discovery and data mining, ACM, 2002, pp. 386–395.
2002. Proceedings. 2002 International Conference on, Vol. 1, IEEE, 2002, [32] L. Bilge, E. Kirda, C. Kruegel, M. Balduzzi, Exposure: Finding malicious
pp. 13–17. domains using passive dnsanalysis., in: Ndss, 2011.
[9] L. Carl, et al., Using machine learning techniques to identify botnet traffic, [33] L. Bilge, D. Balzarotti, W. Robertson, E. Kirda, C. Kruegel, Disclosure:
in: Local Computer Networks, Proceedings 2006 31stIEEE Conference on. detecting botnet command and control servers through large-scale netflow
IEEE, 2006. analysis, in: Proceedings of the 28thAnnual Computer Security Applications
[10] F. Jemili, M. Zaghdoud, M. B. Ahmed, A framework for an adaptive Conference, ACM, 2012, pp. 129–138.
intrusion detection system using bayesian network, in: Intelligence and [34] M. S. A. Khan, Rule based network intrusion detection using genetic
Security Informatics, 2007 IEEE, IEEE, 2007, pp. 66–70. algorithm, International Journal of Computer Applications 18 (8) (2011) 26–
[11] G. R. Hendry, S. J. Yang, Intrusion signature creation via clustering 29.
anomalies, in: Data Mining, Intrusion Detection, Information Assurance, and [35] W. Lu, I. Traore, Detecting new forms of network intrusion using genetic
Data Networks Security 2008, Vol. 6973, International Society for Optics programming, Computational intelligence 20 (3) (2004) 475–494.
and Photonics, 2008, p. 69730C. [36] S. S. Joshi, V. V. Phoha, Investigating hidden markov models capabilities in
[12] C. Kruegel, T. Toth, Using decision trees to improve signature-based anomaly detection, in: Proceedings of the 43rd annual Southeast regional
intrusion detection, in: International Workshop on Recent Advances in conference-Volume 1, ACM, 2005, pp. 98–103.
Intrusion Detection, Springer, 2003, pp. 173–191. [37] W. Fan, M. Miller, S. Stolfo, W. Lee, P. Chan, Using artificial anomalies to
[13] J. Zhang, M. Zulkernine, A. Haque, Random-forests-based network detect unknown and known network intrusions, Knowledge and Information
intrusion detection systems, IEEE Transactions on Systems, Man, and Systems 6 (5) (2004) 507–527.
Cybernetics, Part C (Applications and Reviews) 38 (5) (2008) 649–659. [38] N. B. Amor, S. Benferhat, Z. Elouedi, Naive bayesvs decision trees in
[14] F. Gharibian, A. A. Ghorbani, Comparative study of supervised machine intrusion detection systems, in: Proceedings of the 2004 ACM symposium
learning techniques for intrusion detection, in: Communication Networks on Applied computing, ACM, 2004, pp. 420–424.
and Services Research, 2007. CNSR’07. Fifth Annual Conference on, IEEE, [39] Z. Li, A. Zhang, J. Lei, L. Wang, Real-time correlation of network security
2007, pp. 350–358. alerts, in: e-Business Engineering, 2007. ICEBE 2007. IEEE International
[15] S. Mukkamala, A. H. Sung, A. Abraham, Intrusion detection using an Conference on, IEEE, 2007, pp. 73–80.
ensemble of intelligent paradigms, Journal of network and computer [40] W. Hu, Y. Liao, V. R. Vemuri, Robust support vector machines for anomaly
applications 28 (2) (2005) 167–182. detection in computer security., in: ICMLA, 2003, pp. 168–174.
[16] W. Li, Using genetic algorithm for network intrusion detection, Proceedings [41] C. Wagner, J. Francois, T. Engel, et al., Machine learning approach for ip-
of the United States Department of Energy Cyber Security Group 1 (2004) flow record anomaly detection, in: International Conference on Research in
1–8. Networking, Springer, 2011, pp. 28–39.
[17] A. Abraham, C. Grosan, C. Martin-Vide, Evolutionary design of intrusion [42] T. Shon, J. Moon, A hybrid machine learning approach to network anomaly
detection programs., IJ Network Security 4 (3) (2007) 328–339. detection, Information Sciences 177 (18) (2007) 3799–3821.
[18] J. V. Hansen, P. B. Lowry, R. D. Meservy, D. M. McDonald, Genetic [43] U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, The KDD process for extracting
programming for prevention of cyber-terrorism through dynamic and useful knowledge from volumes of data, Communications of the ACM 39
evolving intrusion detection, Decision Support Systems 43 (4) (2007) 1362– (11) (1996) 27–34.
1374. [44] C. Shearer, The crisp-dm model: the new blueprint for data mining, Journal
[19] A. ˚Arnes, F. Valeur, G. Vigna, R. A. Kemmerer, Using hidden markov of data warehousing 5 (4) (2000) 13–22.
models to evaluate the risks of intrusions, in: International Workshop on [45] Tyagi, Amit Kumar, Building a Smart and Sustainable Environment using
Recent Advances in Intrusion Detection, Springer, 2006, pp. 145–164. Internet of Things (February 22, 2019). Proceedings of International
[20] W. Lee, S. J. Stolfo, K. W. Mok, A data mining framework for building Conference on Sustainable Computing in Science, Technology and
intrusion detection models, in: Security and Privacy, 1999. Proceedings of Management (SUSCOM), Amity University Rajasthan, Jaipur - India,
the 1999 IEEE Symposium on, IEEE, 1999, pp. 120–132. February 26-28, 2019. Available at SSRN:
[21] M. Panda, M. R. Patra, Network intrusion detection using naive bayes, http://dx.doi.org/10.2139/ssrn.3356500
International journal of computer science and network security 7 (12) (2007) [46] Tyagi. Amit Kumar, Cyber Physical Systems (CPSs)- Opportunities and
258–263. challenges for improving cyber security, International Journal of Computer
[22] Y. Hu, B. Panda, A data mining approach for database intrusion detection, Applications, 2016,137 (14).
in: Proceedings of the 2004 ACM symposium on Applied computing, ACM, [47] Sravanthi Reddy, M. Shamila, Amit Kumar Tyagi, Cyber Physical Systems:
2004, pp. 711–716. The Role of Machine Learning and Cyber Security in Present and Future,
[23] Y. Li, J. Xia, S. Zhang, J. Yan, X. Ai, K. Dai, An efficient intrusion detection Computer Reviews Journal, PURKH, Vol. 5 (2019).
system based on support vector machines and gradually feature removal [48] Meghna Manoj Nair, Amit KumarTyagi, RichaGoyal, Medical Cyber
method, Expert Systems with Applications 39 (1) (2012) 424–430. Physical Systems and Its Issues, Procedia Computer Science Volume 165,
[24] R. P. Lippmann, R. K. Cunningham, Improving intrusion detection 2019, Pages 647-65.
performance using keyword selection and neural networks, Computer
networks 34 (4) (2000) 597–603.

www.astesj.com 81

View publication stats