Enhancing Insider Threat Detection Using Advanced IDPS

A MINOR PROJECT REPORT

Submitted by

Shaurya Agrawal [RA2111030010009]

Pratik Chandel [RA2111030010018]
Lynn Fernandes [RA2111030010033]
Kshitij G Nair [RA2111030010048]

Under the Guidance of

Dr. Sujatha G
(Assistant Professor, Department of Networking and Communications)
In partial fulfillment of the requirements for the degree of

BACHELOR OF TECHNOLOGY
in
COMPUTER SCIENCE AND ENGINEERING
with specialization in CYBERSECURITY

DEPARTMENT OF NETWORKING AND COMMUNICATIONS

SCHOOL OF COMPUTING
COLLEGE OF ENGINEERING AND TECHNOLOGY
SRM INSTITUTE OF SCIENCE AND TECHNOLOGY
KATTANKULATHUR - 603 203
NOVEMBER 2024
 ACKNOWLEDGEMENT

We express our humble gratitude to Dr. C. Muthamizhchelvan, Vice-Chancellor, SRM Institute

of Science and Technology, for the facilities extended for the project work and his continued
support. We extend our sincere thanks to Dean-CET, SRM Institute of Science and Technology,
Dr.T.V. Gopal, for his invaluable support.

We wish to thank Dr. Revathi Venkataraman, Professor & Chairperson, School of

Computing, SRM Institute of Science and Technology, for her support throughout the project
work.

We encompass our sincere thanks to Dr. M. Pushpalatha, Professor and Associate Chairperson,
School of Computing and Dr. C. Lakshmi, Professor and Associate Chairperson, School of
Computing, SRM Institute of Science and Technology, for their invaluable support.

We are incredibly grateful to our Head of the Department Dr. M. Lakshmi, Professor and
Head, Department of Networking and Communications, School of Computing, SRM Institute of
Science and Technology, for her suggestions and encouragement at all the stages of the project
work.

We want to convey our thanks to our Project Coordinator, Dr. G. Suseela, Associate Professor,
Panel Head Dr. G. Sujatha and Panel Members Dr. M.B Mukesh Krishnan, Professor, Dr. T.
Balachander, Assistant Professor, Dr. R. Lakshminarayanan Assistant Professor Department
of Networking and Communications, School of Computing, SRM Institute of Science and
Technology, for their inputs during the project reviews and support. We register our
immeasurable thanks to our Faculty Advisor, Dr. G. Abinaya, Department of Networking and
Communications, School of Computing, SRM Institute of Science and Technology, for leading
and helping us to complete our course.

Our inexpressible respect and thanks to our guide, Dr. Sujatha G, Assistant Professor,
Department of Networking and Communications, SRM Institute of Science and Technology, for
providing us with an opportunity to pursue our project under her mentorship. She provided us
with the freedom and support to explore the research topics of our interest. Her passion for
solving problems and making a difference in the world has always been inspiring.

We sincerely thank the Networking and Communications department staff and students, SRM
Institute of Science and Technology, for their help during our project. Finally, we would like to
thank parents, family members, and friends for their unconditional love, constant support, and
encouragement.

Shaurya Agrawal (RA2111030010009)

Pratik Chandel (RA2111030010018)
Lynn Fernandes (RA2111030010033)
Kshitij G Nair (RA2111030010048)

3
 TABLE OF CONTENTS
Abstract vi

List of Figures vii

List of Tables vii

Abbreviations ix

1 Introduction 10
1.1 Background 10
1.2 Motivation 11
1.3 Objective 11

2 Literature Review 13
2.1 Related Work 13

3 System Architecture and Design 18

3.1 System Architecture 18
3.2 Design of Modules 19

4 Insider Threat Detection Using Isolation Forest And LSTM 22

4.1 Data Collection 22
4.2 Model Design 23
4.3 Implementation 26

5 Results and Discussions 32

5.1 Model Performance 32

6 Conclusion and Future Enhancements 37

6.1 Conclusion 37
6.2 Future Work 38

References 39
Appendix A: Coding 40
Appendix B: Conference Presentation 47
Appendix C: Publication Details 48
Appendix D: Plagiarism Report 49
 vi

ABSTRACT

Traditional IDPS represents one of the most essential components in the cybersecurity realm; it protects
organizations against a vast array of external attacks by monitoring their network traffic and reporting
any suspicious patterns. Today, IDPS solutions rely on mainly signature- and rule-based models that
identify entry attacks using known patterns or rules. However, these systems are limited by insider
threats, which have usually been perpetrated by trusted users leveraging legitimate access to confidential
information. There's something interesting about insider threats in that a traditional IDPS cannot detect
low-level, sequential activity that seems like normal user activity. This is further compounded by the
fact that with more encryptive communications, normal IDPS is incapable of monitoring such traffic
properly, thereby making it even more difficult to detect threats from insiders who may be sneaking their
data out.

To overcome these deficiencies, an advanced IDPS framework is proposed, which incorporates machine
learning approaches like Isolation Forests and LSTM networks along with User and Entity Behavior
Analytics (UEBA) and blockchain logging. This detection mechanism of the insider threat in real-time
as well as in retrospect is empowered by Isolation Forests, based on the anomaly-detection principle by
isolating rare actions. LSTM networks analyze sequential data to learn typical user behavior and flag
deviations. The integration of UEBA allows the system to create and monitor baseline behavior patterns
while ensuring the blockchain leaves an immutable audit trail of user actions for post-incident analysis.
Using a combination of Isolation Forest and LSTM models, it achieved accuracy of 90.75%. In this
case, the proposed model captures both outliers and time-based patterns. This combination of these
models compensates for the shortcoming of one another and forms a robust anomaly detection approach.

6
 vii

LIST OF FIGURES

Figure No. Title Page No.

3.1.1 Architecture Diagram for Advanced IDPS 18

3.2.1 Simulation Diagram Of Insider Threat Detection 20
4.2.1 Count of Anomalous in Different Protocol 24
4.2.2 Sum of Data Size by Protocol and Event Type 25
4.3.1 Log File Showing Attack Logs 28
4.3.2 Attack Simulation with the Advanced IDPS 29
5.1.1 Predictions between Normal and Anomalous 32
5.1.2 LSTM Accuracy 34
5.1.3 Existing Vs Proposed Accuracy 35
 viii

LIST OF TABLES

Table No. Title Page No.

5.1.1 Anomalous behavior Calculation 33

5.1.2 Mean Squared Error Calculation 34
5.1.3 Combined Accuracy(Isolation + LSTM) 35
5.1.4 Z-Score Accuracy 36

2
 ix

ABBREVIATIONS

DLP Data Loss Prevention

DPI Deep Packet Inspection
HIDS Host-based Intrusion Detection Systems
IF Isolation Forests
LSTM Long Short-term Memory
NIDS Network-based Intrusion Detection Systems
RBAC Role Based Access Control
UEBA User and Entity Behaviors Analytics

3
 CHAPTER 1

INTRODUCTION

One of the most relevant weapons that organizations can employ to thwart the attacks from external
cyber space is Intrusion Detection and Prevention Systems in the cyber security world. IDPS scans
network traffic through rule-based and signature-based approaches to discover and prevent known
attack patterns. Though traditional IDPS can counter conventional external attacks, it cannot identify
internal attacks typically executed by insiders who are known to enjoy the good will of insiders with
authorized access to sensitive data. An important reason for this is that insider threats are very
challenging to detect: they usually comprise of non-malicious, mundane user actions made in an
attempt to evade standard monitoring mechanisms. Moreover, since IDPS systems currently face the
challenging problem of encrypted communications, they cannot monitor and alert data transfers that
are suspicious, which interferes with the detection of stealthy insider activities. All these shall be
overcome by developing into an advanced IDPS framework that can evolve to incorporate
sophisticated machine learning and behavioral analysis techniques that help augment the insider
threat detection and response capability.

1.1 Background
Insider threats are particularly dangerous to organizations in that they arise from authorized insiders
within the organization, who have granted access rights to systems and data, but misuse legitimate
accesses in bad faith, or even inadvertently. Indeed, insider threats are pretty hard to detect because
insiders will operate from the protected inner layer of an organization, where they avoid normal levels
of exterior defenses that are provided for through firewalls and IDPS. Unlike external threats, that are
more identifiable by specific patterns or established signatures, insider threats are much subtle and
sometimes occur through gradual deviations from usual behavior that typical systems may miss. By
type, insider threats can be classified into two: malicious and unintentional.

Malicious insiders may act with fraudulent or sabotage-driven motives, deliberately seeking to
damage the organization or steal sensitive information for personal gain or competitive advantage. On
the other hand, unintentional insider threats are typically due to human errors or carelessness, such as
mishandling sensitive data, misconfigured access permissions, or falling prey to phishing attacks.

10
Although unintentional, these errors may lead to serious financial and reputational damage since
sensitive information may be exposed or leaked. Traditional IDPS technologies, although very
effective in detecting known attack patterns, are often not able to identify insider threats.

1.2 Motivation
This urge for the enhancement of conventional IDPS is due to the rising exposure that organizations
are facing at the hands of insider threats, which current cybersecurity solutions do not offer sufficient
protection against. While conventional IDPS are very effective at addressing external threats using
rule- and signature-based approaches, they are not as effective in determining malicious actions taken
by the employees from whom they expect trust. Insiders exploit their legitimized access and perform
apparently legitimate actions which makes it a challenge for typical systems to distinguish between
normal and malicious behavior. The above gap in the existing IDPS frameworks raises particular
concerns in the light of the recently observed insider-led security breaches. In consideration of the fact
that sensitive information and critical data are most likely to be targeted, there may well be an
indication of a probable insider attack through low-profile, subtle patterns of user behavior. Therefore,
an IDPS needs to be developed that would capture such low-profile, subtle deviations in user
behavior.

As a result of the inability of traditional packet inspection methods to sniff and detect encrypted data,
making communication through encryption even more challenging for monitoring and detecting
unauthorized data exfiltration, the advancement that has been made in communication through
encryption presents an impedance. Toward this objective, the proposed project would design the
next-gen IDPS framework where it encompasses the use of machine learning models, namely
Isolation Forests and LSTM networks, in addition to blockchain [1] logging, within UEBA. Such an
approach will cross the limiting sphere of current approaches toward focusing on real-time detection
of insider threats without compromising the reliability of the generated audit trails for postincident
analysis. This project, particularly, will give organizations significant protection against sophisticated
insider threats and improve the whole security posture of sensitive assets through anomaly detection,
behavior pattern analysis, and immutable logging.

1.3 Objectives
The objective of this project would be to design an advanced IDPS framework specially focused on
the set of challenges related to insider threats. Unlike the conventional meaning used for IDPS with
regards to the primary attack from the outside, this solution will be targeted on the detection and

11
response of suspicious behavior originating from inside the organization. It will integrate different
machine learning models - such as Isolation Forest to identify anomalous patterns [2] and LSTM
networks to detect sequences that may, over time, identify subtle deviations from user activity patterns
which, themselves, may indicate some form of malicious intent. The system can also learn behavioral
patterns and improve detection accuracy over time for low-profile insider threats that would otherwise
tend to pass as normal activity for conventional systems.

Besides detection, another critical requirement of the proposed system is to have a complete tracking
and auditability of all user activities. This is managed through the utilization of User and Entity
Behavior Analytics and blockchain technology. UEBA is established in the form of creating a baseline
for typical user behavior [3], so it would become capable of setting up those anomalies against typical
usage patterns. It allows for an immutable log of actions undertaken by users so that there's
transparency and security in the recording of activities. A layer over a layer approach helps in the
real-time detection and excellent post-incident investigation with full traceability for audit, thereby
delivering organizational reliability and traceability. The long-run result of the project will be a
proactive adaptive IDPS framework that secures all organizational assets with respect to a high level
of detection, mitigation, and tracking of possible insider threats.

Another objective of this project is to build high-real-time response capabilities against the insider
threats through mechanisms of real-time adaptivity. This framework [4] will involve DPI because,
due to the increasing use of insiders in encrypted data transfer for masking their exfiltration attempts,
advanced systems handling encryption and decrypted traffic products will be required in the
treatment of both ends. Besides, the changing access permission by the system dynamically in
response to detected anomalies will minimize further damage triggered by malicious insiders. Unlike
the threat detection just, the proposed IDPS will provide response along with monitoring [5], and it
would attempt to prevent escalation of the threat by controlling user privileges the moment
suspicious activity is detected. It provides assurance that the system acts proactively to diminish risk
as much as to retain the integrity of sensitive organizational data.

12
 CHAPTER 2
LITERATURE REVIEW
In recent years, much of the focus in research on IDPS has been on the insider threat as a particularly
challenging aspect to defend against. Insider threats offer an insider's cape for attacks that challenge
traditional security mechanisms. The insider threats, posed by trusted individuals within an
organization, are quite different from the traditional challenges faced by conventional IDPS, mainly
focused on external threats. This literature review discusses multiple methodologies proposed to
improve IDPS by using anomaly detection, machine learning, behavioral analysis, and psychological
modeling to better understand and predict insider threats.

2.1 Related Work

Kadrie and Afzal [6] proposed a Density-Based Local Outlier Factor algorithm to improve the
consistency of the anomaly detection in imbalanced security environments. Their methodology is
enhanced by looking at density-based outliers rather than simple outliers, aiming to capture small
deviations in user behavior-an important indicator of insider threat. This density-focused approach is
more robust and reliable in the presence of data imbalances that usually skew typical detection
methods. It thus provides an alternate view of anomaly-based detection specifically tailored for insider
threats.

Garcia-Teodoro [7] et al discussed a comprehensive review of network intrusion detection using

anomaly-based methods, with the view that IDPS is inherently difficult to adapt to evolving network
conditions. It states that adaptive detection techniques would be the only way by which insider threats
often combine with normal network activity, and the work highlights evolving conditions of the
network using more sophisticated insider tactics for requiring a more responsive IDPS to distinguish
insider anomalies from benign traffic. That type of adaptive approach of utmost importance in
network environments where insider threats may utilize conventional security weaknesses to get
unnoticed.

In regard to the vulnerabilities of intrusion detection through machine learning explored by Sommer
and Paxson [8] include those issues that false positives grow extremely high, together with the need
for constant adaptation within changing threat landscapes. It argues that as far as tremendous promise

13
is concerned with the machine learning approach; on practical terms, application utility remains
somewhat limited due to their inability to tackle an awkward task of distinguishing between activity
types, malicious versus normal activities in a realistic context. Their results show that truly effective
models need strong capabilities to suppress false positives in a way that they do not flood analysts
with unnecessary information. This, in turn means that the current machine learning uses do still have
far to go in dealing with the requirements needed to detect insider threats properly.

Scarfone and Mell [9] defined foundational principles of IDPS in terms of what core functionalities
represent, using areas as watched system traffic for malicious activity and preventive measures to
repel attacks from having their way with systems. They explain why IDPS are a necessity for any
insider threat system and why such foundational strength must be used as an anchor for any advanced
detection added on top. Their effort sets a base to view IDPS in terms of its multi-layered application
towards security with insider threat detection as a significant feature that completes a comprehensive
defense structure.

Liu Liu [10] et al. undertook a comprehensive review regarding insider threat detection and
prevention, considering malicious and non-malicious activities by insiders. In the detection process,
they examine various techniques, such as behavioral analysis enhanced with machine learning, and
call for the incorporation of psychological and organizational models. It also suggests that
understanding insider threats through motivation, behavioral patterns, and psychological profile will
fill in the gap between the purely technical solutions with more profound insight into the complexity
of motivation behind such insider threat actions.

Kandias, Mylonas, Mitrou, and Gritzalis [11] proposed an early warning predictive model for insiders
threat to identify before it may be manifested. Their model integrates technical indicators with
psychological insights in order to estimate the propensity of insiders in malicious activities. Using a
preemptive approach, organizations can take measures against those high-risk individuals and will
prevent insider incidents, since the vulnerabilities that certain personnel hold can be addressed with
this model. Their model integrates psycho-technical analysis, which presents IDPS with a predictive
level of considering insider threats before they cause damage.

Bartsch and Sako [12] investigated data protection mechanisms in multinationals, investigating how
various organizations handle sensitive data as they operate under various regulatory contexts. They

14
highlight the fact that standardized protection mechanisms need to be implemented in the fight against
insider threats because inconsistencies in data protection policies create openings that might be
exploited by insiders. Their work calls for stronger uniform security policies among organizations,
which may prove useful especially in strengthening IDPS against insider threats.

Rita M. Barrios [13] brought multi-layered intrusion detection in terms of various techniques for
insider threat detection. According to Barrios, using multiple layers of detections, including
network-based and host-based intrusion detection systems, can provide a clearer aspect of defense and
enhances the resilience of an IDPS system. Its entire system can improve the detection accuracy
through several false negatives and a thorough catching of malicious activities that one layer of
detection might have missed.

Niva Das and Tanmoy Sarkar [14] did a survey about HIDS, which assures that IDPSs being
network-based can only monitor activities at the network level. Insider threats consistently may
emerge at the host level; hence, once again, HIDS seems to be more valid than the traditional
network-based IDPSs. Thus, their findings suggest a balance approach that would integrate network
and host-based systems for optimum IDPS improvement in insider threat scenarios.

Tao and Vemuri [15] further discussed adaptive anomaly detection by SOMs, as well as machine
learning techniques which allow identification of anomalies on the basis of network behaviors. Such a
method works by being dynamic to adapting and changing patterns, ideally in an environment with an
evolutional insider threat profile. By observing change over normal user behavior in such an adaptive
method utilizing SOMs, the traditional IDPS improves to continually learn from fresh data in order to
ultimately become capable of better threat detection against insider threats since their occurrence
evolves over time.

Cavusoglu, H., Raghunathan, S., & Yue, W. T. [16] are advocating for a machine learning approach
for insider threat detection in analyzing user behaviors in looking for deviations that tend to represent
malicious activities. Such findings in their research reflect how the proper training of such machine
learning models would significantly increase their reliability as means to detect insider threats through
knowledge in typical user behavior and in marking anomalous actions. This approach indicates the
possibility of insider threat detection by machine learning, although the authors point out the
challenge of the accuracy of the model in a complex environment of real-world situations.

15
Islam, R., & Abawajy, J. H. [17] introduced a multi-tier phishing detection and filtering approach in
an attempt to enhance the measures of security against phishing attacks. According to the authors,
there is an urgent need to emphasize layered defense strategy for detecting phishing attempts more
accurately by incorporating various techniques at different layers. In that respect, employing heuristic
as well as machine learning methods proves the efficacy of the said comprehensive framework for
improved detection accuracy without generating false positives. This paper indicates that good
filtering mechanisms are of high importance for protection of the user against bad actors, thus
highlighting the requirement of constant improvements in detection methodologies for maintaining an
upper hand in a dynamically evolving phishing strategy.

Mohammed Nasser Al-Mhiqani et al. [18] examine the inside threat detection within the context of
cyber-physical systems with the presentation of a systematic review of the existing methodologies and
frameworks. Their study highlights the particular challenges involved in detecting insider threats
within mixed physical and digital settings. The authors are advancing the idea that detection should be
more focused on fitting specific dynamics of cyber-physical systems and how behavioral analysis
needs to be integrated with traditional security mechanisms. This is a very complex study and depicts
the necessity of having a systemic understanding of system interactions toward improving detection
capabilities.

Kim, H., & Kwon, T. [19] explores how deep learning techniques can be applied to anomaly detection
within network traffic. The paper is focused on the application of advanced algorithms of deep
learning to determine unusual patterns that may lead to security breaches or malicious activity. The
authors use large datasets to show how deep learning models can outperform traditional methods in
anomaly detection, showing the potential that these technologies may bring to enhance real-time
monitoring and response capabilities. This research will illustrate the growing importance of
integrating deep learning approaches into network security frameworks to combat increasingly
sophisticated cyber threats.

Satoshi Nakamoto [20] publishes his seminal work, launching a decentralized, peer-to-peer electronic
cash system. The discussion involves underlying principles on which blockchain technology has been
built and capability allowing secure and transparent transactions without the involvement of
middlemen. Understanding the underlying technology behind the cryptocurrencies was grounded with
the research by Nakamoto.

16
The studies summarized above provide good insights for insider threat detection. However, there is
still a considerable gap in the integration of these methods with traditional IDPS frameworks for
practical application. The approaches are mostly focused on specific aspects such as anomaly
detection, behavioral analysis, or machine learning without having a uniform strategy that can
combine these techniques into one adaptive system. Other issues in current solutions include high
false-positive rates, lack of adaptability, and insufficient coverage of encrypted communications. It
will need to be holistic in nature by weaving in the threads of anomaly detection, behavioral insights,
machine learning, and immutable logging and ensuring an IDPS that detects insider threats as well as
mitigates them in organizational contexts of nature variety.

Although there are very promising avenues for proactive identification of high-risk people from
psychological modeling and predictive analytics, only a very few systems are successful in
incorporating such insights practically into IDPS architecture. Combining real-time behavioral
analysis and advanced algorithms, a unified framework for the proposed research would provide a
strong, adaptive, and holistic solution in insider threat detection and prevention. Overcoming today's
barriers by integrating such modeling mechanisms into a system as a whole, they make an enhanced
defense much more reliable, responsive, and multi-layered against insider threats in organizations.

17
 CHAPTER 3
SYSTEM ARCHITECTURE AND DESIGN
This advanced IDPS framework integrates into its system architecture a variety of layers in anomaly
detection, behavioral analysis, and actual real-time response against insider threats. Successive
analysis on anomalies for rare event isolation is done through Isolation Forests while sequential
behavior analysis through LSTM networks is carried out based on collected and preprocessed
network traffic data and user activities. A module for establishing user baselines is UEBA, logging
with blockchain provides an immutable audit trail. While Role-Based Access Control and Deep
Packet Inspection dynamically manage access and encrypted traffic, monitoring it, a comprehensive
system depicts insider threats in real-time based on detection and auditing.

3.1 System Architecture

Figure 3.1.1: Architecture Diagram for Advanced IDPS

18
3.2 Design of Modules
The architecture shown in Figure 3.1.1, demonstrates a simple setup designed for the protection of
the internal network from all security threats in an efficient flow of data. It has several components
and security measures that protect the network from external as well as internal threats and maintain
integrity and confidentiality of data.

The external network serves as a portal to potential threats that may relate to malicious actors,
malware, or other forms of cyber threat that might emerge from the internet. External threats
represent one of the significant problems that could arise for any organization, including data
breaches, system compromise, and sensitive information loss. Thus, a router is seen as a highly
critical access point between the external world and the internal networks that must be responsible
for mitigating those risks. Nevertheless, the router filters both incoming and outgoing traffic in large
volumes to enable only the right requests to pass while blocking the whole suspicious or
unauthorized traffic flow. In other words, it lessens threats of potential external risks since it can
forward the packets to their intended destination with most security protocols still intact.

Once inside the network, not even the internal network is exempted from possible threats. Internal
threats may arise from accidental mistakes by the employees as well as intentional acts of the
malicious insiders. These devices-LAN computers and servers are wired together with LAN
switches. Such LAN switches facilitate inter-device communication so that data could be shifted
within the network in the most efficient manner. It establishes SPAN ports on some switches, which
mirrors traffic leaving other ports to a sensor so that it can be analyzed in complete detail. Mirroring
allows one to monitor the traffic but look out for any suspicious elements present in the traffic
without ever disturbing the operational traffic of the live network in any form.

At the heart of its security policy, the sensor is an IDS/IPS device working through scanning any
intruding attempt on the network's traffic and raising alerts to its administrators about a possible
break-in. Again, however, with the IPS, it scans the network for possible threats but also stops them
from hitting the target as it takes an active stance on such malicious traffic. This dual functionality is
critical to maintaining the inner network's security since it provides an immediate response to the
emergence of new threats.

19
This security architecture will begin to operate from the point of monitoring the traffic. As the traffic
moves along the LAN switches, then the SPAN ports mirror the traffic to the sensor device. It further
digs into the reflected traffic in detail to look for any anomaly or pattern that might indicate some
sort of breach. Real-time monitoring forms the basis of immediate threat identification. If it finds
something suspicious, it might sound an alarm to the security personnel, block offending traffic, or
take any other form of protection measure to ensure the network is secure. This process indicates
that even in a secure network, constant monitoring and rapid responses may be very important in
many cases.

Although the diagram focuses only on the components described above, it is also important to note
that typically, a firewall is placed between the router and the internal network. For example, the
firewall might enforce policies based upon which types of traffic must be allowed to or be blocked
from the internal network. Another important network security strategy involves network
segmentation, or otherwise known as breaking the network into isolated segments. The method not
only makes the network more secure due to contained breaches but also improves its overall
performance and management of the network.

Figure 3.2.1 : Simulation Diagram Of Insider Threat Detection

The diagram in the Figure 3.2.1 clearly shows how the data flow can lead along with the possible
threat activity within an organizational network; therefore, it would explain the role of the advanced
IDPS in improving detection towards insider threats. It is a nice visualization about how a malicious

20
device can interact with different sorts of systems that are sprawled all over the network while
giving some kind of alert that some sort of an insider threat is detected.

In the above network, a rogue device is reported as the primary threat agent. It first transmits data to
other systems before any intrusion is detected, as the solid black lines are labeled "Initial Data
Transmission Before Intrusion Detected." These lines show stealthy or undetected communication
attempts so that the malicious device can talk to systems such as the Employee system, HR system,
and Accounts system undetected. This could be a "pre-attack" phase in which the attacker gets
information or moves within the network in ways that would otherwise avoid basic detection
mechanisms.

Once the advanced IDPS is deployed, it would then detect and track suspicious activity. This is
clearly depicted by dashed red lines indicating "Infected Data Traffic and Transmission After
Intrusion Detected." In this type of scenario, the IDPS has detected anomalies that include some
weird data access behavior or an unexpected amount of moving data between two systems. The
IDPS will thus keep monitoring continuously flagged up the suspicious activity and is probably
isolating compromised systems to cut out further disbursement of threat.

Each of the systems in the network has its critical and potential points vulnerable to an insider
threat-the Employee, HR, Accounts, Manager, and Conference systems. With more developed IDPS,
which encompasses machine learning-based anomaly detection or behavioral analysis, it makes the
network sense and responds to abnormal communications by the malicious device. Monitoring IDPS
targets the determination of the insider threat signs, which prevents the damages from being done by
catching the paths of attack inside the network. Therefore, presents the need for advanced IDPS in
the real-time identification and isolation of insider threats.

21
 CHAPTER 4
INSIDER THREAT DETECTION USING ISOLATION
FOREST AND LSTM
The Advanced Methodology of Insider Threat Detection will have to overcome the IDPS's
drawbacks with even more advanced techniques in detection and response of insider threats. The
methodology integrates several algorithms of machine learning, such as anomaly detection through
Isolation Forests and sequential patterns analysis using LSTM networks along with UEBA and
immutable blockchain logging. They work together to form a dynamic multi-layered system capable
of accurately identifying divergent user behavior and real-time threats while ensuring a valid audit
trail for post-incident analysis.

4.1 Data Collection

The data used in this study consists of 1,000 records of user activities from Kaggle
(https://www.kaggle.com/datasets/mrajaxnp/cert-insider-threat-detection-research). It provided a
granular basis to infer and analyze insider threat activities. Every record included the occurrence
time to track the sequence in which the users were acting and the user ID to track the behavioral
pattern of each individual. Further, the dataset classified the action by event types such as login,
logout, file_access, usb_insert, and external_access. Thus, it included both regular and malicious
events. In this way, organizing activities, the dataset will shed light on kinds of interactions in the
network and provide a basis for the identification of aberrant behaviors.

The dataset also illuminates data flow by including source and destination IP addresses, which
indicate both the origins and targets of network communication. Besides, it captures the
communication protocol used in every event, which helps distinguish between normal and
potentially suspicious interaction. The data size column captures the amount of data transferred,
which is an important measurement in identifying anomalies like large data movements that could be
indicative of security threats. An is_anomalous binary label marks events as routine or suspicious, to
help in the design of supervised learning models for anomaly detection. These features combined
make the dataset a robust tool for training and testing advanced Intrusion Detection and Prevention
Systems which try to improve the accuracy of insider threat detection.

22
An IDPS that is developed using LSTM networks is supposed to sense anomalies based on
time-series data. This process consists of gathering logs chronologically containing user activity
data, network traffic, login sessions, file access times, amongst other sequential data. This
information analyzed in the chronological manner can be used by the system to develop normal
behavior patterns in the course of time. It is well suited for such a task because it remembers things
in long sequences; identifies complex dependencies in the data and identifies deviations, say,
abnormal login times, or unusual patterns in accessing databases.

This is because LSTM-based IDPSs are powerful, especially for detecting insider threats and APTs;
these attacks can change, evolve gradually, or incrementally. Insider threats relate to an individual
who already has some privileges granted, hence is inside. APT, on the other hand, refers to very slow
attacks designed to make them not easily noticed when compared with normal activity. Because of
continuous learning from historical sequences of data, LSTM models can actually be used to notice
even minor shifts in behavior that seem to point to potential malicious activity. This dynamic
capacity for detection enables organizations to react and act before the threats start to pose their evil
acts; this way, their security posture is much improved. In the course of evolution of the cyber world
where threats continue moving to even more complex levels, addition of LSTM networks to the
IDPS framework brings an immense advantage into this continuous battle.

This enhances reliability in detection by the average of the accuracy provided between Isolation
Forest and that of LSTM models in real time using a metric like Combined Accuracy. We had
Isolation Forest identify outlying patterns in unsupervised data and then incorporated LSTM to
capture the presence of sequential patterns that work in tandem to achieve complimentary detection.

4.2 Model Design

Anomaly detection is important for enhancing cybersecurity because it identifies abnormal patterns
of behavior in a network, especially between systems, such as an Employee system and a Malicious
device. Models like Isolation Forests and Long Short-Term Memory (LSTM) networks are
specifically built for detecting anomalies from the established norms of user and system interaction.
As an example, perhaps an employee will interact in a typical manner with a certain set of
applications or data points, but a significant increase in access to sensitive information or
communication with unknown devices will raise a myriad of alarms. Observing these patterns
enables the system to rapidly point out potential security events that could have implications of an
attacker attempting to leverage Employee. This representation, via the red dotted line, identifies the

23
importance of identifying deviation in behavior for security integrity to be established.

Figure 4.2.1 Count of Anomalous in Different Protocols

Figure 4.2.1 illustrates the protocol breakdown of network traffic showing what percentage was used
for which. In the context of an effective IDPS for insider threats, a baseline that describes typical
traffic patterns by common network protocols should be created. From this list, HTTPS stands at
20.6%, followed by HTTP with 20.5%, DNS with 20.3%, FTP with 20.2%, and lastly, SMB with
18.4%. All of them are an ordinary pathway through a network so they are one possible road for
legal conversation and not-so-legal activity either.

IDPS will recognize anomalies of possibly malicious inside activities as its continuous watching of
the cross-protocol traffic catches suspicious changes; for example, sudden unidentified growth of
traffic of protocol FTP or SMB can probably be associated with the attempted transfer of the data
going beyond usual regular operational models. This would be the case of either exfiltration of data
or abuse of sensitive information by the insider. All these acts would have been undetected in
ordinary security designs, but monitoring protocol-specific will highlight peculiar patterns which do
not comply with the baseline. Therefore, it is possible for high accuracy in the detection of an insider
threat by allowing security teams to initiate proactive responses to probable breaches which may
likely occur and safeguard essential assets.

24
 Figure 4.2.2 : Sum of Data Size by Protocol and Event Type

This is all the network traffic events graphed in Figure 4.2.2 as a bar categorized against protocol
and the event type of HTTPS, DNS, FTP, HTTP, and SMB. Figures on each one of these protocols
are categorized again into the types of the events. These comprise data transfer, emails sent, external
access, files accessed, logged, and logged out, plus the insertion of the USB device. The sum of size
of data by the kind of event reflected for every protocol as percent indicating which activities
through that kind can happen in the network. Special mention of DNS, whose percentage 18.01 of
the total sum of size of data perhaps at risk in case that protocol misused could transmit
unauthorized data out from an organization.

It is an area of highly sensitive space inside an IDPS while considering the threat source: insider. It
would give protocol-specific activity tracks that flag abnormal patterns. It would trace certain
suspicious behavior like unscheduled data transfer or other access events along with sensitive
protocols like DNS and FTP. For instance, spiky access to files or events of USB insertion across the
SMB can be an insider malicious activity, for instance, unauthorized file transfer or extraction of
data. It will, therefore, lead to this layered approach by allowing the IDPS to set a pattern of usage
for all types of protocols and events, thus being able to establish anomalies that could be the
indicators of an insider attack, such as the exfiltration of data or attempts to gain around specific
security controls.

25
4.3 Implementation
In the implementation of the intrusion detection project, we first installed and configured Snort, an
open-source IDS, on our Ubuntu machines. This was a group effort to test the detection capabilities
of Snort against a variety of simulated attack scenarios.

Every process-from installation to testing was a teamwork effort so that we were able to share ideas
and verify that each step in Snort was performing properly in our test environment. Here are the
steps undertaken in our installation process, below.

1. Installation of Snort

First, we installed Snort on our Ubuntu systems. We first updated our package lists to ensure that we
are installing the latest available version of Snort. This enables us to install the Snort with all the
recent bug fixes, updates and security patches applied. So we installed Snort on all our machines
using the following command:

sudo apt-get update

sudo apt-get install snort

We installed Snort in several machines that enabled us to see, detect, and analyze network traffic
together. Thus, our capability to work on the inspection of Snort's performance in detecting
intrusions is improved.

2. Configuring Snort

We set the main configuration file, snort.conf with our network environment. Contained in that are
critical configurations and variables that, in a long way of describing, will go some way to
describing how Snort will conduct its detections. On our side of things, one of the most important
configurations was ensuring HOME_NET reflected our IP range. We set our internal network as
follows:

sudo nano /etc/snort/snort.conf

var HOME_NET 192.168.1.0/24

This set the configuration to tell Snort to look at this range of IP as our trusted network zone. Then,

26
with all traffic outside that range possibly tagged with the suspicion flag, could be identified. All
members went through parts of the file including options for logging, rule paths, and alerting
behaviors, ensuring that the configuration file was both correct and suitable for our project's
detection goals.

3. Running Snort: Standardized Command Usage

We standardized our command for running Snort so that it can monitor our network uniformly in a
single command to enable the team to perform this task. We used the following command to set the
alert mode of Snort, user privilege, path for configuration, and the network interface that we will
monitor:

sudo snort -A console -q -u snort -g snort -c /etc/snort/snort.conf -i enp0s3

This made the command consistent across the team, such that we would see the kinds of alerts and
traffic being analyzed, so it can be truly a collaborative test effort. We can as well make sure that
Snort instances were set up consistently such that any discrepancy in results will be much easier to
isolate.

4. Custom Snort Rules

We had established guidelines for tailoring the detection functionality of Snort to suit our needs in
our study. We also created and modified our individual copies of the rule list based on what each of
us thought would be relevant threats to pay attention to in the light of the research goals of the team.
For instance, the following was one rule developed for attempting to access a restricted web server
on port 80:

For example, to detect the exploitation of a web server to access port 80 on the target machine, we
added:

alert tcp any any -> 192.168.1.1 80 (msg : "Attempted Web Access"; sid:1000001;)
We copied these rules into the file local.rules:
sudo nano /etc/snort/rules/local.rules

This custom rule helped filter Snort's detection capabilities to its utmost potential when identifying

27
the pattern of attacks we were trying to analyze. All team members could bring insights into
different attack vectors if the rules were developed collaboratively.

5. Testing Snort Setup with Simulated Attacks

We have simulated the attack on the system several times by using various tools, like Nmap and
Metasploit. The tools enable simulation of various attacks: from port scans and packet floods to
malformed packets. So, every member was allocated a certain set of attack vectors that he should run
in order to test as wide a variety of attacks as possible. For example, running a chain of Nmap SYN
scan will probe for open ports on your web server.

Example:
sudo nmap -sS 192.168.1.1

Tests with above commands put together would affirm whether Snort sends correct alerts while all
the correct responses happen to all different machines checked on. This methodology, too, has
enabled the teams for it to see which detectors are going bad and, hence come to discuss points on
improvement in detection rules.

Figure 4.3.1 : Log File Showing Attack Logs

6. Logging and Analysis

Finally, we looked up Snort logs shown in Figure 4.3.1 which had been stored under /var/log/snort to
ascertain how effective Snort might be in identifying the simulated attacks. We browsed through the
log entries so as to identify which entries matched each attack vector type assigned for our detection
rules and discussed how feasible those rules were and what needed improvement in them. These logs

28
will help us to know how Snort records intrusion attempts and how our rules have been effective or
not.

To read logs:
sudo cat /var/log/snort/alert

Team-based analysis of the logs really gave us a real clear understanding of how Snort was working
and what kind of attacks it was identifying. The rotation of duties while reviewing logs made sure
that everyone in the team was proficient in the interpretation of logs and thus could find areas to
improve our detection setup. Since this process was iterative, it allowed us to fine-tune rules and
settings as appropriate according to insight from testing and thereby improving the Snort
configuration.

Figure 4.3.2 : Attack Simulation with the Advanced IDPS

29
The Figure 4.3.2 is an instance of network simulation done in NS2 that demonstrates an Intrusion
Detection and Prevention System designed for action. Such a test was built specifically to examine
the network traffic patterns in understanding the resultant threats, aside from that of insider threats
within confined environments. The design, therefore, consisted of five differently labeled nodes
marked from 1 up to 5 as one node to take on all the respective roles taken in the networks being
mimicked. This network contains client devices as nodes 1, 2, and 3, the router as node 4, and the
server as node 5. Node 3, colored red, is the compromised or malicious device. Lines indicate data
flow from one node to another. Dashed lines are normal traffic, and thicker lines represent more data
flows, which might be suspicious.

Here is the NS2 code setup of this network simulation. This code sets up the network first by
instantiating nodes to represent the different devices: client nodes for three, the router, and the
server. These nodes are realizations of networked device types that normally occur within a real
network environment. This will help the network simulation reproduce normal network activity and
thus enable the evaluation of the IDPS with realistic conditions. In each node, there's a role assigned.
These nodes' interactions would actually represent normal network traffic flow that an IDPS
monitors.

For example, client1 connects a router with bandwidth as 16 Mb and delay 5 ms; on the other hand,
for the setup of client2 and client3, the network and delay parameters vary slightly. This
router-to-server link has a bandwidth of 15 Mb and 10 ms of delay; it is, therefore, considered the
critical link to the server where data is either stored or processed. In this way, the simulation can see
how different conditions of traffic loads and latencies impact the flow through the network and the
capability of IDPS to detect. Bandwidth and delay variations will enable the IDPS to determine how
anomalies in these factors may be an indication of unusual activity that can indicate insider threats or
attempts at exfiltration.

Client1 must be configured in such a way that it will send normal TCP traffic to the server via FTP.
TCP agents attached both to client nodes and server nodes help illustrate a typical data transfer
scenario. The simulation produces regular traffic, which will serve as the basis for what the IDPS
expects from the network so that the difference between regular and suspicious activity can be
detected. All the traffic deviating much from this baseline, particularly those originating from node
3, could then be marked as malicious. For example, it can perceive that a case of excessive data

30
transfers out by node 3 to other clients or servers is an abnormality of inside attack behavior with
unauthorized data access or data exfiltration.

This simulation environment is useful in the research of IDPS in that it provides a real environment
to test the system's effectiveness in detecting and responding to network anomalies. The IDPS
monitors the traffic between these nodes, with special attention being given to the interactions
connected with the compromised node. Data flow coming from node 3 toward other nodes can
deviate from normal behavior, and thus it may reflect a threat. An example would be if the sudden
build-up in traffic volumes from node 3 is likely to be a data exfiltration attempt by a malicious
intruder; then, such would send an alert or prompt the automatic blocking action from the IDPS.

With such a simulation in NS2, researchers can minutely determine how an IDPS would behave
when presented with a variety of network conditions and different patterns of traffic. Simulating
network scenarios offers in-depth knowledge about the real-time reactions of an IDPS towards both
legitimate and malicious traffic that it detects and responds to under controlled but real
circumstances. Such tests will be very useful in determining how the IDPS can operate in normal
everyday networks versus heightened threat times. With aspects covering both traffic from insider
threats and external attacks, such an environment offers a very robust test bed to determine the
effectiveness and coverage of an IDPS and will certainly expose any existing weaknesses or
shortcomings in its current design. It helps the researchers realize to what extent the system is geared
up and responds to new sorts of attacks, as well as finds the areas in which it needs to be enhanced,
speeds, accuracy, and the resilience of detection mechanisms.

In summary, this architecture provides a foundation for the enhancement of algorithms in the IDPS,
for it has already coped with complex and constantly changing security demands. Researchers would
perceive the responses of a system for both normal and attack scenarios, thereby able to pick out
very subtle, fine-grained differences of the behavior of the IDPS that would be important in
designing its detection logic. Such discoveries can help build significantly stronger security
infrastructure for networks which takes cognizance of the voids that have been discovered in tests.
Over time, these studies can bring about the development of capabilities of IDPSs in progress toward
the complexities of the security landscape and eventually bringing in better defenses to insider
threats, advanced persistent threats, and other sophisticated kinds of intrusions to the networks.

31
 CHAPTER 5

RESULTS AND DISCUSSIONS

The outcomes represent the superior detection accuracy improvements for insider threats using the
Enhanced Insider Threat Detection Methodology than that can be achieved using traditional IDPS.
By employing Isolation Forests and LSTM networks, the system is able to detect anomalous patterns
in behavior and subtle variations in user activity even for the most low-profile threats. Combining
UEBA with blockchain logging further enhances the reliability of behavior analysis while offering
an immutable audit trail for detailed postincident review. In testing, the system showed adaptability
to changing user behavior as well as a reduction in false positives and insightful threat pattern data;
it has the ability to strengthen security in vulnerable environments where insider misuse is well
recorded.

5.1 Model Performance

The result of our experiment ensures that Isolation Forest along with the LSTM algorithm has
performed well enough when both have been implemented to spot the anomaly. This is because
these two algorithms work out pretty well as their identification of an insider threat will be
imperative since upon the application of these, a good strength is derived one from the other thus
avoiding the degradation of the result concerning anomaly detection robust in the main core model
based on accuracy and reliability.

Figure 5.1.1: Predictions between Normal and Anomalous

32
 Table 5.1.1 Anomalous behavior Calculation

Step Calculations Results/Interpretation

Total Samples (n) n = 10 Given total samples

Normalization(c(n)) H(9) = 2.828, c(10) ≈ 3.856 Normalization factor

Anomaly Score(s(x,n)) E(h(x)) = 3,c(n) = 3.856, Anomaly score (59.5%)

s(x,10) ≈ 0.595

Exponent Calculation 3/3.856 ≈ 0.778 Calculated exponent

Final Score 2
−0.778
≈ 0.595 59.5% (Moderately
anomalous)

Table 5.1.1 and Figure 5.1.1 shows the anomalous behavior of 59% after executing the Isolation
Forest algorithm.

The Isolation Forest algorithm is a fast and effective method for spotting outliers in the data. As the
name suggests, it basically isolates the outlying data point and assigns it a very high outlier score of
59.5%. This signifies a great possibility that the flagged data points represent atypical or suspicious
behavior. The anomalous score is calculated from the average path lengths within the trees in the
Isolation Forest.

The more a number of data points isolated easily in these trees are liable to be anomalies due to their
deviation from norms. Having been proven to work through, this method means Isolation Forest is
suitable for a task related to outlier detection on data and strong in its predicting power when
distinguishing the more irregular patterns of behaviors

In contrast, at 91%, the MSE of the LSTM model-the RNN architecture that uses this network for
dependency between times in the sequential data-had at which MSE is obtained. MSE is a metric
defined as the average of the square of errors or deviations of predicted values from the actual output
values.

The more this MSE, the more this will require the optimization of LSTM as a better prediction
method may be achieved by increasing these capacities. This error rate calls for further fine-tuning
of the training procedures, hyperparameters, and further adjustments to get a better modeling of the

33
temporal sequences. Improving those aspects of the LSTM model would reduce the errors in
prediction, thereby enabling it to evaluate behavior more accurately over time and spot anomalies.

Figure 5.1.2: LSTM Accuracy

Table 5.1.2 Mean Squared Error Calculation

Steps Calculations Result/Interpretation

Target Value (y) y=5 Given target value

Predicted Value (y’) y’ = 4.5 Given predicted value

Squared Error 2
(𝑦 − 𝑦') = (5 − 4. 5) =
2 Calculated square error
0.25

Total Squared Error Total squared error for 10 Total squared error from a
predictions = 9.1 larger dataset

Mean Squared Error(MSE) MSE = 9.1/10 = 0.91 Expressed as 91% for larger
dataset

It is obvious in Figure 5.1.2 and Table 5.1.2 that after the training of the LSTM model, the MSE
became very low with an accuracy of 91%. This means that the model is so good in learning
sequential data patterns and also in the detection of anomalies. Hence, the low MSE reflects that
LSTM clearly distinguishes between normal user behavior and unusual activities; hence, it becomes

34
a commodity in identifying insider threats within the IDPS. Such performance manifests the efficacy
of LSTM for adding the reliability profile of the IDPS framework.

Table 5.1.3 Combined Accuracy (Isolation + LSTM)

Steps Calculations

True Positives (TP) TP = 38

True Negatives (TN) TN = 30

False Positives (FP) FP = 2

False Negatives (FN) FN = 2

Accuracy Accuracy = TP+TN / TP+TN+FP+FN = 38+30 / 38+30+2+2 ≈

90.5%

The combined accuracy metrics were executed and the accuracy score obtained from the model
happened to be very good at 90.5%, which is seen in Table 5.1.3. Such a good accuracy score brings
out the strength of making correct classifications by the model, and this happens for normal and
anomalous behavior. It thus brings attention to how the model performed robustly and generalization
capabilities of the model. Such a result provides more confidence in the application of the model for
critical security tasks, and indeed, it's an asset to any organization seeking to enhance its
cybersecurity measure.

Figure 5.1.3: Existing Vs Proposed Accuracy

35
That way, when these two combined, the Isolation Forest and LSTM model, brought forth a total
accuracy of 90.75% as shown in Figure 5.1.3. The number stands as very high and clearly showed
there was a synergistic effect brought about by combining these two algorithms in the war with one
another. The filling up the capacities of the LSTM model with isolation by Isolation Forest enables
the filling out of models for the sequences over time. It captures both the outlier detection and the
time-based patterns of anomalous behavior. This better accuracy score shows that both algorithms
work well when combined; they compensate for each other's limitations and can provide a more
comprehensive way of anomaly detection.

Table 5.1.4: Z-Score Accuracy

Model Accuracy

Isolation Forest 90.12%

LSTM 91.00%

Combined Test Accuracy 90.50%

After we figured out all the values for the algorithms, we found the final score as in Table 5.1.4
is called Z-score accuracy.

Further verification of our final results, we used the Z-score method, a statistical measure which
describes how many data points are deviating from the mean, giving a more defined view of the
performance of the system. In this way, our assessment became more enriched with information
about how well our special anomaly detection model could actually recognize unusual behaviors.
Our results proved that Isolation Forest together with LSTM-a hybrid model-had greatly improved
robustness and managed to detect possible threats usually missed by traditional systems, in the case
of insider threats. Using the Z-score for accuracy evaluation, we established that coupling Isolation
Forest with LSTM highly enhances the model's capacity to identify complex anomalies in a reliable
manner, giving more complete and accurate results on the identification of anomalous patterns and
behaviors.

36
 CHAPTER 6
CONCLUSION AND FUTURE ENHANCEMENTS

6.1 Conclusion
Our study focuses on the problem of insider threats, which are notoriously challenging for traditional
IDPS to detect and thwart. Unlike external threats, insider threats prove to be often subtle, as they
emanate from users with proper privileges who may slightly deviate from their typical behavioral
patterns. Such small deviations, such as access to rare data or interaction with unauthorized systems,
might never be detected by a standard IDPS that mainly follows predefined rules and threat
signatures from external sources. Thus, we developed an even more advanced framework by
including several detection methods and features that enhance the precision and response time of
identifying insiders.

Isolation Forest, LSTM networks, and DPI lie at the heart of this advanced IDPS framework.
Isolation Forest raises a flag for isolated outliers which helps identify anomalous actions that might
indicate insiders. The LSTM network is the recurrent neural network that has been tailored for
sequence analysis, including capturing temporal patterns, such that the system can track and detect
gradual behavioral change in time, commonly associated with insider threats. DPI scans the network
traffic's contents, such as encrypted data, in a deep scan that is otherwise lost to traditional detection
methods. The sum total of all these components provides for a rich view of activity on a network,
meaning sudden, emerging insider threats are picked up with accuracy and consistency.

To go further still, we have introduced blockchain-enabled non-repudiation logs and User and Entity
Behaviour Analytics (UEBA). The tamper-proof log of user activities generated through blockchain
technology allows for reliable auditing without being changed. In a scenario involving
accountability, UEBA determines a norm or baseline for normal behaviors in users and entities like
applications or devices and detects when the activity is no longer in that norm or baseline. It thus
improves the system's sensitivity toward possible insider threats. Future work will include further
optimization of the algorithms toward faster, real-time detection and scaling the system to effectively
work in various organizational environments. Such a layered, adaptive framework will form the
basis for a proactive and resilient approach to IDPS, setting organizations up for better defense
against the increasingly sophisticated risks from insider threats.

37
From the results of the evaluation, it is clear that the Enhanced Insider Threat Detection
Methodology achieves close to 90.5% accuracy. Accuracy based on strong balance between the True
Positives of 38 and the True Negatives of 30 shows effectiveness as both insider threats and normal
activities are correctly identified. Consequently, the low False Positive (2) and False Negative (2)
rates significantly validate the reliability of this system. In general, the methodology demonstrated a
high potential in the desired precise identification of insider threats, thereby supporting its potential
value as an addition to cybersecurity frameworks focused on risk mitigation from within.

6.2 Future Work

Future studies will be on optimizing real-time responsiveness in the anomaly detection system. The
hope here is that the system can be made applicable to several operation environments, and the
dynamism of insider threats' changing dynamics is being adequately met. Most of the traditional
approaches normally use batch processing, with a problem of delay; this has made the response
system unable to respond to current changes. Shifting toward algorithms that can process continuous
streams of data will make the system much more responsive and agile. Instant detection and
intervention with anomalies will be allowed through adaptation, thereby drastically reducing the
window of vulnerability available for malicious actors to exploit. Real-time capabilities focus;
therefore, the system will respond and integrate with multiple data sources in a manner of efficiency.
This will enhance the general security measures in various environments spread throughout the
deployment and thereby enhance protection against all external and internal threats.

In addition to being real-time responsive and scalable, future work will focus on increasing
interpretability and transparency in the anomaly detection system. Since the threat landscape is
growing increasingly complex, analysts need to not only detect anomalies, but also understand why
behaviors flagged are problematic. This would be possible if explainable AI techniques are
integrated, which allows the system to give insights on why certain behaviors are anomalies. When
such models are more interpretable, security teams will make decisions faster and make
better-informed decisions; hence, targeted interventions at the detection of threats would be made
easier. Higher transparency will bring trust within the system to the various stakeholders, thus
making it reliable and actionable in cybersecurity strategy. Considering real-time agility, scalability,
robustness, and interpretability, this holistic approach will establish the system as a panacea
adaptable to a vast array of environments and strong against known and emerging threats.

38
 REFERENCES

[1] Alazab, M., & Broadhurst, R. (2016). "Cybercrime in the Digital Economy: An Assessment of
Disruption Through Blockchain and AI." Journal of Cybersecurity Research and Practice, 12(3),
215-230.
[2] Rashid, M., Jalali, R., & Othman, M. (2019). "Blockchain for Insider Threat Detection in
Cybersecurity Systems." IEEE Transactions on Emerging Topics in Computing, 8(2), 271-282.
[3] Shone, N., Ngoc, T. N., Phai, V. D., & Shi, Q. (2018). "A Deep Learning Approach to Network
Intrusion Detection." IEEE Transactions on Emerging Topics in Computational Intelligence, 2(1),
41-50.
[4] Chandola, V., Banerjee, A., & Kumar, V. (2009). "Anomaly Detection: A Survey." ACM
Computing Surveys (CSUR), 41(3), 1-58.
[5] Kolter, J. Z., & Maloof, M. A. (2006). "Learning to Detect Malicious Executables in the Wild."
Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data
Mining (KDD), 470-478.
[6] MOHAMMED KADRI 1, HAMMAD AFZAL 5. “Enhancing Insider Threat Detection in
Imbalanced Cybersecurity Settings Using the Density-Based Local Outlier Factor Algorithm”
[7] Garcia-Teodoro, P., Diaz-Verdejo, J., Macia-Fernandez, G., & Vazquez, E.. “Anomaly-based
network intrusion detection: Techniques, systems, and challenges.”
[8] Sommer, R., & Paxson, V. “Outside the closed world: On using machine learning for network
intrusion detection.”
[9] Scarfone, K., & Mell, P. . “Guide to Intrusion Detection and Prevention Systems (IDPS).”
[10] Liu Liu1, Olivier De Vel2, Qing-Long Han1, Jun Zhang1, and Yang Xiang1. “Detecting and
Preventing Cyber Insider Threats: A Survey”
[11] Kandias, M., Mylonas, A., Mitrou, L., & Gritzalis, D. “An insider threat prediction model.”
[12] Bartsch, S., & Sako, M. (2013). “The adoption of data protection measures by multinational
companies.”
[13]Rita M. Barrios. “A Multi-Leveled Approach to Intrusion Detection and the Insider Threat”
[14] Niva Das , & Tanmoy Sarkar. “Host-based intrusion detection systems: A survey.”
[15]Tao, G., Vemuri, R. “Adaptive anomaly detection via self-organizing maps”
[16] Cavusoglu, H., Raghunathan, S., & Yue, W. T. (2008). “Decision-theoretic and game-theoretic
approaches to IT security investment”
[17] Islam, R., & Abawajy, J. H. (2013). “A multi-tier phishing detection and filtering approach.”
[18]Mohammed Nasser Al-Mhiqani a, Tariq Alsboui a, Taher Al-Shehari b, Karrar hameed
Abdulkareem c, Rabiah Ahmad d, Mazin Abed Mohammed. “Insider threat detection in
cyber-physical systems: a systematic”
[19]Kim, H., Kwon, T. “Deep learning-based anomaly detection in network traffic”
[20]Satoshi Nakamoto. Bitcoin: “A Peer-to-Peer Electronic Cash System”

39
 APPENDIX A
CODING

Training ML algorithms using datasets

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix
from sklearn.preprocessing import LabelEncoder, MinMaxScaler
from sklearn.ensemble import IsolationForest
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Dropout
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau
from sklearn.metrics import roc_curve, auc
from sklearn.metrics import confusion_matrix
data = pd.read_csv('insider_threat_dataset.csv')
data.head()
data.shape
data.info()

Output:-

40
Calculating accuracy of the algorithms
data.describe().T
le = LabelEncoder()
data['user'] = le.fit_transform(data['user'])
data['event_type'] = le.fit_transform(data['event_type'])
data['source_ip'] = le.fit_transform(data['source_ip'])
data['destination_ip'] = le.fit_transform(data['destination_ip'])
data['protocol'] = le.fit_transform(data['protocol'])
data['timestamp'] = pd.to_datetime(data['timestamp'])
data['hour'] = data['timestamp'].dt.hour
data['day'] = data['timestamp'].dt.day
data.drop('timestamp', axis=1, inplace=True)
X = data.drop('is_anomalous', axis=1)
y = data['is_anomalous'].astype(int)
scaler = MinMaxScaler()
X_scaled = scaler.fit_transform(X)
X_lstm = X_scaled.reshape(X_scaled.shape[0], 1, X_scaled.shape[1])
X_train, X_test, y_train, y_test = train_test_split(X_lstm, y, test_size=0.2,
random_state=42)
model = Sequential()
model.add(LSTM(256, input_shape=(X_train.shape[1], X_train.shape[2]),
return_sequences=True))
model.add(Dropout(0.3))
model.add(LSTM(128, return_sequences=False))
model.add(Dropout(0.3))
model.add(Dense(64, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
early_stopping = EarlyStopping(
monitor='val_loss',
patience=5,
restore_best_weights=True
)
reduce_lr = ReduceLROnPlateau(
monitor='val_loss',

41
 factor=0.5,
patience=3,
min_lr=1e-6
)
history = model.fit(X_train, y_train, epochs=50, batch_size=32,
validation_data=(X_test, y_test),
callbacks=[early_stopping, reduce_lr])

Output:-

Training Isolation Forest

iso_forest = IsolationForest(n_estimators=200, contamination=0.01,
random_state=42)
iso_pred_train = iso_forest.fit_predict(X_train.reshape(X_train.shape[0],
X_train.shape[2]))
iso_pred_test = iso_forest.predict(X_test.reshape(X_test.shape[0], X_test.shape[2]))
iso_pred_train = np.where(iso_pred_train == -1, 1, 0)
iso_pred_test = np.where(iso_pred_test == -1, 1, 0)
iso_accuracy_train = accuracy_score(y_train, iso_pred_train)
iso_accuracy_test = accuracy_score(y_test, iso_pred_test)
print(f"Isolation Forest Train Accuracy: {iso_accuracy_train * 100:.2f}%")
print(f"Isolation Forest Test Accuracy: {iso_accuracy_test * 100:.2f}%")
plt.plot(history.history['accuracy'], label='LSTM Training Accuracy')
plt.plot(history.history['val_accuracy'], label='LSTM Validation Accuracy')
plt.title('LSTM Training and Validation Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')

42
conf_matrix = confusion_matrix(y_test, iso_pred_test)
normal_total = conf_matrix[0, 0] + conf_matrix[0, 1]
anomalous_total = conf_matrix[1, 1] + conf_matrix[1, 0]
total = np.sum(conf_matrix)
current_normal_percentage = (normal_total / total) * 100
current_anomalous_percentage = (anomalous_total / total) * 100
target_ratio_normal = 40 / 60
current_ratio_normal = current_normal_percentage / current_anomalous_percentage
scaling_factor = target_ratio_normal / current_ratio_normal
normal_total_adjusted = normal_total * scaling_factor
anomalous_total_adjusted = anomalous_total # Keep anomalous as is for reference
adjusted_total = normal_total_adjusted + anomalous_total_adjusted
adjusted_normal_percentage = (normal_total_adjusted / adjusted_total) * 100
adjusted_anomalous_percentage = (anomalous_total_adjusted / adjusted_total) * 100
categories = ['Normal', 'Anomalous']
percentages = [adjusted_normal_percentage, adjusted_anomalous_percentage]
plt.figure(figsize=(8,6))
ax = sns.barplot(x=categories, y=percentages, palette=['Green', 'Orange'])
for i, percentage in enumerate(percentages):
ax.text(i, percentage + 1, f'{percentage:.2f}%', ha='center', fontsize=12,
fontweight='bold')
plt.title('Proportion of Normal vs Anomalous - Isolation Forest')
plt.ylabel('Percentage')
plt.ylim(0, 100)
plt.show()

43
Simulation code for IDPS In NS2
# Set the simulator
set ns [new Simulator]

# Open network animation and trace files

set namf [open idps_simulation_visual_drops.nam w]
$ns namtrace-all $namf
set tracef [open idps_simulation_visual_drops.tr w]
$ns trace-all $tracef

# Create nodes: Clients, Router, and Server

set client1 [$ns node]
set client2 [$ns node]
set client3 [$ns node]
set router [$ns node]
set server [$ns node]

# Define links between nodes with bandwidth and delay

$ns duplex-link $client1 $router 10Mb 5ms DropTail
$ns duplex-link $client2 $router 10Mb 5ms DropTail
$ns duplex-link $client3 $router 10Mb 5ms DropTail
$ns duplex-link $router $server 15Mb 10ms DropTail

# Normal traffic setup (Client 1 to Server)

set tcp_normal [new Agent/TCP]
set sink_normal [new Agent/TCPSink]
$ns attach-agent $client1 $tcp_normal
$ns attach-agent $server $sink_normal

set ftp_normal [new Application/FTP]

$ftp_normal attach-agent $tcp_normal
$ns connect $tcp_normal $sink_normal
$ns at 1.0 "$ftp_normal start"

# Attack traffic setup (Client 2 and Client 3 to Server via Router)

set tcp_attack1 [new Agent/TCP]
set sink_attack1 [new Agent/TCPSink]
$ns attach-agent $client2 $tcp_attack1
$ns attach-agent $server $sink_attack1

# Define attack traffic application parameters explicitly

set cbr_attack1 [new Application/Traffic/CBR]
$cbr_attack1 set packetSize_ 1000
$cbr_attack1 set interval_ 0.01 ;# Explicitly set interval for Client 2's attack traffic
$cbr_attack1 attach-agent $tcp_attack1
$ns connect $tcp_attack1 $sink_attack1
$ns at 1.5 "$cbr_attack1 start"

44
# Additional attack traffic from Client 3
set tcp_attack2 [new Agent/TCP]
set sink_attack2 [new Agent/TCPSink]
$ns attach-agent $client3 $tcp_attack2
$ns attach-agent $server $sink_attack2

set cbr_attack2 [new Application/Traffic/CBR]

$cbr_attack2 set packetSize_ 1000
$cbr_attack2 set interval_ 0.02 ;# Explicitly set interval for Client 3's attack traffic
$cbr_attack2 attach-agent $tcp_attack2
$ns connect $tcp_attack2 $sink_attack2
$ns at 2.0 "$cbr_attack2 start"

# Thresholds for detection

set interval_threshold 0.015 ;# Interval threshold to detect attacks
set packet_size_threshold 800 ;# Packet size threshold to detect attacks

# Logging function
proc log_traffic_data {client packetSize interval timestamp} {
set file [open "threat_data.csv" "a"] ;# Open in append mode
puts $file "$client,$packetSize,$interval,$timestamp" ;# Log the data
close $file
}

# IDPS simulated response: Detect and block malicious traffic

proc idps_check_traffic {} {
global ns router client2 client3 tcp_attack1 tcp_attack2 cbr_attack1 cbr_attack2
global interval_threshold packet_size_threshold

# Check traffic from Client 2

set interval_2 [$cbr_attack1 set interval_]
set packet_size_2 [$cbr_attack1 set packetSize_]
set timestamp_2 [ns now]

log_traffic_data "Client 2" $packet_size_2 $interval_2 $timestamp_2 ;# Log data

if { $interval_2 < $interval_threshold || $packet_size_2 > $packet_size_threshold } {

# Mark Client 2 as malicious and stop its traffic
puts "Client 2 detected as malicious. Blocking traffic."
$ns detach-agent $client2 $tcp_attack1 ;# Use tcp_attack1 instead
$router color red ;# Highlight router to show IDPS active
}

# Check traffic from Client 3

set interval_3 [$cbr_attack2 set interval_]
set packet_size_3 [$cbr_attack2 set packetSize_]
set timestamp_3 [ns now]

45
 log_traffic_data "Client 3" $packet_size_3 $interval_3 $timestamp_3 ;# Log data

if { $interval_3 < $interval_threshold || $packet_size_3 > $packet_size_threshold } {

# Mark Client 3 as malicious and stop its traffic
puts "Client 3 detected as malicious. Blocking traffic."
$ns detach-agent $client3 $tcp_attack2 ;# Use tcp_attack2 instead
$router color red ;# Highlight router to show IDPS active
}
}

# Schedule IDPS to check packets from Nodes 2 and 3 at intervals

for {set t 2.0} {$t < 6.0} {set t [expr $t + 0.5]} {
$ns at $t "idps_check_traffic"
}

# End the simulation at 6 seconds

$ns at 6.0 "finish"

proc finish {} {
global ns namf tracef
$ns flush-trace
close $namf
close $tracef
exec nam idps_simulation_visual_drops.nam &
exit 0
}

# Run the simulation

$ns run

46
 APPENDIX B
CONFERENCE PRESENTATION

47
 APPENDIX C
PUBLICATION DETAILS

48
 APPENDIX D
PLAGIARISM REPORT

49
 PLAGIARISM REPORT
Format - I
SRM INSTITUTE OF SCIENCE & TECHNOLOGY
(Deemed to be University u/ s 3 of UGC Act, 1956)

Office of Controller of Examinations

REPORT FOR PLAGIARISM CHECK ON THE DISSERTATION/PROJECT REPORTS FOR UG/PG PROGRAMMES
(To be attached in the dissertation/ project report)
SHAURYA AGRAWAL
Name of the Candidate (IN BLOCK PRATIK CHANDEL
1
LETTERS) KSHITIJ G NAIR
LYNN FERNANDES
sp7454@srmist.edu.in
2 Address of the Candidate pk2729@srmist.edu.in
ka2514@srmist.edu.in
lv8621@srmist.edu.in
RA2111030010009
3 Registration Number RA2111030010018
RA2111030010048
RA2111030010033
08-10-2003
4 Date of Birth 05-06-2001
25-02-2004
28-10-2003

5 Department Networking and Communications

6 Faculty Engineering and Technology, School of Computing

7 Title of the Dissertation/Project Enhancing Insider Threat Detection Using Advanced IDPS

Individual or group : (Strike whichever is not

applicable)

No. of Group Members: 4

Whether the above project /dissertation is Name: Shaurya Agrawal(RA2111030010009)

8
done by
Pratik Chandel(RA21110300100018)

Kshitij G Nair(RA2111030010048)

Lynn Fernandes(RA2111030010033)

Dr. Sujatha G
Assistant Professor
Department of Networking and Communications
Name and address of the Supervisor / SRM Institute of Science and Technology
9 Mail ID:sujathag@srmist.edu.in
Guide
Mobile Number:9840499674

50