Revolutionizing Cyber Threat Detection

with Large Language Models

Mohamed Amine Ferrag, Mthandazo Ndhlovu, Norbert Tihanyi, Lucas C. Cordeiro,
Merouane Debbah, and Thierry Lestable
Technology Innovation Institute, 9639 Masdar City, Abu Dhabi, UAE
Email: firstname.lastname@tii.ae

Abstract—Natural Language Processing (NLP) domain is expe- areas of cybersecurity, including malware detection in Android
riencing a revolution due to the capabilities of Pre-trained Large applications, identification of spam emails, intrusion detection
Language Models ( LLMs), fueled by ground-breaking Trans- in automotive systems, and anomaly detection in system logs
arXiv:2306.14263v1 [cs.CR] 25 Jun 2023

formers architecture, resulting into unprecedented advancements.

Their exceptional aptitude for assessing probability distributions [5]–[7]. The primary contribution of this work is treating input
of text sequences is the primary catalyst for outstanding improve- data as sequential information and then leveraging the power
ment of both the precision and efficiency of NLP models. This of transformer models. The findings demonstrate the versatility
paper introduces for the first time SecurityLLM, a pre-trained of BERT for cyber threat detection and its potential to combat
language model designed for cybersecurity threats detection. The emergent security challenges.
SecurityLLM model is articulated around two key generative
elements: SecurityBERT and FalconLLM. SecurityBERT oper- This paper proposes a novel network-based cyber threat
ates as a cyber threat detection mechanism, while FalconLLM detection method that leverages the LLM model for detecting
is an incident response and recovery system. To the best of our the ever-evolving cyber threat landscape. The aim is to provide
knowledge, SecurityBERT represents the inaugural application
of BERT in cyber threat detection. Despite the unique nature a proper textual representation of traffic flow data to leverage
of the input data and features, such as the reduced significance the potential of LLM in identifying massive contextual rep-
of syntactic structures in content classification, the suitability of resentations. This textual representation must ensure accurate
BERT for this duty demonstrates unexpected potential, thanks to threat detection and data privacy protection. We aim to propose
our pioneering study. We reveal that a simple classification model, future directions for advancing the cyber threat detection field
created from scratch, and consolidated with LLMs, exceeds the
performance of established traditional Machine Learning (ML) by leveraging the transformer models’ feature.
and Deep Learning (DL) methods in cyber threat detection, Our original contributions are as follows:
like Convolutional Neural Networks (CNN) or Recurrent Neural
Networks (RNN). The experimental analysis, conducted using • Our research introduces a novel privacy-preserving en-
a collected cybersecurity dataset, proves that our SecurityLLM
model can identify fourteen (14) different types of attacks with
coding approach called Fixed-Length Language Encoding
an overall accuracy of 98% (FLLE). By combining FLLE with the Byte-level Byte-
Index Terms—Security, Attacks Detection, Generative AI, Fal- Pair Encoder (BBPE) Tokenizer, we can effectively rep-
conLLM, BERT, Large Language Models. resent text traffic data in a structured manner. Through
the implementation of this technique, we have achieved
I. I NTRODUCTION significant performance improvements compared to using
The increasing number of cyber threats has presented sig- text data with varying sizes.
nificant challenges to the security of various systems and • We implement and train the BERT architecture from
networks [1]. As adversaries consistently evolve their tactics, scratch on encoded data for multi-category classification.
the need for advanced and effective detection mechanisms • We adopt FalconLLM as an incident response and re-
becomes paramount. In this context, natural language process- covery system to analyze the cyber threat and propose
ing (NLP) techniques have emerged as an intriguing potential solutions.
for cyber threat detection [2]. Among these techniques, the • We evaluate the efficiency of our proposed approach
Bidirectional Encoder Representations from the Transformers using an IoT cyber security dataset. Based on our ex-
(BERT) model [3] received substantial attention due to its perimental analysis, our approach successfully detects
exceptional ability to capture contextual information from text fourteen (14) distinct types of attacks with an impressive
sequences. overall accuracy of 98%.
BERT, a pre-trained transformer-based language model, has
exceptionally succeeded in various NLP tasks. Its core poten- The remainder of this paper is organized as follows:
tial to comprehend the multiple dependencies and variations Section II delves into the related work. Subsequently, Sec-
within the sequence has led to its investigation into cyber threat tion III outlines the significant steps involved in developing
identification. By exploiting BERT’s contextual understanding, SecurityLLM, encompassing two keystone elements: Secu-
security researchers have found unique techniques to handle rityBERT and FalconLLM. In Section IV, we evaluate the
diverse cybersecurity concerns [4]. performance of the SecurityLLM model. Lastly, we conclude
Recently researchers are studying the utilization of BERT our research and provide insight into potential future research
and the adaptation of pre-trained language models, in diverse directions of interest in Section V.
 II. R ELATED WORK conLLM. SecurityBERT operates as a cyber threat detection
Our study investigates how Large Language Models (LLMs) mechanism, while FalconLLM is an incident response and
can be effectively leveraged for network-based cyber threat recovery system. The integration of FalconLLM and Security-
detection. Reviewing recent studies, we highlight key sim- BERT, two distinct techniques, can improve the identification
ilarities and advancements in Bidirectional Encoder Repre- of network-based threats, as we will show in Section IV.
sentations from Transformers (BERT) applications addressing Figure 1 hereafter the workflow of SecurityLLM.
cyber threats. A noteworthy study by Alkhatib et al. [8]
demonstrated the feasibility of using BERT for learning the A. SecurityBERT model
sequence of arbitration identifiers (IDs) in a Controller Area
Network (CAN) via a "masked language model" unsupervised 1) Cyber Security Data Collection: We start by collect-
training objective. They proposed the CAN-BERT transformer ing cyber security data through a variety of open-source
model for anomaly detection in current automotive systems databases and repositories for cybersecurity incidents and
and showed that the BERT model outperforms its predecessors threats. Examples include the Common Vulnerabilities and
regarding accuracy and F1-score. Similarly, Rahali et al. [5] Exposures (CVE) database, Open Web Application Security
applied BERT to the static analysis of Android application Project (OWASP), and many others for network security [10].
source code, using preprocessed features to identify existing There are also open-source datasets for specific threats, like
malware. They used BERT to comprehend the contextual phishing or malware. In our research, we utilized data from the
relationships of code words and classify them into representa- EdgeIIoTset dataset [11], a publically available resource. This
tive malware categories. Their results further underscored the dataset contains fourteen (14) attacks related to the Internet of
high performance of transformer-based models in malicious Things (IoT) and Industrial IoT (IIoT) connectivity protocols,
software detection. categorized into five threats: DoS/D-DoS attacks, Information
Chen et al. [7] introduced BERT-Log, an approach for gathering, Man-in-the-middle (MITM), Injection attacks, and
anomaly detection and fault diagnosis in large-scale computer Malware attacks. The DoS/DDoS attack category encompasses
systems that treat system logs as natural language sequences. TCP SYN Flood, UDP flood, HTTP flood, and ICMP flood
They leveraged a pre-trained BERT model to learn the seman- attacks. The Information Gathering category includes attacks
tic representation of normal and anomalous logs, fine-tuning like port scanning, operating system fingerprinting, and vul-
the model with a fully connected neural network to detect nerability scanning. MITM attacks include tactics such as
abnormalities. Seyyar et al. [6] proposed a model for detecting DNS Spoofing and ARP Spoofing. Injection attacks feature
anomalous HTTP requests in web applications, employing Cross-Site Scripting (XSS), SQL injection, and file-uploading
DL techniques and NLP. In a distinct approach, Aghaei et attacks. Lastly, the Malware category covers backdoors, pass-
al. [9] presented SecureBERT, a language model specifically word crackers, and ransomware attacks.
tailored for cybersecurity tasks, focusing on Cyber Threat 2) Relevant features extraction: Given a PCAP file with
Intelligence (CTI) and automation. The SecureBERT model a network traffic log, we extract relevant features from a
offers a practical way of transforming natural language CTI specific time window and return them in a structured format
into machine-readable formats, thereby minimizing the neces- suitable for analysis. Specifically, we identify and separate
sity for labor-intensive manual analysis. The authors devised each network flow in the PCAP file. For each flow identified,
a unique tokenizer and a method for adjusting pre-trained we extract a set of predefined features. Then, we organize the
weights to ensure that SecureBERT understands general En- extracted features into a CSV file format for analysis. Our
glish and cybersecurity-related text. However, SecureBERT is initial exploration involved removing null features, resulting
not designed to process network-based cyber threat attacks. in the discovery of 61 out of the initial 1176 features.
The aforementioned studies leverage pre-trained BERT 3) Textual representation: To leverage the power of LLM,
models and customize them to meet their unique security we process the dataset, comprising numerical and categorical
needs, either by fine-tuning or using them as feature genera- values, and transform it into textual representation. Specifi-
tors. These models benefit from the textual form and sequential cally, we added context to the data by incorporating column
nature of their security-related data, including sources such names and concatenating them with their respective values.
as code, emails, and log sequences. These studies effectively Each new value is hashed and combined with other hashed
utilize BERT’s ability to comprehend contextual relationships values within the same instance, resulting in the generation
within sequences to carry out precise detection and classifica- of a sequence. This sequence offers a fixed-length representa-
tion tasks. However, to the best of our knowledge, no study tion of network data while maintaining privacy. By applying
has explored building a BERT model from scratch specifically this textual representation mechanism, we introduce a novel
for network-based cyber threat detection. This is a significant encoding technique called Fixed-Length Language Encoding
consideration given the unique nature of the input data and (FLLE), which can be described as follows:
attributes in this field, such as the decreased importance of FLLE description: Let use define a DataFrame denoted by
syntactic structures in content identification. Df as a matrix with i rows and j columns. Here, Df [i, j]
represents the matrix element at the intersection of the ith row
III. S ECURITY LLM MODEL and jth column in DataFrame Df . We denote the ith row of
This section presents the important steps of SecurityLLM, Df by ri = Df [i, :]. In Df , the first row contains the column
which comprises two key elements: SecurityBERT and Fal- names, which serve as labels or identifiers for each column.
 A Pre-trained Language Model (SecurityBERT) FalconLLM 40B

Text Data Input (FLLE output) BERT Embedding Load Falcon LLM
Original Network Traffic data

BERT Encoder Prompt Design for incident

Response and Recovery
Feature extraction ByteLevelBPETokenizer

Contextuel Representation
Prompt FalconLLM
Textual Representation
using FLLE Tokenized Trafic Data

Softmax Classifier

Features Category
Return results
1AE4Fd067EC23AB Ransomware dbaf efb 447 ejf
Normal DDoS XSS
AEC89021D56FA231 Normal Token4 Token3 Token2 Token1

1) Dataset Preparation 2) Dataset Tokenization 3) Model Training 4) Incident Response and Recovery

Fig. 1: Workflow of our SecurityLLM: Leveraging Contextualized Text Representations for Accurate Semantic Analysis

We denote these column names as cj , where j represents the

column index, i.e., Df [1, j] = cj . TEST ID ENC

Let us define s(i, j) as a concatenation operation where the 51 6 72

column name cj , a dollar sign, and the value of the column
jth in the (i+1)th row ri are concatenated into a single string, 13 9 8

excluding the first row which contains the column names i.e.,

H(TEST$51) H(ID$6) H(ENC$72)

s(i, j) = cj || ”$” || Df [i + 1, j] (1)
=
H(TEST$13) H(ID$9) H(ENC$8)
Next, define H(x) as a hashing operation on a string x and
let L be a list where each element of the list is separated by
a space, i.e., L = {l1 l2 l3 . . . lk }. Fig. 2: Creating DataList example
Then, the textual representation of each row i in the
DataFrame Df can be expressed as follows: Algorithm 1 Fixed-Length Language Encoding
Require: DataFrame Df with i rows and j columns
L ← L ∪ {H(s(i, n))} ∀ (1 ≤ n ≤ j) (2) 1: procedure FLLE(Df )
By repeating this procedure for each row, we obtain a new 2: DL ← [] ▷ Initialize DL to be an empty list
DataList denoted as DL. In DL, each row represents an L 3: for m = 1 to i do ▷ Iterate through each row in Df
list, i.e.; 4: L = [] ▷ Initialize L to be an empty list
5: for n = 1 to j do ▷ Iterate through each column
  6: L ← H(s(m, n)) ▷ Append H(s(m, n)) to L
L1 = [H(s(1, 1)) H(s(1, 2)) . . . H(s(1, j))] 7: end for
 L2 = [H(s(2, 1)) H(s(2, 2)) . . . H(s(2, j))]
DL ← L ▷ Append L to DL

8:
DL = 
 
..
end for

 .  9:
Li = [H(s(i, 1)) H(s(i, 2)) . . . H(s(i, j))] 10: return DL
11: end procedure
In other words, the DataList DL is a list of lists where each
inner list contains the hashed, concatenated column values for
each row in the DataFrame. The FFLE algorithm can be seen
in Algorithm (1). Figure 2 showcases an example of a simple ByteLevelBPETokenizer is particularly useful for handling
DataList creation. out-of-vocabulary (OOV) words, which are not present in the
4) Byte-level BPE (BBPE) Tokenizer: We adopt the tokenizer’s vocabulary of human language [14]. By breaking
ByteLevelBPETokenizer, a tokenizer class in the Hugging down our language presentation of network traffic data into
Face Transformers library for tokenizing text into subword smaller subwords likely present in the tokenizer’s vocabulary
units first used for GPT 2 [12]. It is based on the Byte- as a sequence of bytes, we can efficiently process traffic data
Pair Encoding (BPE) algorithm, a data compression tech- by leveraging the power BERT.
nique that replaces the most frequent pair of consecutive The "TrainByteLevelBPETokenizer" procedure (Algorithm
bytes in a sequence with a single, unused byte [13]. The 2) starts by defining the directory path for text files. If the
directory doesn’t exist, it’s created. Next, an instance of • attention_masks_eval: These are used to inform the
the ByteLevelBPETokenizer is initialized. A new file in the model about which tokens should be attended to, and
directory above is opened for writing, and each row from the which should not. In many cases, sequences are padded
input text_data is written to this file, each row ending with with special tokens to make all sequences the same length
a new line character. After all the text data is written, the file for batch processing. Attention masks prevent the model
is closed. The procedure then defines the vocabulary size as from attending to these padding tokens. Typically, an
5000 and a list of special tokens. Lastly, the tokenizer is trained attention mask has the same length as the corresponding
using the file name, vocabulary size, minimum frequency of input_ids sequence and contains a 1 for real tokens and
2, and special tokens list. a 0 for padding tokens.

Algorithm 2 ByteLevelBPETokenizer Training

Algorithm 3 Encode Evaluation Data Sequences
1: procedure T RAIN B YTE L EVEL BPET OK -
ENIZER (text_data)
1: chunk_size ← 5000
2: txt_f iles_dir ← ”content/tmp/text_split” 2: num_chunks ← ⌈ len(eval_data)
chunk_size ⌉
3: create directory txt_f iles_dir 3: Initialize input_ids_eval, attention_masks_eval as
4: tokenizer ← new ByteLevelBP ET okenizer empty lists
5: f ile_name ← join txt_f iles_dir and 4: for i = 0 to num_chunks do
′
text_data.txt′ 5: start_idx ← i × chunk_size
6: open f ile_name for writing as file 6: end_idx ← (i + 1) × chunk_size
7: for each row in text_data[′ text′ ] do 7: chunk ← eval_data[start_idx : end_idx]
8: row ← row + newline 8: Encode each sequence in chunk as encoded_seqs
9: write row to f ile 9: Unpack input_ids_chunk,
10: end for attention_masks_chunk from encoded_seqs
11: close f ile 10: Add input_ids_chunk, attention_masks_chunk to
12: vocab_size ← 5000 input_ids_eval, attention_masks_eval
13: special_tokens ← ["<s>", "<pad>", "</s>", "<unk>", 11: end for
"<mask>"] 12: Concatenate the input IDs and attention masks as
14: train tokenizer with file_name, vocab_size, minimum input_ids_eval, attention_masks_eval
frequency 2, and special_tokens
15: end procedure
6) Contextual representation: We adopted the BERT archi-
tecture, which leverages transformers for textual representation
5) SecurityBERT embedding: The SecurityBERT embed- and cyber threat classification. Specifically, we pre-trained
ding is presented by the Algorithm 3. The algorithm starts SecurityBERT using our tokenized dataset. In this process,
by setting the chunk_size to 5000. It then calculates the SecurityBERT takes each token from the tokenized text and
number of chunks, num_chunks, by dividing the length represents it as an embedding vector, denoted as X ∈ Rd ,
of the eval_data by chunk_size, and rounding up to where d represents the dimensionality of the embedding space.
the nearest integer. Two empty lists, input_ids_eval and Then SecurityBert utilizes a transformer-based architecture
attention_masks_eval, are initialized to hold the encoded consisting of multiple encoder layers. Each encoder layer
input IDs and attention masks, respectively. The algorithm comprises multi-head self-attention mechanisms and position-
then enters a loop, iterating from 0 to num_chunks. wise feed-forward neural networks. The self-attention mech-
This loop determines the start and end indices for each anism [15] allows the model to capture dependencies and
chunk of the eval_data. It retrieves a chunk of data us- relationships between different words within a sentence, facili-
ing these indices and encodes each sequence in the chunk, tating contextual understanding. The self-attention mechanism
storing the result in encoded_seqs. This encoded data is in BERT can be mathematically expressed as follows:
then unpacked into two components: input_ids_chunk and
attention_masks_chunk. These components are appended to
QK T
 
the input_ids_eval and attention_masks_eval lists. Once Attention(Q, K, V ) = sof tmax √ V (3)
all chunks have been processed, the algorithm concatenates dk
all the input IDs and attention masks in input_ids_eval
and attention_masks_eval respectively, along dimension where Q, K, and V are the query, key, and value matrices,
0, thereby creating a complete set of input IDs and at- respectively, and dk represents the dimensionality of the keys
tention masks for the evaluation data. The input_ids_eval vector.
and attention_masks_eval are important components of the Through self-attention, BERT encodes contextual represen-
input to transformer-based models. tations by capturing the importance of different words within a
• input_ids_eval: This is a sequence of integers repre- sentence based on their semantic relationships and syntactic re-
senting the input data after being tokenized. Each integer lationships. The resulting contextual embeddings are obtained
maps to a token in the model’s vocabulary. through feed-forward operations and layer normalization.
 Cyber Threats detected by SecurityBERT of all documents D. This is to remove rare words that
might not provide much informational value for further
SecurityBERT has detected cyber threats, namely, Ransomware attacks processing or model training.
and DDoS attacks. Proposes an incident response and recovery system.
• Vocabulary Creation:
FalconLLM model incident response and recovery output v(D, V ) = {w|w ∈ D, rank(w, D) ≤ V } (7)
As an AI language model, I can suggest some incident response and
recovery systems to protect against Ransomware attacks, DDoS attacks,
Here, the function v(D, V ) creates a vocabulary by
and other cyber threats. Here are some best practices: choosing the top V words w from the set of all documents
1- Regularly backup data: This is the most important step in incident D based on their frequency rank rank(w, D). This forms
response. Back up your data to an external hard drive, cloud storage, or
both. This ensures that your data is always safe and protected, even if a
the vocabulary that the model will recognize.
cyber-attack occurs. • Special Token Addition:
2- Train employees: Train employees on how to detect and respond
to cyber-attacks. Educate them on how to spot phishing emails and
suspicious links. Encourage them to report suspicious activity to the IT
v′ = v ∪ S (8)
or security teams.
3- Use firewalls and antivirus software: Use firewalls and antivirus This states that the new vocabulary, v ′ , is a union of
software to protect your network and devices from cyber-attacks. This the original vocabulary v and the set of special tokens
software can detect and block malicious traffic and malware. S. These special tokens typically include markers for the
4- Implement a disaster recovery plan: Implement a disaster recovery plan
that outlines how to respond to a cyber-attack. This plan should include start and end of sentences, unknown words, padding, etc.,
steps to isolate the affected systems, restore data, and get the business and are essential for certain SecurityBERT operations.
back up and running. • Tokenizer Training:
5- Monitor network activity: Monitor network activity for suspicious
traffic or unusual patterns. Use network monitoring tools to detect Td = map(v ′ ) (9)
potential threats and take action before they become a bigger problem.
6- Use two-factor authentication: Use two-factor authentication to secure Finally, the function map(v ′ ) trains the tokenizer Td .
your accounts and devices. This makes it harder for attackers to access
your data and systems. The trained tokenizer now maps future text inputs to the
established vocabulary v ′ , effectively turning unstructured
text into a form that the SecurityBERT can process. The
Fig. 3: Incident response and recovery example generated by trained tokenizer Td can take any text segment from the
FalconLLM (Example 1) document ′ d′ and turn it into a series of tokens that
SecurityBERT can understand.
7) Training SecurityBERT: The training of SecurityBERT These steps allow us to transform the raw text into a numer-
involves several crucial steps, each carefully calibrated to ical representation that SecurityBERT can process effectively.
ensure optimal performance in security-centric tasks. These 8) Cyber threat detection: After the pre-training stage,
steps include data collection and preprocessing, tokenizer we fine-tuned SecurityBERT for the cyber threat detection
training, model configuration, and the training process itself. classification task. We added one linear layer followed by a
The tokenizer for SecurityBERT is trained through a series of Softmax activation function on top of the pre-trained Securi-
steps, which can be mathematically represented as follows: tyBERT model, and the entire network is fine-tuned using our
• Text Normalization: labeled data. This process enables SecurityBERT to adapt its
learned contextual representations to the specific requirements
n(D) = {n(d)|d ∈ D} (4)
of threat detection, leading to improved performance.
In this function, n(D) represents the normalization pro-
B. FalconLLM model
cess applied to each document d in the set of all doc-
uments D. Text normalization typically involves con- After the detection of cyber threats using SecurityBERT
verting all text to lowercase, removing punctuation, and model, we adopt FalconLLM 1 as an incident response and
sometimes even stemming or lemmatizing words (reduc- recovery system. FalconLLM is an open-source large language
ing them to their root form). model on Hugging Face 2 and has dominated the top place
• Tokenization: on the Open Source LLM Leaderboard (15/06/2023). Falcon-
LLM is an autoregressive decoder-only model with 40 billion
t(d) = {t(w)|w ∈ d, d ∈ D} (5) parameters that was trained on a whopping 1 trillion tokens.
The tokenization function t(d) breaks down each docu- The training process carried out over two months, utilized
ment d in the set D into its constituent words or tokens 384 GPUs on AWS. The data used for pretraining Falcon
w. These tokens are the basic units of text that a machine- was gathered from public web crawls, forming the basis of
learning model can understand and process. its pretraining dataset, so-called RefinedWeb (RW).
• Frequency Filtering: FalconLLM’s sophisticated language understanding capabil-
ities can be harnessed to analyze and interpret complex threat
f (D, F ) = {w ∈ D|freq(w, D) ≥ F } (6) data from various sources. It can read through and comprehend
vast amounts of textual logs from security systems, incident
This function f (D, F ), defines a High-Pass filter, cut-
ting out tokens w that have a frequency of occurrence, 1 https://falconllm.tii.ae/

freq(w, D) less than the minimum frequency F in the set 2 https://huggingface.co/tiiuae/falcon-40b-instruct

reports, and other sources, extracting relevant information and Cyber Threats detected by SecurityBERT
identifying patterns that may signal a cyber threat. Once a
potential incident has been identified, FalconLLM can help SecurityBERT has detected Port Scanning attacks. Proposes security
policies and procedures, and provide guidance on best practices for data
determine the threat’s severity and potential impact by com- protection, password management, and social engineering awareness.
paring it against its vast internal knowledge database. Then,
it can propose potential mitigation strategies and recovery FalconLLM model incident response and recovery output
procedures, guiding the response team through a carefully
Port Scanning attacks are security breach where an attacker tries to
outlined step-by-step process. FalconLLM’s ability to continu- discover open ports on a target system. This can be done by sending
ously learn from new data can also help improve the response data packets to the target system and analyzing the responses to determine
process. It can retrospectively analyze past incidents, identify which ports are open. To prevent Port Scanning attacks, it is recommended
to regularly update software and security patches, restrict access to ports
gaps in response procedures, and recommend adjustments. that are not being used, and monitor network activity for suspicious
This proactive and iterative approach allows for a faster, more activity. Additionally, it is important to have strong passwords and
efficient incident response and recovery process, minimizing regularly update them, as well as be cautious of social engineering tactics
such as phishing emails. It is also recommended to have a security plan
potential damages caused by cyber threats.Figures 3, 4, 5, and and regularly review it to ensure it is up-to-date and effective.
6 present examples of how FalconLLM can be used as an
incident response and recovery system against cyber threats
detected by the SecurityBERT model. Fig. 6: Security policies and procedures generated by Falcon-
LLM (Example 4)
Cyber Threats detected by SecurityBERT
The functionality of FalconLLM can be conceptualized
SecurityBERT has detected a cyber threat in our industrial network: a mathematically by modeling its core components as variables.
Backdoor attack. Can you propose to me a mitigation solution? These essential elements in the FalconLLM process flow are
outlined below:
FalconLLM model incident response and recovery output
• Data Inputs (DI): This signifies the unprocessed textual
Yes, I can. There are a few steps that can be taken to mitigate the logs derived from security systems, incident reports, and
Backdoor threat. Firstly, it’s important to identify the source of the attack.
This could be done by monitoring the network traffic and looking for any various other data sources, which FalconLLM subjects to
suspicious activity. Once the source has been identified, it’s important to analysis.
block any further attempts to access the network. This can be done by • Threat Detection (TD): This step extracts pertinent
implementing firewalls, intrusion detection systems, and other security
measures. Additionally, it’s important to update all software and patches information and identifies patterns that indicate a cyber
to ensure that the network is as secure as possible. It’s also a good idea threat.
to regularly audit the network for any potential vulnerabilities. • Threat Assessment (TA): This step evaluates a threat’s
severity and potential ramifications by contrasting it
Fig. 4: A mitigation solution generated by FalconLLM (Ex- against FalconLLM’s proprietary knowledge database.
ample 2) • Mitigation Strategy Proposal (MSP): This constitutes
the suggested countermeasures and recovery methods.
• Learning Process (LP): This encompasses FalconLLM’s
capability to learn and adapt from newly encountered data
Cyber Threats detected by SecurityBERT constantly.
• Incident Analysis and Improvement (IAI): This is
SecurityBERT has detected an uploading attack in real time. I want you
to design a security strategy in order to find these suspicious uploaded FalconLLM’s method of retrospectively analyzing past
files. incidents and making improvements.
• Damage Minimization (DM): The primary aim of Fal-
FalconLLM model incident response and recovery output
conLLM is to mitigate the potential impact of cyber
Based on the information provided, I can suggest the following security threats detected by the SecurityBERT model, thereby
strategy to detect suspicious uploaded files: enhancing overall cybersecurity.
1- Use a web application firewall to prevent unauthorized access to the
server. The mathematical abstraction of FalconLLM’s process can
2- Implement input validation to prevent SQL injection attacks and other be presented as follows:
attacks that can upload files.
3- Use file integrity monitoring tools to monitor changes to critical files
and directories. DM = f (DI, T D, T A, M SP, LP, IAI) (10)
4- Implement access control to restrict access to the server and sensitive
files. In this formula, f denotes the operation executed by Fal-
5- Regularly update software and applications to ensure they are secure
and up-to-date. conLLM.
6- Train employees to identify and report suspicious activity.
7- Regularly review security policies and procedures to ensure they are IV. P ERFORMANCE E VALUATION
effective.
In this section, we evaluate the performance of our proposed
model, SecurityLLM, through rigorous testing and compar-
Fig. 5: A security strategy generated by FalconLLM (Example ative analysis. We initiate the discussion by detailing the
3) configuration and the inherent parameters of the SecurityLLM,
which are optimized to ensure maximum performance and TABLE II: Data Distribution
efficiency. Then, we focus on the experimental results that Attack Type Nb. of Samples Train. Data Eval. Data
demonstrate the effectiveness of SecurityLLM in classifying Normal 1,615,643 1,292,514 323,129
DDoS_UDP 121,568 88,027 22,007
various network attack classes. DDoS_ICMP 116,436 93,149 23,287
SQL_injection 51,203 40,962 10,241
A. Configuration of SecurityLLM Password 50,153 40,122 10,031
Vulnerability_scanner 50,110 40,088 10,022
Table II presents the distribution of different types of DDoS_TCP 50,062 40,050 10,012
cyber attack samples across training and evaluation data sets. DDoS_HTTP 49,911 39,929 9,982
Uploading 37,634 30,107 7,527
The SecurityLLM model, as presented in Table I, is config- Backdoor 24,862 19,890 4,972
ured with several unique parameters designed to optimize its Port_Scanning 22,564 18,051 4,513
performance and functionality. The model uses a Byte-Pair XSS 15,915 12,732 3,183
Ransomware 10,925 8,740 2,185
Encoding Tokenizer, which provides a reliable and effective MITM 1,214 320 80
means of splitting input data into manageable tokens. The Fingerprinting 1,001 801 200
training data utilized by the model amounts to 661,767,168 Max Count 1,615,643 1,292,514 323,129
tokens, with a limited vocabulary size of 5000. The minimum
token frequency for SecurityLLM is set at 2, while the model
supports a maximum sequence length of 737 and a minimum
sequence length of 619. The truncation settings limit the
sequence length to a maximum 512, ensuring data consistency
and model stability. Special tokens used by the model include
<s>, <pad>, </s>, <unk>, and <mask>. Regarding processing
power, SecurityLLM works with a batch size of 128 and a
hidden size of 128. The model architecture comprises two
hidden layers and utilizes four attention heads to process
the input data. The intermediate size is set at 512, and
the maximum position embeddings at 512, providing enough
room for extensive and complex computations. The model can
identify and respond to 14 different attacks, demonstrating Fig. 7: Accuracy and loss history of SecurityLLM training in
its versatility and broad applicability. The SecurityLLM is 4 epochs
based on 11,174,415 parameters that are fine-tuned for optimal
performance. Lastly, the model runs on an Nvidia A100 TABLE III: Classification Report of SecurityLLM
GPU, a powerful hardware accelerator that enables rapid data Class Precision Recall F1-Score Support
processing and real-time response capabilities. Normal 1.00 1.00 1.00 323,129
DDoS_UDP 1.00 1.00 1.00 22,007
DDoS_ICMP 1.00 1.00 1.00 23,287
TABLE I: Configuration of SecurityLLM SQL_injection 0.85 0.83 0.84 10,241
Parameter Value Password 0.85 0.81 0.83 10,031
DDoS_TCP 1.00 1.00 1.00 10,012
Tokenizer type Byte-Pair Encoding Tokenizer
DDoS_HTTP 0.89 0.99 0.94 9,982
Training data 661,767,168 tokens Vul_scanner 1.00 0.94 0.97 10,022
Vocabulary size 5000 Uploading 0.79 0.86 0.83 7,527
Minimum token frequency 2 Backdoor 0.82 0.94 0.87 4,972
Maximum sequence length 737 Port_Scanning 0.87 1.00 0.93 4,513
Minimum sequence length 619 XSS 0.94 0.76 0.84 3,183
Truncation settings Max length=512 Ransomware 1.00 0.40 0.57 2,185
Special tokens <s>, <pad>, </s>, <unk>,<mask> Fingerprinting 0.00 0.00 0.00 200
Batch size 128 MITM 1.00 1.00 1.00 80
Hidden size 128 Macro Avg 0.87 0.84 0.84 441,371
Number of hidden layers 2 Weighted Avg 0.98 0.98 0.98 441,371
Number of attention heads 4 Accuracy 0.98
Intermediate-size 512
Maximum position embeddings 512
Number of attacks 14
Total number of parameters 11,174,415 epochs. A visual depiction of the confusion matrix from the
Hardware accelerator GPU Nvidia A100 SecurityLLM classification is presented in Figure 8.
For the ’Normal’ class and most types of DDoS attacks,
including ’DDoS_UDP’, ’DDoS_ICMP’, and ’DDoS_TCP’,
B. Experimental Results the model achieved perfect scores in terms of precision,
Table III presents the detailed classification report for the recall, and F1-score, showing a high accurate classification
SecurityLLM model. Precision, Recall, F1-Score, and Support performance on these types. It is noteworthy to mention the
measurements evaluated the model’s performance on various high support count for the ’Normal’ class, which amounts
network attack classes. Figure 7 shows the accuracy and loss to 323,129 instances. The performance on ’SQL_injection’,
history during the SecurityLLM training changes over four ’Password’, ’DDoS_HTTP’, ’Uploading’, ’Backdoor’, and
 high recall of 0.99 and 1.00 respectively, indicating that the
model could identify almost all instances of these attacks
when they occur. ’Vul_scanner’ had a high precision and a
slightly lower recall of 0.94, resulting in an F1-score of 0.97,
showing good performance in identifying this type of attack.
’Ransomware’ showed a high precision of 1.00, but with a
significantly lower recall of 0.40, resulting in an F1-score of
0.57, suggesting that while the model made correct predictions
for the ’Ransomware’ class, it missed a significant portion of
actual instances. The classes ’XSS’ and ’MITM’ showed good
performance with F1-scores of 0.84 and 1.00, respectively,
demonstrating that the model handled these classes well.
Interestingly, the ’Fingerprinting’ class had a precision, recall,
and F1-score of 0, indicating a complete misclassification for
these instances by the model. This area requires future work
for improvement.
The average recall and F1-score were all 0.84 on the macro
Fig. 8: Confusion matrix of SecurityLLM classification level. The weighted average was considerably higher at 0.98
for all three metrics, suggesting a good performance overall.
TABLE IV: Comparison of SecurityLLM with traditional ML The slight difference between these two averages may be due
and Deep learning models to the imbalanced nature of the dataset, as classes with larger
AI type Security AI Model Accuracy support have a greater influence on the weighted average. The
DT 67% overall accuracy of the model, measuring the proportion of
RF 81% correct predictions made out of all predictions, was found to
Traditional ML
SVM 78%
KNN 79%
be 0.98, showing a high degree of the predictive power of the
CNN 95% SecurityLLM model in identifying different types of network
Deep learning model
RNN 94% attacks.
DNN 93%
Transformer model 95% Table IV presents the comparative accuracy of the proposed
Large language model SecurityLLM 98% SecurityLLM, namely SecurityLLM, against the traditional
DT: Decision Tree, RF: Random Forest, SVM: Support Vector Machines, Machine Learning (ML) models and Deep Learning (DL)
KNN: K-Nearest Neighbors, CNN: Convolutional Neural Network, RNN:
Recurrent Neural Network, and DNN: Deep Neural Network. models. Traditional ML models include Decision Tree (DT),
Random Forest (RF), Support Vector Machines (SVM), and
TABLE V: Comparison with recent works on cyber threat K-Nearest Neighbors (KNN). On the other hand, the Deep
detection Learning models comprise Convolutional Neural Network
(CNN), Recurrent Neural Network (RNN), Deep Neural Net-
Frameworks Year Detect LLM Recovery PCAP
Rahali et al. [5] 2022 ✗ ✓ ✗ ✗
work (DNN), and Transformer models. Among traditional ML
Aghaei et al. [9] 2022 ✗ ✓ ✗ ✗ models, RF exhibits the highest accuracy of 81%, followed
Thapa et al. [3] 2022 ✗ ✓ ✗ ✗ by KNN and SVM with respective accuracies of 79% and
Hamouda et al. [16] 2022 ✓ ✗ ✗ ✓ 78%. DT shows the lowest accuracy among traditional ML
Friha et al. [17] 2022 ✓ ✗ ✗ ✓
Alkhatib et al. [8] 2022 ✗ ✓ ✗ ✗ models at 67%. When we compare these models with Deep
Chen et al. [7] 2022 ✗ ✓ ✗ ✗ Learning models, a substantial improvement in accuracy can
Seyyar et al. [6] 2022 ✗ ✓ ✗ ✗ be observed. CNN and Transformer models both exhibit a
Selvaraja et al. [18] 2023 ✓ ✗ ✗ ✓
Chen et al. [19] 2023 ✗ ✓ ✗ ✗
top accuracy of 95%. RNN and DNN follow closely with
Douiba et al. [20] 2023 ✓ ✗ ✗ ✓ accuracies of 94% and 93%, respectively. The most noteworthy
Jahangir et al. [21] 2023 ✓ ✗ ✗ ✓ observation, however, emerges from the SecurityLLM model,
Hu et al. [22] 2023 ✓ ✗ ✗ ✓ a Large Language Model variant. SecurityLLM achieves an
Friha et al. [23] 2023 ✓ ✗ ✗ ✓
Chakraborty et al. [24] 2023 ✓ ✗ ✗ ✓
accuracy of 98%, significantly surpassing all traditional ML
Wang et al. [25] 2023 ✓ ✗ ✗ ✓ and Deep Learning models in the comparison. This exceptional
Liu et al. [26] 2023 ✓ ✗ ✗ ✓ result underlines the efficacy of SecurityLLM for the intended
Aouedi et al. [27] 2023 ✓ ✗ ✗ ✓ application, confirming its superiority over traditional ML and
Our Work 2023 ✓ ✓ ✓ ✓
Detect : Cyber Threat Detection, LLM: Large Language Model, Recovery: advanced Deep Learning techniques regarding model accu-
Incident Response and Recovery, PCAP: Packet data of a network racy. The superior performance of SecurityLLM suggests that
✓: Supported, ✗: Not Supported. Large Language Models can significantly improve accuracy
for security-related tasks.
Table V provides a comparison of various recent works on
’Port_Scanning’ classes was relatively lower but still com- cyber threat detection in terms of four key parameters: Cy-
mendable, with F1-scores ranging from 0.83 to 0.94. Notably, ber Threat Detection (Detect), utilization of Large Language
’DDoS_HTTP’ and ’Port_Scanning’ achieved a remarkably Models (LLM), Incident Response and Recovery (Recovery),
and usage of Packet data of a network (PCAP). The majority [3] C. Thapa, S. I. Jang, M. E. Ahmed, S. Camtepe, J. Pieprzyk, and
of the research conducted in 2022, including [5], [9], and [8], S. Nepal, “Transformer-based language models for software vulnera-
bility detection,” in Proceedings of the 38th Annual Computer Security
integrated LLMs, but they did not support detection, recovery, Applications Conference, 2022, pp. 481–496.
nor did they utilize packet data. Nonetheless, [16] and [17] [4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training
defied this norm by supporting cyber threat detection and of deep bidirectional transformers for language understanding,” arXiv
preprint arXiv:1810.04805, 2018.
leveraging packet data. On the other hand, the works in 2023, [5] A. Rahali and M. A. Akhloufi, “Malbert: Malware detection using
such as [18], [20], and [25], emphasize more on cyber threat bidirectional encoder representations from transformers*,” 2021 IEEE
detection and the use of packet data, but they largely lack in International Conference on Systems, Man, and Cybernetics (SMC), pp.
3226–3231, 2021.
the application of large language models and do not provide [6] Y. E. Seyyar, A. G. Yavuz, and H. M. Ünver, “An attack detection
incident response and recovery solutions. This shift in focus framework based on bert and deep learning,” IEEE Access, vol. 10,
suggests a growing emphasis on threat detection and data pp. 68 633–68 644, 2022.
[7] S. Chen and H. Liao, “Bert-log: Anomaly detection for system logs
packet utilization, potentially sidelining the usage of language based on pre-trained language model,” Applied Artificial Intelligence,
models and incident recovery. However, our work investigated vol. 36, 2022.
the implementation of SecurityLLM from scratch and supports [8] N. Alkhatib, M. Mushtaq, H. Ghauch, and J.-L. Danger, “Can-bert do
it? controller area network intrusion detection system based on bert
all four components: cyber threat detection, large language language model,” in 2022 IEEE/ACS 19th International Conference on
models, incident response and recovery, and packet data of Computer Systems and Applications (AICCSA). IEEE, 2022, pp. 1–8.
a network. This suggests a more comprehensive and holistic [9] E. Aghaei, X. Niu, W. Shadid, and E. Al-Shaer, “Securebert: A domain-
specific language model for cybersecurity,” in International Conference
approach to cyber threat detection and response, potentially on Security and Privacy in Communication Systems. Springer, 2022,
setting a new standard for future research and developments pp. 39–56.
in this field. [10] M. A. Ferrag, L. Maglaras, S. Moschoyiannis, and H. Janicke, “Deep
learning for cyber security intrusion detection: Approaches, datasets, and
comparative study,” Journal of Information Security and Applications,
V. C ONCLUSION vol. 50, p. 102419, 2020.
[11] M. A. Ferrag, O. Friha, D. Hamouda, L. Maglaras, and H. Janicke,
This paper showcases the exceptional potential of LLM in “Edge-iiotset: A new comprehensive realistic cyber security dataset of
cybersecurity, specifically through implementing SecurityLLM iot and iiot applications for centralized and federated learning,” IEEE
Access, vol. 10, pp. 40 281–40 306, 2022.
for threat detection and incident response. The innovative ap- [12] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi,
plication of BERT architecture for cyber threat detection, em- P. Cistac, T. Rault, R. Louf, M. Funtowicz et al., “Huggingface’s trans-
bodied in SecurityBERT, demonstrated remarkable efficiency, formers: State-of-the-art natural language processing,” arXiv preprint
arXiv:1910.03771, 2019.
contradicting initial assumptions regarding its incompatibility [13] Y. Shibata, T. Kida, S. Fukamachi, M. Takeda, A. Shinohara, T. Shino-
due to the reduced significance of syntactic structures. More- hara, and S. Arikawa, “Byte pair encoding: A text compression scheme
over, the incorporation of FalconLLM resulted in a sophis- that accelerates pattern matching,” 1999.
ticated incident response and recovery system. Experimental [14] A. Araabi, C. Monz, and V. Niculae, “How effective is byte pair
encoding for out-of-vocabulary words in neural machine translation?”
results underscored the superiority of this approach over con- arXiv preprint arXiv:2208.05225, 2022.
ventional machine learning and deep learning models, includ- [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
ing convolutional neural networks, deep learning networks, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in
neural information processing systems, vol. 30, 2017.
and recurrent neural networks. The SecurityLLM model, tested [16] D. Hamouda, M. A. Ferrag, N. Benhamida, and H. Seridi, “Ppss: A
on a collected cybersecurity dataset, exhibited an outstanding privacy-preserving secure framework using blockchain-enabled feder-
capability to identify fourteen distinct types of attacks with an ated deep learning for industrial iots,” Pervasive and Mobile Computing,
p. 101738, 2022.
accuracy rate of 98%, thereby highlighting the transformative [17] O. Friha, M. A. Ferrag, L. Shu, L. Maglaras, K.-K. R. Choo, and
potential of LLMs in cybersecurity applications. M. Nafaa, “Felids: Federated learning-based intrusion detection system
for agricultural internet of things,” Journal of Parallel and Distributed
While this paper has made significant progress in advancing Computing, vol. 165, pp. 17–31, 2022.
the use of LLMs in cybersecurity, future research directions [18] S. Selvarajan, G. Srivastava, A. O. Khadidos, A. O. Khadidos, M. Baza,
can take several routes to further enhance these promising A. Alshehri, and J. C.-W. Lin, “An artificial intelligence lightweight
findings. One potential avenue involves fine-tuning and ex- blockchain security model for security and privacy in iiot systems,”
Journal of Cloud Computing, vol. 12, no. 1, p. 38, 2023.
panding the SecurityLLM model to enhance its performance [19] Y. Chen, Z. Ding, X. Chen, and D. Wagner, “Diversevul: A new
across various attack types, incorporating adversarial attacks vulnerable source code dataset for deep learning based vulnerability
and more complex threats. In addition, due to the evolving detection,” arXiv preprint arXiv:2304.00409, 2023.
[20] M. Douiba, S. Benkirane, A. Guezzaz, and M. Azrour, “An improved
nature of cyber threats, continuous updating and training of anomaly detection model for iot security using decision tree and gradient
SecurityLLM model on the latest real-world datasets will be boosting,” The Journal of Supercomputing, vol. 79, no. 3, pp. 3392–
imperative to maintain its efficacy. 3411, 2023.
[21] H. Jahangir, S. Lakshminarayana, C. Maple, and G. Epiphaniou, “A deep
learning-based solution for securing the power grid against load altering
R EFERENCES threats by iot-enabled devices,” IEEE Internet of Things Journal, 2023.
[22] F. Hu, W. Zhou, K. Liao, H. Li, and D. Tong, “Towards federated
[1] N. Moustafa, N. Koroniotis, M. Keshk, A. Y. Zomaya, and Z. Tari, learning models resistant to adversarial attacks,” IEEE Internet of Things
“Explainable intrusion detection for cyber defences in the internet of Journal, 2023.
things: Opportunities and solutions,” IEEE Communications Surveys & [23] O. Friha, M. A. Ferrag, M. Benbouzid, T. Berghout, B. Kantarci, and K.-
Tutorials, 2023. K. R. Choo, “2df-ids: Decentralized and differentially private federated
[2] S. Silvestri, S. Islam, S. Papastergiou, C. Tzagkarakis, and M. Ciampi, learning-based intrusion detection system for industrial iot,” Computers
“A machine learning approach for the nlp-based analysis of cyber threats & Security, p. 103097, 2023.
and vulnerabilities of the healthcare ecosystem,” Sensors, vol. 23, no. 2, [24] C. Chakraborty, S. M. Nagarajan, G. G. Devarajan, T. Ramana, and
p. 651, 2023. R. Mohanty, “Intelligent ai-based healthcare cyber security system using
 multi-source transfer learning method,” ACM Transactions on Sensor
Networks, 2023.
[25] X. Wang, C. Fang, M. Yang, X. Wu, H. Zhang, and P. Cheng, “Resilient
distributed classification learning against label flipping attack: An admm-
based approach,” IEEE Internet of Things Journal, 2023.
[26] J. Liu, Y. Tang, H. Zhao, X. Wang, F. Li, and J. Zhang, “Cps attack
detection under limited local information in cyber security: An ensemble
multi-node multi-class classification approach,” ACM Transactions on
Sensor Networks, 2023.
[27] O. Aouedi and K. Piamrat, “F-bids: Federated-blending based intrusion
detection system,” Pervasive and Mobile Computing, p. 101750, 2023.