custombox[Prompt][] custombox1[Example][]

# MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented
Generation

Marco Simoni  Istituto di Informatica e Telematica, Consiglio Nazionale Delle
Ricerche, Pisa, Italy  Andrea Saracino  TeCIP, Scuola Universitaria Superiore
Sant’Anna, Pisa, Italy  Vinod Puthuvath  University of Padua, Italy and Delft
University of Technology, Netherlands  Maurco Conti  University of Padua,
Italy and Delft University of Technology, Netherlands

###### Abstract

In this paper, we introduce MoRSE (Mixture of RAGs Security Experts), the
first specialised AI chatbot for cybersecurity. MoRSE aims to provide
comprehensive and complete knowledge about cybersecurity. MoRSE uses two RAG
(Retrieval Augmented Generation) systems designed to retrieve and organize
information from multidimensional cybersecurity contexts. MoRSE differs from
traditional RAGs by using parallel retrievers that work together to retrieve
semantically related information in different formats and structures. Unlike
traditional Large Language Models (LLMs) that rely on Parametric Knowledge
Bases, MoRSE retrieves relevant documents from Non-Parametric Knowledge Bases
in response to user queries. Subsequently, MoRSE uses this information to
generate accurate answers. In addition, MoRSE benefits from real-time updates
to its knowledge bases, enabling continuous knowledge enrichment without
retraining. We have evaluated the effectiveness of MoRSE against other state-
of-the-art LLMs, evaluating the system on 600 cybersecurity specific
questions. The experimental evaluation has shown that the improvement in terms
of relevance and correctness of the answer is more than 10% compared to known
solutions such as GPT-4 and Mixtral 7x8.

##  I Introduction

The increasing frequency and sophistication of new cyber threats have made
cybersecurity a critical priority across all sectors, with a 15% increase in
three years 111https://bit.ly/3zxkf2y on data breaches only. In recent years,
the amount of cybersecurity-related information has exploded, providing
important resources to protect against these threats by mitigating risk and
improving cybersecurity measures. However, this rapid proliferation of
information has led to a cluttered and often unstructured data landscape,
complicating the task of deriving actionable insights for professionals [1,
2]. In fact, a timely, accurate, and comprehensive understanding of
vulnerabilities, exploits, and defense tactics is crucial, as the promptness
of such information can significantly impact cybersecurity decisions [3, 4].
Recently, large language models (LLMs) have become important tools for
synthesizing huge amounts of information in various fields, including
cybersecurity [5]. However, their reliability varies for technical topics
where inaccuracies are critical [6, 7, 8, 9]. LLMs can produce hallucinatory
responses, meaning that they produce answers that are not true or reliable,
especially struggling with the dynamic and evolving nature of cyber threats
[10, 11, 12]. This problem is particularly pronounced in code generation
tasks, where LLMs often produce non-functioning code for complex queries [13].
Specifically, when the model does not know the correct answer, hallucinations
are inevitable (Epistemic [14]). This can happen if the model does not have
enough training data or if its capacity is limited. An example of
hallucination in cybersecurity is shown in Example I (ground truth) and
Example I (the answer of GPT-4) to the question: What is CVE-2017-5162?. This
shows that GPT-4 is not able to provide the correct answer. To overcome these
limitations, it is crucial to build continuous learning mechanisms into LLMs
that allow them to dynamically update their knowledge base with the latest
information [15]. Although retraining can be time-consuming, updating with new
information is essential to maintain the relevance of LLM responses [16, 17].
For this reason, companies are quickly discovering the value of Retrieval
Augmented Generation chatbots. Although RAG systems have made significant
advances on cybersecurity, researchers and practitioners often lacks
comprehensive models that unify diverse cybersecurity data and dynamic, real-
time updated systems that are essential for addressing the evolving cyber
threat landscape. In fact, effective analysis of threats, vulnerabilities and
exploits requires a thorough understanding of individual entities and their
broader interactions.

Ground Truth. CVE: CVE-2017-5162
Description: An issue was discovered in BINOM3 Universal Multifunctional
Electric Power Quality Meter. Lack of authentication for remote service gives
access to application set up and configuration.

GPT-4 0125-Preview Response. CVE: CVE-2017-5162
Description: CVE-2017-5162 is a vulnerability associated with Broadcom Wi-Fi
chipsets. It is one of the vulnerabilities part of the BroadPwn exploit.
BroadPwn is a critical security flaw that allows a remote attacker to
compromise a vulnerable device over the air, without any interaction from the
user.

##### Contribution

In this paper, we present MoRSE (Mixture of RAGs Security Experts), a novel
framework developed for Question and Answering tasks in the field of
cybersecurity. Unlike other chatbots MoRSE is the first RAG framework that
aims to provide comprehensive coverage of cybersecurity knowledge. To achieve
this, MoRSE uses two RAG systems in cascade. These two RAGs divide the process
into two distinct phases: Information Retrieval phase, which is managed by
Multiple Retrievers, followed by the answer generation phase, which is driven
by Large Language Models. The first RAG, called Structured RAG, comprises
retrievers that are tailored to fast retrieval tasks, as they retrieve
information from preprocessed, structured data. The second, Unstructured RAG,
is slower due to its higher complexity, but retrieves a larger amount of
information as it enables the exploration of data in its original form. The
second RAG is only activated if the first RAG does not find any relevant
documents for a specific query. Each retriever is specialized in a specific
area of cybersecurity and collects data from key resources such as
MITRE222https://attack.mitre.org/, CVE repositories333https://cve.mitre.org/,
Metasploit444https://www.metasploit.com/, and ExploitDB555https://www.exploit-
db.com/. The name of our system, MoRSE, is in fact inspired by the Mixture of
Experts (MoE) paradigm [18], reflecting the specialized skills of the
individual retrievers. In addition, MoRSE receives new knowledge through real-
time updates of its retrievers’ knowledge bases, which enables a continuous
expansion of knowledge. Indeed, a key advantage of non-parametric memory
models such as RAG is the ability to update knowledge at test time. In
contrast, Parametric-only models require the entire model to be retrained,
which can be both time-consuming and resource-intensive.

Using a comprehensive set of 600 cybersecurity questions, consisting of 150
General Cybersecurity questions, 150 Multi-Hop (i.e., questions involving
multiple entities contained in different documents) Cybersecurity questions,
and 300 CVE questions (on vulnerabilities), we evaluated MoRSE alongside other
commercial, well-known LLMs, including GPT-4666https://chat.openai.com/,
GEMINI777https://gemini.google.com/app, MIXTRAL [19], and
HACKERGPT888https://chat.hackerai.co/. The questions were classified based on
the Diamond Model [20], which we used to create questions that are
representative of the real needs of the cybersecurity world. Two experts
validated the ground truth of the questions, with a Cohen’s Kappa [21, 22]
index as a measure of agreement between the two experts equal to $0.82$ out of
1.

The comparative analysis shows the superior performance of MoRSE for
cybersecurity queries. In terms of relevance and correctness of answers [23],
MoRSE outperforms other models by more than $15\%$ for General questions and
by more than $10\%$ for Multi-Hop questions and CVE Questions. In terms of
accuracy, MoRSE outperforms GPT-4 by 50% for CVE Questions. This confirms its
effectiveness in specialized domains. We validated these results with the LLM
as a Judge method [24], which derives Elo ratings for each model and confirms
MoRSE’s leading performance compared to all competitors.

The main contributions of our work can be summarized as follows:

  * •

We introduce MoRSE, an open-source framework999https://github.com/Mixture-of-
RAGs-Security-Experts/MoRSE which is the first attempt to integrate two RAG
systems to handle multidimensional cybersecurity contexts. This architecture
enables a unique synthesis of different data sources and improves the depth
and relevance of security insights.

  * •

We introduce a three-part evaluation test suite that measures the relevance,
similarity and correctness of RAG systems in conjunction with LLMs. We have
further validated these results with two additional test suites based on the
LLM as a Judge approach. To the best of our knowledge, we are the first to
provide such contribution.

  * •

We demonstrate how MoRSE can exploit its unique real-time cybersecurity
keyword detection capability to improve the correctness of responses by 10%
compared to GPT-4, addressing the critical need for timely and accurate
security analysis.

  * •

MoRSE differs from conventional RAGs by using parallel retrievers that work
together to retrieve semantically related information in different formats and
structures. This is particularly important in the cybersecurity domain, where
different data types such as exploit code, TTP descriptions, CVEs and white
papers often exist for a particular threat but are rarely related to each
other. MoRSE exploits these parallel retrievers and LLMs to integrate related
information and provide comprehensive answers to queries.

##### Organisation

The rest of the article is structured as follows. Section II discusses
background information on LLMs and RAGs. Section III describes the MoRSE
architecture. Section IV describes the experiments performed to evaluate the
performance of MoRSE, including a comparative analysis with known commercial
models. Section V provides an overview of related research work that includes
various cybersecurity tools such as knowledge graphs, entity extraction tools,
chatbots, and cyber threat intelligence (CTI). The conclusions and future work
directions are presented in Section VI.

##  II background

This section provides an overview of the basic concepts necessary for
understanding the architecture of MoRSE.

###  II-A Large Language Models

Large language models (LLMs) represent a significant advancement in the field
of Natural Language Processing (NLP) and are based on the Transformer model
[25]. These models are trained on large text datasets and are able to generate
coherent and contextually relevant texts based on input prompts. The
capabilities of LLMs go beyond text generation and include tasks such as
language translation, summarizing, answering questions, etc.

The introduction of models such as GPT [26] and BERT [27] has demonstrated the
potential of LLMs to revolutionize language understanding and generation
through unsupervised and bidirectional training [26, 28]. With the development
of GPT-3 [29], the scalability of these models reached new heights,
illustrating their ability to perform a wide range of NLP tasks without task-
specific training.

Despite their advantages, LLMs face several challenges. Ethical
considerations, such as the spread of bias and misinformation, are a major
concern [30]. In addition, the environmental impact of training and operating
these computationally intensive models has raised questions about their
sustainability [31]. Efforts to overcome these challenges include research
into more efficient training methods and models that are able to understand
and generate texts with greater accuracy and less bias [32].

###  II-B Retrieval Augmented Generation

Retrieval-Augmented Generation (RAG) combines traditional language models with
external databases to improve natural language processing (NLP) tasks [33, 34,
35]. RAG models use a retriever to retrieve relevant information and a
generator to generate answers based on the retrieved information. This
improves accuracy and relevance, especially for domain-specific queries [36,
16].

RAG’s strengths lie in its ability to update knowledge bases without
retraining and customise components for specific tasks such as cybersecurity
[33, 34].

However, RAG struggles with latency and scalability issues, especially when
processing concurrent queries [33]. Despite these limitations, RAG remains a
versatile tool for a range of NLP applications, from chatbots to content
creation. Ongoing research focuses on optimizing retrieval mechanisms and
computational power [36, 16].

###  II-C Definitions

We report in the following a set of definitions which will be used in the rest
of the paper:

  * •

Retriever is a component that identifies and retrieves relevant information or
documents from a knowledge base. This process is essential to provide the
necessary context and content that LLM uses to generate accurate and informed
answers [35, 37].

  * •

Knowledge Base is a repository of information that the retriever accesses to
find relevant data or documents. This is fundamental to the system’s ability
to retrieve contextually relevant content, essential for generating well-
informed and accurate answers [38].

  * •

Embeddings are numerical representations of text that assign a low dimension
to a term. Within this context, embedding vectors of analogous terms exhibit
proximity, encapsulating semantic meaning. This facilitates the comparison
between queries and the knowledge base [39].

  * •

Context refers to the relevant information or data retrieved by the system,
which surrounds and informs a particular query. The model requires this
contextual information to formulate answers that are precise, comprehensive,
and directly linked to the content found in the knowledge base [40].

  * •

Prompt refers to the structured input that is created from the retrieved
context, which is then fed into the generative model. This prompt guides the
model in generating a coherent, contextually relevant response that directly
addresses the user’s request [41].

  * •

Semantic Similarity evaluates how closely the content of a user’s query
matches the information in the knowledge base, focusing on meaning rather than
word-for-word matching. This evaluation guarantees the relevance and accuracy
of the retrieved data and supports the generative model in creating
appropriate answers. In MoRSE, we use Cosine Similarity [42] to measure the
proximity of embedding vectors because it has a high correlation with human
judgment [43].

  * •

Multi-hop queries are defined as requests for information that necessitate
indirect reasoning over multiple pieces of interconnected data. They typically
arise in complex question-answering tasks where a single piece of evidence is
insufficient to resolve the query, and the system must hop across different
data points or documents to piece together a response.

##  III MoRSE Architecture

This section describes the structure of the MoRSE system in detail. It
explains the function of the individual components and how they interact to
process a request and generate a response. We used the Langchain framework
101010https://www.langchain.com/ to develop the MoRSE architecture. The Table
I contains the key to the symbols used in this explanation to facilitate the
understanding of the following discussion. The first two symbols, ($\alpha$)
and ($\beta$), are embedding models, while the last two, ($\gamma$) and
($\theta$), are Transformer models.

TABLE I: Mapping of Symbols to Values. Symbol | Value
---|---
$\alpha$ |  BAAI/bge-large-en [44]
$\beta$ |  BAAI/bge-large-en-v1.5 [45, 46, 44]
$\gamma$ |  valhalla/t5-base-e2e-qg [47]
$\theta$ |  dslim/bert-large-NER [27]

###  III-A MoRSE Overview

As shown in Figure 1, MoRSE consists of two main components: a graphical user
interface (GUI) and the MoRSE core. The GUI enables interaction with the user
by allowing the input of queries and displaying the answers in a structured
way111111https://github.com/Mixture-of-RAGs-Security-Experts/MoRSE.

The MoRSE core consists of three key components, which in turn manage the user
query and compose the answer:

  * •

Query Handling Module: This module performs the pre-processing of user queries
and specializes in the management of multi-hop queries and complex questions,
especially in the context of Common Vulnerability and Exposures (CVEs) and
Common Weakness Enumerations (CWEs).

  * •

Structured RAG: The first of the two RAGs is composed by retrievers that
retrieve information from pre-processed, structured data. The pre-processing
phase involves converting chunks of text from various sources that are part of
the knowledge base, such as academic papers and cybersecurity websites, into
well-defined structures. These structures are designed to contain generated
questions and contextualized entity descriptions that facilitate the precise
retrieval of information in response to user queries.

  * •

Unstructured RAG: This RAG is used if the structured RAG could not find a
suitable answer. It searches for information in unstructured and unprocessed
raw text that belongs to its knowledge base. Accessing unstructured data
allows the exploration of data in its original form without the limitations
imposed by preprocessing, thus providing a wider range of search options in
exchange for a higher response time. This type enables the exploration of data
in its original form without the restrictions imposed by preprocessing.

Figure 1: MoRSE Overview.

The RAGs will compose the answer to each query and return it to the GUI for
structured visualization. In the following we will detail the components of
the MoRSE Core.

###  III-B MoRSE Core

Figure 2: MoRSE Core Workflow.

##### MoRSE Core Workflow

Figure 2 shows the first stage of MoRSE Core process, starting with the Query
Handling model. This module converts the original query $x$ into an optimized
version $x^{*}$ (see subsection III-C). First, $x^{*}$ is forwarded to the
Structured RAG module for processing. The structured RAG path, denoted as
$\mathcal{S}$, begins with the Structured Retrievers, focused on high accuracy
and fast responses to efficiently process most queries. Their primary
function, $\mathcal{S}(x^{*})$, is to identify and retrieve information
pertinent to the query.

When activated, the structured retrieval process, which is executed via
$\mathcal{S}(x^{*})$, assigns a set of potentially relevant documents from a
predefined Knowledge Base to the query $x^{*}$. In particular,
$\mathcal{D}=\text{top-k}(\mathcal{S}(x^{*}))$ represents the selection of the
top $k$ documents that $\mathcal{S}$ considers most relevant for the query
based on a similarity score. If $\mathcal{D}$ is not empty
($|\mathcal{D}|>0$), this means that a relevant context has been found. The
workflow then proceeds to use this context and moves on to the next phase,
where the retrieved information ($\mathcal{D}$) is wrapped in a Prompt, which
is used by LLM to generate a response.

If the Structured Retrievers do not yield relevant documents
($|\mathcal{D}|=0$), the workflow moves to the unstructured path and calls the
Unstructured Retrievers, denoted as $\mathcal{U}$. At this stage,
$\mathcal{E}=\text{top-k}(\mathcal{U}(x^{*}))$ represents the set of documents
retrieved by $\mathcal{U}$, which are designed to process complex queries that
are not readily covered by structured data patterns. After a successful
retrieval of relevant information in one of the two ways — indicated by
$|\mathcal{D}|>0$ for structured retrieval or $|\mathcal{E}|>0$ for
unstructured retrieval —the Wrapper module integrates the acquired context and
generates a prompt for the Large Language Model (LLM). The LLM then performs
Answer Generation, creating a detailed response to the user’s question.

##### RAG Architecture

The RAG architecture of the MoRSE system, which is used in both the Structured
(III-D) and Unstructured RAG (III-E), follows the same underlying logic shown
in Figure 3. This architecture is divided into two parts:

  1. 1.

The Retrieval part, which consists of Parallel Retrievers used to collect
relevant information for the query.

  2. 2.

The Generation part, in which the Large Language Model (LLM) uses the context
provided in the Prompt to generate responses. After the retrieval phase, the
collected information (info $1$ to info $N$) is merged into a Context, which
is Wrapped, along with the user query, in a Prompt used by LLM to generate the
Answer. The logic of the architecture is formalized in the Algorithm 1.

Figure 3: RAG Architecture Overview. Algorithm 1 RAG with $N$ Parallel
Retrievers

1:User query $Q$

2:Answer $A$

3:Predefined: Retrievers $r_{1},r_{2},\ldots,r_{N}$

4:procedure ExecuteRAG($Q$)

5: Initialize context set $C\leftarrow\emptyset$

6: for $i\leftarrow 1$ to $N$ do $\triangleright$ In parallel for each $i$-th
retriever

7: $I_{i}\leftarrow r_{i}(Q)$ $\triangleright$ Retrieve information using
$i$-th retriever

8: Sort $I_{i}$ by relevance scores to find the most pertinent documents

9: $I_{\text{top}}\leftarrow\text{top}_{k}(I_{i})$ $\triangleright$ Select the
top-k documents to form a new set $I_{\text{top}}$

10: if not empty($I_{\text{top}}$) then

11: $C\leftarrow C\cup\\{I_{\text{top}}\\}$ $\triangleright$ Incorporate the
top-k documents into the context $C$

12: end if

13: end for

14: if $C=\emptyset$ then

15: return ”No relevant information found.”

16: end if

17: $P\leftarrow\text{wrap}(C,Q)$ $\triangleright$ Construct a prompt from the
aggregated context C and the User query Q.

18: $A\leftarrow LLM(P)$ $\triangleright$ Use the Large Language Model to
generate an answer.

19: return $A$

20:end procedure

###  III-C Query Handling

This component improves the intelligence of the MoRSE system by managing
complex query types and enriching the context. Below are the specific
functions and the composition of this component:

####  III-C1 Functionalities

  * •

Multi-Hop Question Handling: Deals with queries involving multiple related
entities and allows the system to handle and answer complex multi-hop
questions. Existing Retrieval Augmented Generation systems struggle with
multi-hop queries due to their design limitations and the lack of a dedicated
benchmark dataset for this type of query [48, 49].

  * •

Context Enriching: Generates additional questions from each identified entity,
expanding and enriching the context available for generating informed answers.

  * •

Solving the CVE-CWE Conundrum: Effectively handles queries related to Common
Vulnerabilities and Exposures (CVE) and Common Weakness Enumerations (CWE),
which are challenging for generative models due to their technical complexity
[50, 51].

####  III-C2 Components

  * •

User Query: Initiates the process when a user submits a query via the
graphical user interface.

  * •

CVE-CWE Keyword Extraction: Extracts keywords related to CVEs and CWEs when a
query is received.

  * •

Get CVE Description: Retrieves detailed descriptions of CVEs, including
information about vulnerabilities, affected software and finders.

  * •

Get CWE Description: Retrieves descriptions of CWEs that provide information
about the type of software vulnerabilities, potential impact and mitigation
strategies.

  * •

Entity Extractor: Utilizes the Haystack framework
121212https://haystack.deepset.ai/ with the $\theta$ model to identify and
extract relevant entities (people or concepts) from user queries, improving
the system’s ability to handle Multi-Hop queries.

The complete workflow is outlined in Algorithm 2, demonstrating the mechanism
of Query Handling within the MoRSE system.

Algorithm 2 Query Handling Process

1:A user query $q$ on a specific cybersecurity topic.

2:A series of refined queries $Q^{\prime}$ for in-depth analysis.

3:procedure VulnerabilityExtractor($q$)

4: Extract keywords $K$ related to CVE and CWE from $q$.

5: Retrieve detailed descriptions $D$ for each keyword in $K$ from relevant
databases.

6: Update $q$ by substituting each keyword in $K$ with its corresponding
description from $D$, yielding $q^{\prime}$.

7: return $q^{\prime}$

8:end procedure

9:procedure EntityExtractor($q^{\prime}$)

10: Utilize the Haystack framework with ”dslim/bert-base-NER” to detect
entities $E$ in $q^{\prime}$.

11: Start with an empty Queries_List $Q$ and include $q^{\prime}$.

12: for each detected entity $e\in E$ do

13: if $e$ refers to a person (PER) then

14: Append a question about $e$ to $Q$: "Who is $e$?"

15: else if $e$ pertains to an object or concept (OBJ/CON) then

16: Append a question about $e$ to $Q$: "What is $e$?"

17: end if

18: end for

19: return $Q$

20:end procedure

21:procedure QueryHandling($q$)

22: $q_{vuln}\leftarrow\textsc{VulnerabilityExtractor}(q)$

23: $Q^{\prime}\leftarrow\textsc{EntityExtractor}(q_{vuln})$

24: return $Q^{\prime}$

25:end procedure

###  III-D Structured RAG

As illustrated in Figure 4, the Structured RAG module works post-Query
Handling by forwarding refined queries to seven Parallel Retrievers, called
Structured Retrievers, each of which specializes in specific cybersecurity
topics. Given a query, the information contained in the knowledge base of a
Retriever is inserted into the Context if its similarity to the query is above
a predefined threshold. In order to establish the threshold for each
retriever, we conducted an analysis on the scores of top 50 results from a
series of test queries. Thresholds were then determined by assessing the
distribution of scores 131313https://github.com/Mixture-of-RAGs-Security-
Experts/MoRSE/tree/main/Retriever-Threshold-Scores. In particular, we used the
median value of the test distributions as threshold for the MITRE Retriever
and the Malware Retriever, as these typically retrieve shorter texts. For
Question Retrieval System, CWE Retriever, Metasploit Retriever and Entity
Retriever, we chose the third quartile (Q3) of the test distributions as
threshold, as they generally retrieve longer texts. The ExploitDB Retriever
works without a threshold and uses the TF-IDF algorithm [52]. To mitigate
embedding biases [53], we used two different embeddings for the retrievers,
($\alpha$) and ($\beta$). The following paragraphs delineate each retriever’s
functionalities.

Figure 4: Structured RAG Workflow.

##### Mitre Retriever

The knowledge base of this retriever comes from the website of the MITRE
Corporation141414https://attack.mitre.org/software/. It is structured as a
graph database containing two primary node categories: Malware and Techniques.
Each Malware node in the database contains a name and a description of MITRE.
We create Technique nodes, which consist of technique names and descriptions,
by collecting and analysing technique-related links from the MITRE website.
This retriever utilizes embeddings ($\alpha$). To ensure accurate matches, the
system exclusively evaluates malware with a similarity score exceeding 0.7,
corresponding to the Test distribution’s median.

##### Metasploit Retriever

We have developed the Metasploit Retriever so that it can be effectively
integrated into the Metasploit Framework. Its knowledge base includes over
4900 cybersecurity elements, including exploits, encoders, payloads, and
various modules. To increase the retrieval speed, we only index salient parts
of codes such as code descriptions and exploit information. It performs a
semantic search using a similarity value of 0.75 (Q3 of the test distribution)
with ($\alpha$) embedding, along with a keyword search supported by the TF-IDF
algorithm [52].

##### ExploitDB Retriever

The knowledge base of this retriever is made up of exploits from the ExploitDB
framework. These codes often lack descriptions, so we use a keyword search
with the TF-IDF algorithm [52] to focus only on important data such as CVE
identifiers and author names, which are usually at the beginning of the
scripts. To increase the retrieval speed, we only index the first 600
characters of each script, as this information is often contained in the first
sections of the code.

##### Question Retrieval System

This system acts as a knowledge base containing questions extracted from
chunks of the original documents to better select the most important parts of
the document and the explanations contained therein. A user query is compared
with these questions and if there is a match, the chunk from which the
matching question was extracted is retrieved. During preprocessing, documents
are divided into 2000-character chunks. Model ($\gamma$) generates about seven
questions per chunk, which are refined with Mistral-7B-Instruct-v0.2 [54] for
better alignment. The system uses ($\beta$) embeddings and deploys four
retrievers in sequence, each selecting the ten most relevant documents. The
results are merged, filtered with a similarity threshold of 0.6 (Q3 of the
test distribution) and reordered based on the Lost In the Middle Principle
[55], by placing key information at the beginning or end of the context.
Redundant questions targeting the same document chunk are removed to
streamline the context.

##### Entity Retriever

This retriever contains entities from document chunks together with their
descriptions extracted from the context of the chunk. During the pre-
processing phase, we segment the documents into 500-word chunks. This
segmentation enables the model ($\theta$) to identify and classify relevant
entities more precisely. The contextual descriptions for each entity are then
created using the mistralai/Mistral-7B-Instruct-v0.2 model and converted by
($\beta$) embeddings into a searchable format that is retrievable with a
similarity threshold of 0.5 (Q3 of the test distribution).

##### Malware Retriever

This retriever contains more than 1000 malware source codes originating from
GitHub pages. The Malware Retriever uses a semantic search with ($\alpha$)
embeddings and a threshold value of 0.7 (median of the test distribution) to
match search queries with malware names. In case of matches, all related files
are displayed on the graphical interface.

##### CWE Retriever

The CWE Retriever uses a semantic search with ($\alpha$) embeddings to match
user queries with CWE descriptions, as described in Section III-C. Operating
with a threshold of 0.7 (Q3 of the test distribution), it showcases the 10
most relevant CWEs. When a query closely aligns with a CWE, the retriever
presents detailed information, including code examples.

##### Context Construction

When creating the final context for the prompt, inputs from two main sources
(code snippets and contextual information) are organized to ensure visibility
and impact during the Generation Phase:

  * •

Code Snippets: Following the Lost In the Middle principle [55], code snippets
from Metasploit and ExploitDB are prioritized at the beginning of the prompt
so that they are immediately visible.

  * •

Contextual Information: To ensure that essential and concise information is
presented to the LLM, the content from the queries Mitre, CWE and Entity
retrievers is placed at the end of the prompt. The more comprehensive outputs
of the Question Retriever, which is equipped with a reordering function that
respects the Lost In the Middle principle [55], are placed in the middle.

###  III-E Unstructured RAG

The unstructured RAG, shown in Figure 5, plays a crucial role in the MoRSE
system by handling cybersecurity queries that the structured RAG cannot solve.
The module utilizes retrievers, called buffers, to store documents in chunks
of 2000 characters while maintaining the integrity of the original
information. All buffers work as hybrid retrievers that use both semantic
search and keyword search with the BM25 algorithm [56]. Unlike other
configurations, these retrievers do not have a fixed threshold for semantic
search; instead, they are configured to return the top five documents
regardless of similarity scores. This decision enables further Context
Transformation process to apply semantic thresholds, ensuring the flexibility
and comprehensiveness of the retrieval process.

Figure 5: Unstructured RAG Workflow

##### Classification of Buffers

Buffers are categorized according to the type of data they process, and there
are four different types of buffers:

  * •

Text Buffers: Process content from websites and blogs with five separate
buffers, each analyzing data using the ($\alpha$) embedding.

  * •

Metasploit Buffers: five buffers containing entire codes from the Metasploit
framework that uses the ($\alpha$) embedding for effective processing.

  * •

Code Buffers: A single buffer processes code snippets from Exploit DB and also
uses ($\alpha$) embedding for optimal analysis.

  * •

Paper Buffers: Academic papers are managed by three buffers that use the
($\beta$) embedding to better handle the complex language typical of academic
content [57]. This choice is based on the higher performance values of
embedding, which indicate better retrieval capabilities.

The process Context Transformation refines information from buffers through
four phases:

  * •

Splitting Stage: Document are split into 300-character chunks, improving
relevance selection and reducing noise from larger chunks.

  * •

Redundant Content Removal: In this phase, $\beta$ embeddings are used to
remove redundant content from the segmented chunks and improve the clarity and
uniqueness of the output.

  * •

Filtration Stage: Relevant data chunks are selected using ($\beta$) embeddings
with a threshold of 0.6 set to the third quartile (Q3) of similarity results
from tests with 156 queries to ensure relevance from the knowledge base.

  * •

Reordering Phase: Finally, the data is ordered according to the (Lost in the
Middle) principle to prioritize the visibility of important information in the
response.

TABLE II: Comparative Assessment of RAG Models on 156 General and 150 Multi-Hop Cybersecurity Questions | General Cybersecurity Questions | Multi-Hop Cybersecurity Questions
---|---|---
Model | Relevance | Similarity | Correctness | Relevance | Similarity | Correctness
| $\mu$ | $\sigma$ | $\mu$ | $\sigma$ | $\mu$ | $\sigma$ | $\mu$ | $\sigma$ | $\mu$ | $\sigma$ | $\mu$ | $\sigma$
MoRSE | 0.90 | 0.05 | 0.95 | 0.02 | 0.71 | 0.08 | 0.93 | 0.05 | 0.93 | 0.03 | 0.70 | 0.09
GPT-4 0125-Preview | 0.74 | 0.35 | 0.93 | 0.03 | 0.59 | 0.07 | 0.75 | 0.33 | 0.92 | 0.03 | 0.62 | 0.10
MIXTRAL 7X8 | 0.77 | 0.43 | 0.92 | 0.03 | 0.58 | 0.08 | 0.60 | 0.42 | 0.92 | 0.04 | 0.61 | 0.09
HACKERGPT | 0.66 | 0.33 | 0.92 | 0.04 | 0.58 | 0.08 | 0.73 | 0.18 | 0.79 | 0.05 | 0.47 | 0.03
GEMINI 1.0 Pro | 0.61 | 0.43 | 0.91 | 0.05 | 0.58 | 0.08 | 0.70 | 0.37 | 0.90 | 0.07 | 0.58 | 0.10

##  IV EXPERIMENTS and EVALUATION

The evaluation of Retrieval Augmented Generation systems and Large Language
Models in the field of cybersecurity is particularly challenging due to their
dual role in information retrieval and content generation. The lack of
standardized benchmarks covering a wide range of real-world operational cyber
tasks complicates the evaluation of LLMs for cybersecurity [23, 58, 59].

Key evaluation challenges include verifying the accuracy of the retrieved
information, the effectiveness of its use by LLM, and the overall quality of
the content generated. Traditional methods that focus on language
comprehension may not adequately reflect real-world performance [60].

To effectively address these challenges, we developed a three-part evaluation
strategy for MoRSE and compared its performance with other known LLMs and RAG
systems in answering cybersecurity questions. MoRSE was compared to competing
models such as GPT-4 0125-Preview, MIXTRAL, HACKERGPT, and GEMINI 1.0 Pro. The
three different evaluation test suites are:

  * •

Using the RAGAS framework [23], we evaluate MoRSE’s responses against a ground
truth using a set of metrics.

  * •

Using a method proposed by Zheng et al. [24], we computed Elo Ratings for
MoRSE and competing models by reference-guided pairwise comparison using GPT-4
0125-Preview as a judge. This provides a quantitative measure of relative
performance.

  * •

Following Zheng et al. [24], GPT-4 0125-Preview also rates the responses on a
scale of 1 to 5 based on their comparison with the ground truth references,
which have 5 as the highest default score.

We conducted the assessment using three different types of cybersecurity
questions. The first category, General Cybersecurity Questions, includes 150
simple one-liners that address a wide range of cybersecurity topics. The
second category, Multi-Hop Cybersecurity Questions, includes 150 complex
queries that require a thorough, multi-layered understanding. The third
category focuses on 300 Common Vulnerability Exposure (CVE) questions that
address specific security vulnerabilities. The questions were classified based
on the Diamond Model [20], which we used to create questions representative of
real cybersecurity world needs, and the sample size was statistically selected
based on standard methods 151515https://github.com/Mixture-of-RAGs-Security-
Experts/MoRSE/tree/main/CohensKappaEvaluation. Two experts with 12 and 2 years
of experience validated the ground truth. We used Cohen’s Kappa [21, 22] as a
metric to evaluate the agreement between the two experts. They categorized the
answers as [Correct], [Incorrect] or [Partially_Correct] depending on the
context of the questions. The experts agreed well (Cohen’s Kappa = 0.82),
indicating an Almost Perfect agreement [61].

For all three evaluation test suites, we generated the answers from MoRSE
using the mistralai/Mistral-7B-Instruct-v0.2 model, operating on an NVIDIA
A100 80GB GPU.

###  IV-A First Test Suite: Ground Truth Assessing alignment using the RAGAS
framework

Using the RAGAS framework [23], we focused on three metrics: answer relevance,
answer similarity, and answer correctness. To calculate these metrics, we used
GPT-4 0125-Preview as the underlying model for all calculations.

Answer Relevance, shown in Equation 1, measures how pertinent the generated
answer is to the given prompt. It is calculated by generating related
questions from the model’s answer and comparing their embeddings to the
original question using cosine similarity:

$\text{Answer
Relevance}=\frac{1}{N}\sum_{i=1}^{N}\cos(\mathbf{E}_{g}^{i},\mathbf{E}_{o}),$
(1)

where $\mathbf{E}_{g}^{i}$ and $\mathbf{E}_{o}$ are the $\beta$-embeddings of
the generated and original questions, respectively, and $N$ equal to 3, is the
number of generated questions.

Answer Similarity, shown in Equation 2, evaluates semantic congruence between
model-generated responses and predefined correct answers, calculated as:

$\text{Answer Similarity}=\frac{\mathbf{V}_{\text{ground
truth}}\cdot\mathbf{V}_{\text{generated}}}{\|\mathbf{V}_{\text{ground
truth}}\|\|\mathbf{V}_{\text{generated}}\|},$ (2)

where $\mathbf{V}_{\text{ground truth}}$ and $\mathbf{V}_{\text{generated}}$
represent vector representations of the ground truth and generated answers,
respectively.

Answer Correctness, shown in Equation 3 and 4, evaluates the factual accuracy
of generated answers against ground truth. It combines semantic similarities
and factual correctness:

$AC=w_{FC}\cdot FC+w_{SS}\cdot SS,$ (3)

where $FC$ is the factual correctness, quantified using the F1 score that
considers True Positives ($TP$), False Positives ($FP$), and False Negatives
($FN$):

$FC=\frac{|TP|}{|TP|+0.5\cdot(|FP|+|FN|)},$ (4)

and $SS$ is the semantic similarity between the generated and ground truth
answers. $w_{FC}$ and $w_{SS}$ are the weights assigned to $FC$ and $SS$,
respectively 0.75 and 0.25. In order to calculate TP, FP and FN, RAGAS
framework uses the following prompt instruction: Extract the following from
the given question and ground truth: “TP”: statements that are present in both
the answer and the ground truth, “FP”: statements present in the answer but
not found in the ground truth, “FN”: relevant statements found in the ground
truth but omitted in the answer.

Each of these three metrics requires an embedding model to compute distances
between sentences and a large language model (LLM) for evaluating answer
relevance and correctness. We chose GPT-4 0125-Preview as the LLM and
($\beta$) as the embedding model.

##### Performance Analysis on General and Multi-Hop Questions

Table II shows the results of MoRSE and the other models for General
Cybersecurity Questions and Multi-Hop Cybersecurity Questions. The metrics for
each model are expressed as mean ($\mu$) and standard deviation ($\sigma$),
which indicate the average performance and variability, respectively.

Insights from General Cybersecurity Questions: For the general cybersecurity
questions, MoRSE showed superior performance in all metrics, with a mean
relevance score of 0.90, a similarity score of 0.95, and a correctness score
of 0.71, indicating a high degree of agreement between the answers and the
query prompts, as well as factual accuracy. In comparison, all other models
showed lower consistency and effectiveness, especially in terms of
correctness.

Insights Multi-Hop Cybersecurity Questions: When evaluating complex multi-hop
cybersecurity queries, the MoRSE model outperforms its competitors and proves
that it is capable of answering complicated questions. The data shows that
MoRSE scores consistently high on all metrics, with average scores of 0.93 for
relevance and similarity and 0.70 for correctness. Other models show
significant performance degradation, particularly in correctness, with GPT-4
0125-Preview achieving a mean score of 0.62 and MIXTRAL 0.61, indicating a
lower capacity to handle multi-hop questions.

##### Performance Analysis on CVE Questions

Table III shows how MoRSE and GPT-4 0125 preview approach 300 CVE queries. We
chose GPT-4 0125 preview because it performed best as the second model in both
general and multi-hop contexts (see Table II). We focus on answer similarity
and correctness metrics for scoring answers because they are strictly based on
ground truth and answer relevance does not measure factuality. Moreover, we
compute Accuracy metric, calculated by checking whether the models correctly
identified the vulnerability in the given queries. Regarding Correctness, the
MoRSE model scored 0.64 because its responses often include explanation of
related exploit codes, which are not present in the ground truth that only
describes the vulnerability specifics. This extra information, while useful,
lowers the correctness score as it deviates from the expected response. GPT-4
0125-Preview lags behind in this domain-specific challenge. MoRSE achieved an
accuracy of 84%, surpassing the GPT-4 0125-Preview model, which had an
accuracy of 34%. Our comparison reveals that MoRSE significantly outperforms
GPT-4 0125-Preview in accurately identifying vulnerabilities.

TABLE III: Performance comparison of models on 300 CVE Queries. Model | Similarity | Correctness | Accuracy
---|---|---|---
| $\mu$ | $\sigma$ | $\mu$ | $\sigma$ |
MoRSE | 0.903 | 0.028 | 0.640 | 0.111 | 84%
GPT-4 0125-Preview | 0.852 | 0.004 | 0.558 | 0.107 | 34%

We cannot calculate the accuracy metric for general and multi-hop queries
because, unlike CVE queries, they lack strictly factual data points, such as a
specific vulnerability to identify. For CVEs, accuracy is straightforward: we
check if the model identified the correct vulnerability. In contrast, general
and multi-hop questions often lack such clear data and require assessment
based on multiple aspects depending on the question type.

###  IV-B Retrievers Impact analysis

To calculate the impact of each retriever on 600 questions, we applied a
systematic methodology. First, we collected all contexts generated for 150
general questions, 150 multi-hop questions and 300 CVE questions. We then
analyzed the frequency with which each retriever was able to successfully
retrieve relevant information within these contexts. The frequency of
successful retrievals for each retriever was then calculated as a percentage
of the total questions in each category. In this way, we were able to quantify
the performance and impact of each retriever in both general and multi-hop
question scenarios. Figures 6(a), 6(b) and 6(c) show the impact of the
different retrievers for each question category. For general questions, the
Question Retrieval System has the highest impact at 56.3%, followed by the
Entity Retriever with 21.7%. Other retrievers, such as MITRE, CWE, ExploitDB,
Metasploit and Malware Retriever, have an impact of between 6.1% and 9.0%. For
multi-hop questions, the Question Retrieval System significantly influences
results with a 35.4% contribution, and the Entity Retriever also plays an
important role with a 28.3% contribution. The Metasploit Retriever has an
increased influence of 12%. The other retrievers (Malware, CWE, ExploitDB and
MITRE) have an influence of between 5% and 7%. Figure 6(c) shows the impact on
CVE questions, where the ExploitDB and Metasploit Retrievers have the highest
impact at 18% and 31% respectively. The other retrievers (Malware, CWE, MITRE,
Entity, and Question Ret. Sys.) have an impact ranging from 1% to 14%.

MalwareMetasploitExploitDBCWEMITREEntityQ. Ret.
Sys.$0$$20$$40$$60$$6.1$$8.6$$7$$7$$9$$21.7$$56.3$Impact (%) (a) Impact of
Retrievers on General Questions

MalwareMetasploitExploitDBCWEMITREEntityQ. Ret.
Sys.$0$$20$$40$$6.3$$12$$6$$5$$7$$28.3$$35.4$Impact (%) (b) Impact of
Retrievers on MultiHop Questions

MalwareMetasploitExploitDBCWEMITREEntityQ. Ret.
Sys.$0$$20$$40$$1$$31$$18$$10$$5$$10$$14$Impact (%) (c) Impact of Retrievers
on CVE Questions

Figure 6: Impact of Retrievers on General, MultiHop and CVE Questions

###  IV-C Second Test Suite with LLM as Judge: Reference-Guided Pairwise
Comparison

The prompt for reference-guided pairwise comparison. Please act as an
impartial judge and evaluate the quality of the responses provided by two AI
assistants to the user question displayed below. Your evaluation should
consider correctness and helpfulness. You will be given a reference answer,
assistant A’s answer, and assistant B’s answer. Your job is to evaluate which
assistant’s answer is better. Begin your evaluation by comparing both
assistants’ answers with the reference answer. Identify and correct any
mistakes. Avoid any position biases and ensure that the order in which the
responses were presented does not influence your decision. Do not allow the
length of the responses to influence your evaluation. Do not favor certain
names of the assistants. Be as objective as possible. After providing your
explanation, output your final verdict by strictly following this format:
“[A]” if assistant A is better, “[B]” if assistant B is better, and “[C]” for
a tie. [User Question] Insert user question here. [The Start of Reference
Answer]
Insert reference answer here.
[The End of Reference Answer] [The Start of Assistant A’s Answer]
Insert Assistant A’s answer here.
[The End of Assistant A’s Answer] [The Start of Assistant B’s Answer]
Insert Assistant B’s answer here.
[The End of Assistant B’s Answer]

In our evaluation, we used GPT-4 0125-Preview as a judge to evaluate and
compare the performance of MoRSE, GPT-4 0125-Preview, MIXTRAL 7X8, GEMINI 1.0
Pro and HACKERGPT. This method is based on the research proposed by Zheng et
al. [24], which shows that GPT-4 has a high agreement with human judgment with
an 80% agreement rate. As shown in Prompt IV-C, given a query and the
corresponding reference response, we ask GPT-4 0125-Preview to choose the best
response between Model A and Model B. After GPT-4 0125-Preview completed its
evaluation of all possible battles among the competing models, documented in
Table IV, we derived three different metrics for each model: Elo Rating,
Bootstrap-enhanced Elo Ratings and Maximum Likelihood Estimation:

  * •

Elo Ratings: it quantifies the comparative skill levels across entities in
competitive scenarios, making it apt for evaluating models based on their
head-to-head outcomes. This approach involves initial computation using a
linear update algorithm, opting for a conservative $K$-factor to ensure
stability in ratings, minimizing bias from recent encounters. Equation 5 shows
the Elo rating formula used in our context:

$R_{new}=R_{old}+K\times(S-E),$ (5)

where $R_{new}$ and $R_{old}$ represent the new and old Elo ratings,
respectively. The constant $K$, set equal to 4, controls the volatility of the
rating, $S$ denotes the actual match outcome (1 for a win, 0.5 for a tie, and
0 for a loss), and $E$ is the expected outcome, as calculated in Equation 6:

$E=\frac{1}{1+10^{\frac{R_{new}-R_{old}}{400}}}.$ (6)

  * •

Maximum Likelihood Estimation: Utilizing logistic regression for MLE, we
further analyzed the pairwise comparisons, deducing each model’s probability
of outperforming another, thereby enriching our evaluative scope with
probabilistic insights. The logistic regression model used for MLE can be
formalized in Equation 7:

$\log\left(\frac{p}{1-p}\right)=\beta_{0}+\beta_{1}X_{1}+\dots+\beta_{p}X_{p},$
(7)

where $p$ is the probability of model $A$ winning over model $B$, $X_{i}$
represents the feature vector indicating the participation of models in each
match, and $\beta_{i}$ are the coefficients learned by the regression,
correlating with the Elo scores. The computed coefficients $\beta_{i}$ are
then scaled and adjusted to derive the final Elo scores for each model.

  * •

Bootstrap-enhanced Elo Ratings: To address potential biases related to battle
sequencing, we employed a Bootstrap approach, enhancing the robustness of Elo
ratings and facilitating confidence interval estimations for a more reliable
assessment. The Bootstrap method involves: sampling battles, calculating Elo
ratings, forming a rating distribution across rounds, and determining
confidence intervals (median, 2.5%, and 97.5% percentiles).

##### Performances on General Cybersecurity Question

As shown in Table V, MoRSE leads with the highest Elo score of 1244 and an MLE
Elo of 1252.43, indicating superior performance in general cybersecurity
knowledge. It has a strong bootstrap score of 1225.54, indicating consistent
results within the 1275-1175 confidence interval. It is followed by GPT-4 with
an Elo of 1083 and MLE Elo of 1107.07, also showing reliable performance.
GEMINI and MIXTRAL show moderate performance, with Elo ratings of 926 and 885,
respectively. HACKERGPT performs the worst with the lowest Elo score.

##### Performances on Multi-Hop Cybersecurity Questions

As shown in Table V, in Multi-Hop queries, MoRSE again excels with an Elo of
1280 and an MLE Elo of 1323.67. Its bootstrap score of 1170.62 with a narrow
interval indicates a high stability of performance. GPT-4 remains a
competitive second with an Elo of 1157. MIXTRAL, despite its lower performance
on general questions, performs better on Multi-Hop questions with a better Elo
of 921, while GEMINI struggles, as reflected in its Elo of 907. HACKERGPT
continues to show less stability in this area as well.

##### Complete Questions Insights

Considering 156 General Questions and 150 Multi-Hop Questions, MoRSE stands
out with an Elo of 1267 and an MLE Elo of 1294.66, complemented by a solid
bootstrap score within a trustworthy interval, proving its robustness. GPT-4’s
performance is also solid with an Elo of over 1100. The performances of GEMINI
and MIXTRAL are very similar, with Elo Ratings between 900 and 950. HACKERGPT
remains less consistent overall.

TABLE IV: Fraction of Wins (No Ties) of Each Combination of Models for General Cybersecurity Questions and Multi-Hop Cybersecurity Questions. Model | GPT-4 0125-Preview | GEMINI 1.0 Pro | MIXTRAL 7X8 | HACKERGPT
---|---|---|---|---
General Qs | Mul.Hop Qs | General Qs | Mul.Hop Qs | General Qs | Mul.Hop Qs | General Qs | Mul.Hop Qs
MoRSE | 70% | 64% | 84% | 90% | 88% | 86% | 91% | 95%
GPT-4 0125-Preview | - | - | 67% | 83% | 88% | 77% | 91% | 92%
GEMINI 1.0 Pro | - | - | - | - | 55% | 40% | 70% | 70%
MIXTRAL 7X8 | - | - | - | - | - | - | 70% | 78%
TABLE V: Elo Ratings, MLE Ratings, and Bootstrap Ratings for each competing Models. |  General Cyber. Questions |  Multi-Hop Cyber. Questions |  Complete Cyber. Questions
---|---|---|---
Model | Elo | MLE Elo | Bootstrap | Elo | MLE Elo | Bootstrap | Elo | MLE Elo | Bootstrap
MoRSE | 1244 | 1252.43 | 1225.54 (1275-1175) | 1280 | 1323.67 | 1170.62 (1178-1163) | 1267 | 1294.66 | 1183.81(1192-1175)
GPT-4 0125 | 1083 | 1107.07 | 1096.08 (1105-1063) | 1157 | 1201.19 | 1086.08 (1116-1048) | 1184 | 1151.37 | 1041.05(1050-932)
GEMINI | 926 | 955.67 | 994.53 (1003-984) | 907 | 860.86 | 948.16 (981-917) | 918 | 935.27 | 986.81(996-978)
MIXTRAL | 885 | 882.53 | 891.73 (900-884) | 921 | 941.46 | 929.75 (930-927) | 905 | 907.12 | 945.65(954-935)
HACKERGPT | 863 | 802.30 | 875.52 (925-850) | 734 | 672.34 | 846.32 (852-838) | 727 | 711.70 | 841.59(849-833)

###  IV-D Third Test Suite: LLM as Judge with Five-Level Judgment Criteria
Based on Top-Scoring References

The prompt for Five-Level Judgment Criteria Referencing the Top-Scoring
Reference. Task Description: A Question, a response to evaluate, a reference
answer that gets a score of 5, and a score rubric representing a evaluation
criteria are given. 1\. Write a detailed feedback that assess the quality of
the response strictly based on the given score rubric, not evaluating in
general. 2\. After writing a feedback, write a score that is an integer
between 1 and 5. You should refer to the score rubric. 3\. The output format
should look as follows: ”Feedback: {{write a feedback for criteria}} [RESULT]
{{an integer number between 1 and 5}}” 4\. Please do not generate any other
opening, closing, and explanations. Be sure to include [RESULT] in your
output. Score Rubrics: Is the response correct, accurate, and factual based on
the reference answer? • Score 1: The response is completely incorrect,
inaccurate, and/or not factual. • Score 2: The response is mostly incorrect,
inaccurate, and/or not factual. • Score 3: The response is somewhat correct,
accurate, and/or factual. • Score 4: The response is mostly correct, accurate,
and factual. • Score 5: The response is completely correct, accurate, and
factual. The Question: [Question]
Response to evaluate: [Response to Evaluate]
Reference Answer (Score 5): [Reference Answer]

As illustrated in Prompt IV-D, our methodology involves utilizing GPT-4
0125-Preview to assign scores ranging from 1 to 5 to each model answers based
on a Top-Scoring Reference.

These scores are based on a comparison with a reference answer, which is given
a perfect score of 5. The procedure for this scoring corresponds to the
methodology described in
161616https://huggingface.co/learn/cookbook/rag_evaluation.

As can be seen from Table VI, MoRSE outperforms in all question categories
(General questions, Multi-Hop questions, and CVE questions). GPT-4
0125-Preview follows MoRSE as the second most competent model and shows strong
versatility in all categories, but with a significantly lower performance than
MoRSE. Other models, GEMINI, HACKERGPT and MIXTRAL, show different levels of
performance, with none reaching the effectiveness of MoRSE or GPT-4
0125-Preview in terms of general cybersecurity knowledge or the more targeted
questions on Multi-Hop and CVE.

TABLE VI: Scores Across All Cybersecurity Questions types. Model | General Qs | Multi-Hop Qs | CVE Qs
---|---|---|---
MoRSE | 4.54 | 4.65 | 4.024
GPT-4 0125 | 3.14 | 3.54 | 2.193
GEMINI | 2.91 | 2.43 | -
HACKERGPT | 2.65 | 3.05 | -
MIXTRAL | 2.58 | 1.40 | -

###  IV-E Test Case Analysis for MoRSE RAG Retrievers

This section reports the evaluation on the performance of both the structured
and unstructured RAG components within the MoRSE system, as shown in Table
VII. We evaluated each retriever against a customized set of 100 test
questions based on their specialized knowledge in cybersecurity, focusing on
processing time efficiency, dense retriever size, denoted by Size and number
of documents (No. Doc). We also test the reliability of the structured
Retrievers using test queries with error rate analysis to estimate future
reliability using a confidence interval.

##### Analysis of the performance of Structured Retrievers

Table VII shows the performance metrics for Structured RAG Retrievers within
the MoRSE system. The ExploitDB Retriever Size is not listed as it uses TF-IDF
for retrieval. The Malware Retriever processes queries with average times of
0.061 seconds on GPU and 0.110 seconds on CPU. The CWE Retriever and the MITRE
Retriever have average processing times of 0.083 and 0.057 seconds on GPU and
0.108 and 0.13 seconds on CPU.

For the ExploitDB Retriever, the average processing times are 1.254 seconds on
GPU and 2.377 seconds on CPU. The Metasploit Retriever shows average times of
0.995 seconds on GPU and 1.367 seconds on CPU when processing a 40MB dense
retriever. In addition, Question Retrieval System and Entity Retriever are
noteworthy as they have the largest dense retrievers within the Structured RAG
components at 3.863 GB and 554 MB respectively. The Question Retrieval System
has an average processing time of 2.536 seconds on the CPU and 2.492 seconds
on the GPU. The Entity Retriever achieves a moderate processing time of 0.250
seconds on the GPU and 0.268 seconds on the CPU due to its relatively high
density.

##### Analysis of the Performances of Unstructured RAG Components

As shown in Table VII, GPU processing significantly increases performance,
especially for components that work with large data sets. The Text Buffers saw
significant performance gains when transitioning to the GPU, likely due to
their large data size of 185 MB, which is reflected in a reduction in average
time from 85.22 seconds on the CPU to 1.789 seconds on the GPU. The Code
Buffer also benefited from GPU acceleration, but to a lesser extent,
demonstrating that its tasks is less demanding on processing power. For
Metasploit Buffer, the GPU significantly improved efficiency by managing the
186 MB dataset more effectively and reducing the average time. Metasploit
Buffer size is different the Structured Metasploit Retriever Size because the
first contains the entire codes. The Context Transformation with the highest
computational requirements showed the GPU’s advantage in processing large
amounts of data, a critical aspect of tasks that require fast data analysis
and processing, highlighting the adaptability of the MoRSE system to hardware
improvements.

Unstructured RAG components show a significant performance increase when
running on GPUs, so they are preferred to run on GPU architectures, but only
when the structured RAG fails. This is to ensure that the considerable memory
resources required for the large buffers of unstructured RAG are utilized and
sufficient memory is retained for the generation task.

##### Structured Retrievers Failure Rate Analysis

The aim is to estimate the probability of the Structured Retrievers not
finding relevant text data in response to a user query, so that a switch to
the Unstructured RAG is necessary to continue the search for relevant
information. To this end, we have focused on looking only at the Contextual
Information part of the final Context (see Figure 4). We have individually
multiplied the retrievers’ failure rates listed in Table VII, determining a
collective probability of failure at approximately 0.2569%, equal to an
empirical rate ($\hat{p}$) of 0.0026. When assessing reliability, we used a
95% confidence interval [62], the empirical rate, a z-score of 1.96, and the
Failure Rates (Fail.Rate) from Table VII. This analysis shows that the maximum
failure rate under the tested conditions does not exceed 0.46%, which confirms
the robustness of the system when processing text-based queries.

TABLE VII: Performance summary of Structured and Unstructured RAG components tested on CPU and GPU (NVIDIA A100 80GB). Fail. Qs. states for Failed Test Queries, while Tot. Qs. states for Total Test Queries Component | Mean Time (s) | Time Std. | Fail. Qs | Tot. Qs | Fail. Rate | Size | Embed. | Thres. | No. Doc.
---|---|---|---|---|---|---|---|---|---
| GPU | CPU | GPU | CPU |  |  |  |  |  |  |
Structured RAG Retrievers |  |
Malware Retriever | 0.061 | 0.110 | 0.009 | 0.013 | 6 | 100 | 6% | 3.9MB | $\alpha$ | 0.7 | $\sim 1000$
Metasploit Retriever | 0.995 | 1.367 | 0.592 | 1.307 | 5 | 100 | 5% | 40MB | $\alpha$ | 0.75 | $\sim 4900$
ExploitDB Retriever | 1.254 | 2.377 | 0.998 | 1.198 | 8 | 100 | 8% | - | - | - | $\sim 13000$
CWE Retriever | 0.083 | 0.108 | 0.056 | 0.019 | 28 | 100 | 28% | 5.86MB | $\alpha$ | 0.73 | $\sim 1000$
MITRE Retriever | 0.057 | 0.13 | 0.07 | 0.029 | 19 | 100 | 19% | 3.2MB | $\alpha$ | 0.7 | $\sim 800$
Entity Retriever | 0.250 | 0.268 | 0.078 | 0.018 | 23 | 100 | 23% | 554MB | $\beta$ | 0.5 | $\sim 7000$
Question Ret. Sys. | 2.492 | 2.536 | 0.298 | 0.338 | 21 | 100 | 21% | 3.863GB | $\beta$ | 0.6 | $\sim 7000$
Unstructured RAG Components |  |
Paper Buffers | 0.838 | 39.8 | 0.139 | 2.94 | 1 | 100 | 1% | 86.4MB | $\beta$ | Top 5 | $\sim 380$
Text Buffers | 1.789 | 85.22 | 0.298 | 6.29 | 1 | 100 | 1% | 185MB | $\alpha$ | Top 5 | $\sim 6000$
Code Buffers | 0.416 | 19.81 | 0.069 | 1.46 | 8 | 100 | 8% | 43.0MB | $\alpha$ | Top 5 | $\sim 1000$
Metasploit Buffers | 1.799 | 85.68 | 0.299 | 6.32 | 5 | 100 | 5% | 186MB | $\alpha$ | Top 5 | $\sim 4900$
Context Transf. | 4.84 | 230.5 | 0.805 | 17.01 | - | 100 | - | - | $\alpha$ | 0.6 | -

##  V Related Work

We now overview recent developments in Named Entity Recognition (NER),
Knowledge Graphs (KGs), and Large Language Models (LLMs) that have contributed
to more sophisticated, automated, and adaptive cybersecurity systems.

##### Named Entity Recognition in Cybersecurity

Significant progress has been made in the evolving landscape of Named Entity
Recognition (NER) for cybersecurity. In particular, the use of BERT and its
Whole Word Masking variant with a BiLSTM-CRF framework has shown remarkable
improvements in entity recognition metrics [63]. Similarly, by fusing rule-
based, dictionary-based methods and CRF, the RDF-CRF model has significantly
improved entity recognition in the cybersecurity domain [64]. In addition, a
hybrid model that combines deep learning with dictionary-based methods has
significantly improved precision and recognition in the identification of
complex entities [65]. A study by Srivastava et al. [66] highlighted the
differential effectiveness of word embeddings such as fastText, GloVe and
BERT, with fine-tuned BERT embeddings with a feed forward network achieving an
F1 score of 0.974, highlighting the importance of model adaptation to specific
domains. In addition, the introduction of the JCLB model, which combines
contrastive learning with a Belief Rule Base [67], showed improved accuracy
through semantic expansion and optimized BRB parameters. Li et al. developed
NEDetector [68], which improves NER by identifying cybersecurity neologisms
with 89.11% accuracy, outperforming traditional trending tools in detecting
threats on platforms like Twitter. Extractor distills attack behavior from CTI
reports into clear, actionable insights and leverages provenance graphs to
improve cyber analytics in threat hunting with real-world effectiveness [69].
Koloveas et al. [70] created the inTIME framework, which leverages machine
learning to transform web data into actionable CTI, streamlining the
intelligence lifecycle with a unified platform for intelligence collection,
analysis and sharing. At the same time, a new threat modeling language has
been developed by Xiong et al. [71] based on the MITRE Enterprise ATT&CK
Matrix that integrates key elements of enterprise security to improve defense
strategies through simulations. Husari et al. [72] further refines CTI
analysis by automating the extraction of threat actions from unstructured
texts, employing NLP and IR for semantic extraction and aligning attack
patterns with standards like STIX 2.1, achieving significant precision and
recall in its evaluations.

MoRSE goes beyond current NER technologies by providing dynamic entity
recognition and response generation. It identifies named entities in user
queries in real time and uses this information to generate precise,
contextualised responses.

##### Knowledge Graphs for Cybersecurity

Knowledge graphs (KGs) are revolutionizing cybersecurity, from threat
intelligence to education. Agrawal et al. [73] have shown how KGs from
unstructured text enhance cybersecurity learning, with student feedback
highlighting improved understanding and engagement. Sewak et al. developed
CRUSH [74], which integrates Large Language Models (LLMs) such as
GPT-3.5/GPT-4/ChatGPT with Enterprise Knowledge Graphs (EKGs) to create Threat
Intelligence Graphs (TIG) that achieve up to 99% recall in identifying
malicious scripts. Li et al. developed AttacKG [75], which automates the
extraction of attack techniques from CTI reports into structured knowledge
graphs, which greatly improves the analysis of attack patterns with high
accuracy and supports advanced threat detection efforts. Liu et al. [76] use
NLP to convert over 29k cybersecurity reports into 113,543 actionable Cyber
Threat Intelligence (CTI) points by highlighting campaign triggers for better
classification accuracy. Piplay et al. [77] describes a system for generating
Cybersecurity Knowledge Graphs (CKGs) from After Action Reports (AARs) using a
‘Malware Entity Extractor’ and neural networks that improves security analysis
through refined query responses. Gao et al. [78] use of a heterogeneous
information network (HIN) and meta-path approach within a graph convolutional
network is characterized by its sophisticated threat type identification
capabilities validated by real-world data. Sikos et al. [79] discuss how
knowledge graphs help in cybersecurity threat intelligence and automated
reasoning, and highlights their importance in analyzing cyber data. Ren et al.
[80] present a knowledge graph for APT attack mapping that combines deep
learning with network defense expertise. Mitra et al. [81] augment CKGs with
provenance information to combat fake cybersecurity information and ensure
data reliability.

In comparison, MoRSE addresses the limitations of Knowledge Graphs (KGs) in
cybersecurity by providing dynamic updates with real-time adjustments as
opposed to the manual revisions of KGs. It also provides interactive query
capabilities that are real-time and interactive as opposed to the static KGs.
In addition, MoRSE offers improved customizability and modularity, which
allows for greater adaptability and easier expansion of the knowledge base
compared to the fixed structures of KGs.

##### LLMs and Chatbots in Cybersecurity Landscape

Research into chatbots in cybersecurity highlights their role in education,
ethics and regulation. Yoo et al. (2024) and Abu-Amara et al. (2024) explore
the impact of GDPR and the use of gamified chatbots in education, while
Pieterse (2024) evaluates the utility of ChatGPT in guiding students through
cybersecurity CTF challenges, noting that it reaches its limits in providing
direct solutions [82, 83, 84]. Mitra et al. [85] developed LOCALINTEL, which
presents an automated system that uses LLMs to create organisation-specific
threat intelligence from global and local databases to improve the efficiency
of SoC operations. Juttner et al. [86] use ChatGPT to make IDS alerts
understandable to non-experts, improving network security in the home and home
office environment. However, concerns about trust, privacy and ethics
highlight the need for further research before widespread adoption. Yoo et al.
[87] use CNN classifiers and AI chatbot to detect and combat SNS phishing
attacks before they can occur. This is more promising than traditional methods
because it provides real-time support and actions on Telegram, with validated
effectiveness against LSTM models. Iqbal et al. [88] explore the dual utility
of ChatGPT in cybersecurity, highlighting its benefits for defense strategies
and the risks of its misuse in cyberattacks, and call for more research on its
offensive capabilities. Chamberlain and Casey [89] examine the application of
ChatGPT in penetration testing and CTF exercises, pointing to its potential to
create dynamic scenarios and enhance the learning process. Aghaei et al. [90]
developed SecureBERT, which automates critical cybersecurity tasks by
introducing a specialized language model for Cyber Threat Intelligence (CTI)
that uses a customized tokenizer and pre-trained weights for improved
performance on NLP tasks. Ameri et al. [91] use BERT for feature
classification and achieves improved accuracy from 76% to 94.4% by optimizing
the hyperparameters. It demonstrated robustness with a standard deviation of
±0.6% across all validations and outperformed models such as GPT2, ULMFiT,
ELMo, CNN, LSTM and BiLSTM on cybersecurity tasks. Voros et al. [92] use
knowledge distillation from LLMs to efficiently categorize URLs, reduce the
number of parameters and improve inline scanning. Happe et al. [93] use
GPT-3.5 to extend penetration testing and demonstrates LLMs as AI sparring
partners in security testing. Lu et al. [94] integrate graph structural
information and in-context learning into LLM-based software vulnerability
detection, significantly outperforming conventional models. Yu et al. [95] use
GPT-3 to generate semantic honeywords containing users’ PII, which improves
their indistinguishability and increases defenses against security breaches.

Unlike traditional LLMs and chatbots, MoRSE stands out by providing instant
access to a constantly updating cybersecurity knowledge base, quickly
integrating the latest threats and solutions, and enabling extensive
customization for various cybersecurity needs. MoRSE does not focus on niche
cybersecurity topics; instead, it aims to provide comprehensive coverage of
cybersecurity knowledge. In addition, MoRSE enhances the user experience by
providing user-friendly access to complex cybersecurity information,
increasing flexibility and expanding accessibility.

##  VI Conclusions and Future Work

With cyber threats on the rise, effective cybersecurity strategies are
becoming increasingly important. Integrating continuous learning and Retrieval
Augmented Generation (RAG) into large language models (LLMs) improves their
accuracy and timeliness in responding to these threats. In this paper, we have
investigated the use of two Retrieval Augmented Generation (RAG) systems,
namely Structured RAG and Unstructured RAG, to provide precise and structured
answers to cybersecurity queries. In Structured RAG, we have focused on
implementing parallel retrievers to quickly and effectively find the relevant
document in response to a user query. Unstructured RAG, on the other hand, is
designed to answer even the most complicated cybersecurity queries. We have
implemented an evaluation suite to assess the relevance, similarity, and
correctness of the answers generated by our system compared to ground truth.
The performance of our system was compared to other renowned commercial Large
Language Models (LLMs) using two additional test suites that adhere to the LLM
as a Judge paradigm. The results show that our system outperforms competing
models by more than 10% in terms of the correctness and relevance of the given
answers and by 50% in terms of accuracy against GPT-4 for questions related to
vulnerabilities.

In order to make the framework publicly available, our goal is to improve
MoRSE with a Prefix-aware Greedy replacement policy (PAGRP) [96] for the
semantic cache. This takes into account the initial segments of queries or
data to make more informed decisions about what data to store in the cache.
The PAGRP prioritizes the cache items based on their access frequency and
size, ensuring that the system stores the most useful data and minimizes the
likelihood of cache misses. In addition, we plan to replace the MITRE
retriever with a comprehensive Knowledge Graph. This Knowledge Graph will
include mitigation and detection approaches and aggregate real malware reports
associated with the corresponding MITRE software. This solution enables better
threat insights by allowing the computation of community analysis, centrality
algorithms, and threat similarities quickly and in real-time.

## References

  * [1] B. Shin and P. B. Lowry, “A review and theoretical explanation of the ‘cyberthreat-intelligence (cti) capability’that needs to be fostered in information security practitioners and how this can be accomplished,” _Computers & Security_, vol. 92, p. 101761, 2020.
  * [2] F. B. Kokulu, A. Soneji, T. Bao, Y. Shoshitaishvili, Z. Zhao, A. Doupé, and G.-J. Ahn, “Matched and mismatched socs: A qualitative study on security operations center issues,” in _Proceedings of the 2019 ACM SIGSAC conference on computer and communications security_ , 2019, pp. 1955–1970.
  * [3] N. Sun, M. Ding, J. Jiang, W. Xu, X. Mo, Y. Tai, and J. Zhang, “Cyber threat intelligence mining for proactive cybersecurity defense: a survey and new perspectives,” _IEEE Communications Surveys & Tutorials_, 2023.
  * [4] R. Kaur, D. Gabrijelčič, and T. Klobučar, “Artificial intelligence for cybersecurity: Literature review and future research directions,” _Information Fusion_ , p. 101804, 2023.
  * [5] I. H. Sarker, “Llm potentiality and awareness: A position paper from the perspective of trustworthy and responsible ai modeling,” _Authorea Preprints_ , 2024.
  * [6] N. Kandpal, H. Deng, A. Roberts, E. Wallace, and C. Raffel, “Large language models struggle to learn long-tail knowledge,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 15 696–15 707.
  * [7] Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,” _High-Confidence Computing_ , p. 100211, 2024.
  * [8] J. Kocoń, I. Cichecki, O. Kaszyca, M. Kochanek, D. Szydło, J. Baran, J. Bielaniewicz, M. Gruza, A. Janz, K. Kanclerz _et al._ , “Chatgpt: Jack of all trades, master of none,” _Information Fusion_ , vol. 99, p. 101861, 2023.
  * [9] P. P. Ray, “Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope,” _Internet of Things and Cyber-Physical Systems_ , 2023.
  * [10] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” _ACM Computing Surveys_ , vol. 55, no. 12, pp. 1–38, 2023.
  * [11] J. Kasai, K. Sakaguchi, R. Le Bras, A. Asai, X. Yu, D. Radev, N. A. Smith, Y. Choi, K. Inui _et al._ , “Realtime qa: What’s the answer right now?” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [12] A. Mallen, A. Asai, V. Zhong, R. Das, D. Khashabi, and H. Hajishirzi, “When not to trust language models: Investigating effectiveness of parametric and non-parametric memories,” in _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 9802–9822. [Online]. Available: https://aclanthology.org/2023.acl-long.546
  * [13] J. Liu, C. S. Xia, Y. Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [14] Y. A. Yadkori, I. Kuzborskij, A. György, and C. Szepesvári, “To believe or not to believe your llm,” 2024. [Online]. Available: https://arxiv.org/abs/2406.02543
  * [15] K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval augmentation reduces hallucination in conversation,” in _Findings of the Association for Computational Linguistics: EMNLP 2021_ , M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 3784–3803. [Online]. Available: https://aclanthology.org/2021.findings-emnlp.320
  * [16] G. Izacard and E. Grave, “Leveraging passage retrieval with generative models for open domain question answering,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_ , 2020.
  * [17] S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. De Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. Rae, E. Elsen, and L. Sifre, “Improving language models by retrieving from trillions of tokens,” in _Proceedings of the 39th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, Eds., vol. 162. PMLR, 17–23 Jul 2022, pp. 2206–2240. [Online]. Available: https://proceedings.mlr.press/v162/borgeaud22a.html
  * [18] S. Masoudnia and R. Ebrahimpour, “Mixture of experts: a literature survey,” _Artificial Intelligence Review_ , vol. 42, pp. 275–293, 2014.
  * [19] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand _et al._ , “Mixtral of experts,” _arXiv preprint arXiv:2401.04088_ , 2024.
  * [20] S. Caltagirone, A. D. Pendergast, and C. Betz, “The diamond model of intrusion analysis,” 2013. [Online]. Available: https://api.semanticscholar.org/CorpusID:108270876
  * [21] J. Wang and B. Xia, “Relationships of cohen’s kappa, sensitivity, and specificity for unbiased annotations,” in _Proceedings of the 4th International Conference on Biomedical Signal and Image Processing, ICBIP 2019, Chengdu, China, August 13-15, 2019_. ACM, 2019, pp. 98–101. [Online]. Available: https://doi.org/10.1145/3354031.3354040
  * [22] M. McHugh, “Interrater reliability: The kappa statistic,” _Biochemia medica : časopis Hrvatskoga društva medicinskih biokemičara / HDMB_ , vol. 22, pp. 276–82, 10 2012.
  * [23] S. Es, J. James, L. Espinosa Anke, and S. Schockaert, “RAGAs: Automated evaluation of retrieval augmented generation,” in _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , N. Aletras and O. De Clercq, Eds. St. Julians, Malta: Association for Computational Linguistics, Mar. 2024, pp. 150–158. [Online]. Available: https://aclanthology.org/2024.eacl-demo.16
  * [24] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing _et al._ , “Judging llm-as-a-judge with mt-bench and chatbot arena,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [25] A. Vaswani _et al._ , “Attention is all you need,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , 2017.
  * [26] A. Radford _et al._ , “Improving language understanding by generative pre-training,” _OpenAI Blog_ , 2018.
  * [27] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [28] J. Devlin _et al._ , “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2019.
  * [29] T. B. Brown _et al._ , “Language models are few-shot learners,” in _Advances in Neural Information Processing Systems_ , 2020.
  * [30] E. M. Bender _et al._ , “On the dangers of stochastic parrots: Can language models be too big?” in _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , 2021.
  * [31] E. Strubell _et al._ , “Energy and policy considerations for deep learning in nlp,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019.
  * [32] A. Wang _et al._ , “Efficient training of large language models: A survey of techniques and practices,” _arXiv preprint arXiv:2006.04962_ , 2020.
  * [33] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel _et al._ , “Retrieval-augmented generation for knowledge-intensive nlp tasks,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 9459–9474, 2020.
  * [34] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang, “Retrieval augmented language model pre-training,” in _Proceedings of the 37th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119. PMLR, 13–18 Jul 2020, pp. 3929–3938. [Online]. Available: https://proceedings.mlr.press/v119/guu20a.html
  * [35] V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense passage retrieval for open-domain question answering,” _arXiv preprint arXiv:2004.04906_ , 2020.
  * [36] S. Borgeaud _et al._ , “Improving lexical choice in neural machine translation,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 2021.
  * [37] G. Zhou, T. He, J. Zhao, and P. Hu, “Learning continuous word embedding with metadata for question retrieval in community question answering,” in _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 2015, pp. 250–259.
  * [38] K. Li, H. Zhou, Z. Tu, and B. Feng, “Cskb: A cyber security knowledge base based on knowledge graph,” in _Security and Privacy in Digital Economy: First International Conference, SPDE 2020, Quzhou, China, October 30–November 1, 2020, Proceedings 1_. Springer, 2020, pp. 100–113.
  * [39] N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, “Mteb: Massive text embedding benchmark,” _arXiv preprint arXiv:2210.07316_ , 2022.
  * [40] O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton-Brown, and Y. Shoham, “In-context retrieval-augmented language models,” _Transactions of the Association for Computational Linguistics_ , vol. 11, pp. 1316–1331, 2023.
  * [41] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” _ACM Computing Surveys_ , vol. 55, no. 9, pp. 1–35, 2023.
  * [42] D. Chandrasekaran and V. Mago, “Evolution of semantic similarity—a survey,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 2, pp. 1–37, 2021.
  * [43] A. Aynetdinov and A. Akbik, “Semscore: Automated evaluation of instruction-tuned llms based on semantic textual similarity,” _CoRR_ , vol. abs/2401.17072, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2401.17072
  * [44] P. Zhang, S. Xiao, Z. Liu, Z. Dou, and J.-Y. Nie, “Retrieve anything to augment large language models,” _arXiv preprint arXiv:2310.07554_ , 2023.
  * [45] P. Zhang, Z. Liu, S. Xiao, N. Shao, Q. Ye, and Z. Dou, “Soaring from 4k to 400k: Extending llm’s context with activation beacon,” _arXiv preprint arXiv:2401.03462_ , 2024.
  * [46] C. Li, Z. Liu, S. Xiao, and Y. Shao, “Making large language models a better foundation for dense retrieval,” _arXiv preprint arXiv:2312.15503_ , 2023.
  * [47] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _Journal of machine learning research_ , vol. 21, no. 140, pp. 1–67, 2020.
  * [48] Y. Tang and Y. Yang, “Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries,” 2024.
  * [49] W. Xiong, X. L. Li, S. Iyer, J. Du, P. Lewis, W. Y. Wang, Y. Mehdad, W.-t. Yih, S. Riedel, D. Kiela _et al._ , “Answering complex open-domain questions with multi-hop dense retrieval,” _arXiv preprint arXiv:2009.12756_ , 2020.
  * [50] A. Abdallah and A. Jatowt, “Generator-retriever-generator: A novel approach to open-domain question answering,” _arXiv preprint arXiv:2307.11278_ , 2023.
  * [51] L. Gao, X. Ma, J. Lin, and J. Callan, “Precise zero-shot dense retrieval without relevance labels,” in _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 1762–1777. [Online]. Available: https://aclanthology.org/2023.acl-long.99
  * [52] J. Ramos _et al._ , “Using tf-idf to determine word relevance in document queries,” in _Proceedings of the first instructional conference on machine learning_ , vol. 242, no. 1. Citeseer, 2003, pp. 29–48.
  * [53] M.-E. Brunet, C. Alkalay-Houlihan, A. Anderson, and R. Zemel, “Understanding the origins of bias in word embeddings,” in _International conference on machine learning_. PMLR, 2019, pp. 803–811.
  * [54] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier _et al._ , “Mistral 7b,” _arXiv preprint arXiv:2310.06825_ , 2023.
  * [55] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” _Transactions of the Association for Computational Linguistics_ , vol. 12, pp. 157–173, 2024.
  * [56] S. Wang, S. Zhuang, and G. Zuccon, “Bert-based dense retrievers require interpolation with bm25 for effective passage retrieval,” in _Proceedings of the 2021 ACM SIGIR international conference on theory of information retrieval_ , 2021, pp. 317–324.
  * [57] S. Xiao, Z. Liu, P. Zhang, and N. Muennighoff, “C-pack: Packaged resources to advance general chinese embedding,” 2023.
  * [58] J. Gennari, S.-h. Lau, S. Perl, J. Parish, and G. Sastry, “Considerations for evaluating large language models for cybersecurity tasks,” 2024.
  * [59] M. Sultana, A. Taylor, L. Li, and S. Majumdar, “Towards evaluation and understanding of large language models for cyber operation automation,” in _2023 IEEE Conference on Communications and Network Security (CNS)_. IEEE, 2023, pp. 1–6.
  * [60] S. Wang, Y. Song, A. Drozdov, A. Garimella, V. Manjunatha, and M. Iyyer, “$k$NN-LM does not improve open-ended text generation,” in _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 15 023–15 037. [Online]. Available: https://aclanthology.org/2023.emnlp-main.929
  * [61] J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” _Biometrics_ , vol. 33, no. 1, pp. 159–174, 1977. [Online]. Available: http://www.jstor.org/stable/2529310
  * [62] D. S. Moore, G. P. McCabe, and B. A. Craig, _Introduction to the Practice of Statistics_. W. H. Freeman and Company, 2019.
  * [63] S. Zhou, J. Liu, X. Zhong, and W. Zhao, “Named entity recognition using bert with whole world masking in cybersecurity domain,” in _2021 IEEE 6th International Conference on Big Data Analytics (ICBDA)_. IEEE, 2021, pp. 316–320.
  * [64] F. Yi, B. Jiang, L. Wang, and J. Wu, “Cybersecurity named entity recognition using multi-modal ensemble learning,” _IEEE Access_ , vol. 8, pp. 63 214–63 224, 2020.
  * [65] C. Gao, X. Zhang, and H. Liu, “Data and knowledge-driven named entity recognition for cyber security,” _Cybersecurity_ , vol. 4, no. 1, p. 9, 2021.
  * [66] S. Srivastava, B. Paul, and D. Gupta, “Study of word embeddings for enhanced cyber security named entity recognition,” _Procedia Computer Science_ , vol. 218, pp. 449–460, 2023.
  * [67] C. Hu, T. Wu, C. Liu, and C. Chang, “Joint contrastive learning and belief rule base for named entity recognition in cybersecurity,” _Cybersecurity_ , vol. 7, no. 1, p. 19, 2024.
  * [68] Y. Li, J. Cheng, C. Huang, Z. Chen, and W. Niu, “Nedetector: Automatically extracting cybersecurity neologisms from hacker forums,” _Journal of Information Security and Applications_ , vol. 58, p. 102784, 2021.
  * [69] K. Satvat, R. Gjomemo, and V. Venkatakrishnan, “Extractor: Extracting attack behavior from threat reports,” in _2021 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE, 2021, pp. 598–615.
  * [70] P. Koloveas, T. Chantzios, S. Alevizopoulou, S. Skiadopoulos, and C. Tryfonopoulos, “intime: A machine learning-based framework for gathering and leveraging web data to cyber-threat intelligence,” _Electronics_ , vol. 10, no. 7, p. 818, 2021.
  * [71] W. Xiong, E. Legrand, O. Åberg, and R. Lagerström, “Cyber security threat modeling based on the mitre enterprise att&ck matrix,” _Software and Systems Modeling_ , vol. 21, no. 1, pp. 157–177, 2022.
  * [72] G. Husari, E. Al-Shaer, M. Ahmed, B. Chu, and X. Niu, “Ttpdrill: Automatic and accurate extraction of threat actions from unstructured text of cti sources,” in _Proceedings of the 33rd annual computer security applications conference_ , 2017, pp. 103–115.
  * [73] G. Agrawal, Y. Deng, J. Park, H. Liu, and Y. Chen, “Building knowledge graphs from unstructured texts: Applications and impact analyses in cybersecurity education,” _Inf._ , vol. 13, no. 1, p. 52, 2022.
  * [74] M. Sewak, V. Emani, and A. Naresh, “Crush: Cybersecurity research using universal llms and semantic hypernetworks,” 2023.
  * [75] Z. Li, J. Zeng, Y. Chen, and Z. Liang, “Attackg: Constructing technique knowledge graph from cyber threat intelligence reports,” in _European Symposium on Research in Computer Security_. Springer, 2022, pp. 589–609.
  * [76] J. Liu, J. Yan, J. Jiang, Y. He, X. Wang, Z. Jiang, P. Yang, and N. Li, “Tricti: an actionable cyber threat intelligence discovery system via trigger-enhanced neural network,” _Cybersecurity_ , vol. 5, no. 1, p. 8, 2022.
  * [77] A. Piplai, S. Mittal, A. Joshi, T. Finin, J. Holt, and R. Zak, “Creating cybersecurity knowledge graphs from malware after action reports,” _IEEE Access_ , vol. 8, pp. 211 691–211 703, 2020.
  * [78] Y. Gao, X. Li, H. Peng, B. Fang, and S. Y. Philip, “Hincti: A cyber threat intelligence modeling and identification system based on heterogeneous information network,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 34, no. 2, pp. 708–722, 2020.
  * [79] L. Sikos, “Cybersecurity knowledge graphs,” _Knowledge and Information Systems_ , 2023.
  * [80] Y. Ren, Y. Xiao, Y. Zhou, Z. Zhang, and Z. Tian, “Cskg4apt: A cybersecurity knowledge graph for advanced persistent threat organization attribution,” _IEEE Transactions on Knowledge and Data Engineering_ , 2022.
  * [81] S. Mitra, A. Piplai, S. Mittal, and A. Joshi, “Combating fake cyber threat intelligence using provenance in cybersecurity knowledge graphs,” in _2021 IEEE International Conference on Big Data (Big Data)_. IEEE, 2021, pp. 3316–3323.
  * [82] C. Yoo, J. Wolff, and W. Lehr, “Lessons from gdpr for ai policymaking,” _Virginia Journal of Law & Technology_, 2024, https://scholarship.law.upenn.edu/cgi/viewcontent.cgi?article=1376&context=faculty_articles.
  * [83] F. Abu-Amara, R. Hosani, H. Tamimi _et al._ , “Spreading cybersecurity awareness via gamification: zero-day game,” _International Journal of Information Technology_ , 2024, https://link.springer.com/article/10.1007/s41870-024-01810-4.
  * [84] H. Pieterse, “Friend or foe–the impact of chatgpt on capture the flag competitions,” in _International Conference on Cyber Warfare and Security_ , vol. 19, no. 1, 2024, pp. 268–276.
  * [85] S. Mitra, S. Neupane, T. Chakraborty, S. Mittal, A. Piplai, M. Gaur, and S. Rahimi, “Localintel: Generating organizational threat intelligence from global and local cyber knowledge,” _arXiv preprint arXiv:2401.10036_ , 2024.
  * [86] V. Jüttner, M. Grimmer, and E. Buchmann, “Chatids: Explainable cybersecurity using generative ai,” _arXiv preprint arXiv:2306.14504_ , 2023.
  * [87] J. Yoo and Y. Cho, “Icsa: Intelligent chatbot security assistant using text-cnn and multi-phase real-time defense against sns phishing attacks,” _Expert Systems with Applications_ , vol. 207, p. 117893, 2022.
  * [88] F. Iqbal, F. Samsom, F. Kamoun, and Áine MacDermott, “When chatgpt goes rogue: exploring the potential cybersecurity threats of ai-powered conversational chatbots,” _Frontiers in Communications and Networks_ , 2023.
  * [89] D. Chamberlain and E. Casey, “Capture the flag with chatgpt: Security testing with ai chatbots,” in _International Conference on Cyber Warfare and Security_ , vol. 19, no. 1, 2024, pp. 43–54.
  * [90] E. Aghaei, X. Niu, W. Shadid, and E. Al-Shaer, “Securebert: A domain-specific language model for cybersecurity,” in _International Conference on Security and Privacy in Communication Systems_. Springer, 2022, pp. 39–56.
  * [91] K. Ameri, M. Hempel, H. Sharif, J. Lopez Jr, and K. Perumalla, “Cybert: Cybersecurity claim classification by fine-tuning the bert language model,” _Journal of Cybersecurity and Privacy_ , vol. 1, no. 4, pp. 615–637, 2021.
  * [92] T. Vörös, S. P. Bergeron, and K. Berlin, “Web content filtering through knowledge distillation of large language models,” in _2023 IEEE International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT)_. IEEE, 2023, pp. 357–361.
  * [93] A. Happe and J. Cito, “Getting pwn’d by ai: Penetration testing with large language models,” in _Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2023, pp. 2082–2086.
  * [94] G. Lu, X. Ju, X. Chen, W. Pei, and Z. Cai, “Grace: Empowering llm-based software vulnerability detection with graph structure and in-context learning,” _Journal of Systems and Software_ , p. 112031, 2024.
  * [95] F. Yu and M. V. Martin, “Honey, i chunked the passwords: Generating semantic honeywords resistant to targeted attacks using pre-trained language models,” in _International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment_. Springer, 2023, pp. 89–108.
  * [96] C. Jin, Z. Zhang, X. Jiang, F. Liu, X. Liu, X. Liu, and X. Jin, “Ragcache: Efficient knowledge caching for retrieval-augmented generation,” _CoRR_ , vol. abs/2404.12457, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2404.12457