# Cyber-risk Perception and Prioritization for Decision-Making and Threat
Intelligence

Mario Angelelli  Department of Innovation Engineering, University of Salento
Centre of Applied Mathematics and Physics for Industry, University of Salento
INdAM - Istituto Nazionale di Alta Matematica, GNSAGA  Serena Arima  Centre of
Applied Mathematics and Physics for Industry, University of Salento
Department of Human and Social Sciences, University of Salento  Christian
Catalano  Department of Innovation Engineering, University of Salento  CRLab -
Cybersecurity Research Lab, University of Salento  Enrico Ciavolino  Centre of
Applied Mathematics and Physics for Industry, University of Salento
Department of Human and Social Sciences, University of Salento

###### Abstract

Proactive cyber-risk assessment is gaining momentum due to the wide range of
sectors that can benefit from the prevention of cyber-incidents. The
increasing connectivity of digital and (cyber-)physical systems requires more
attention to cybersecurity to enhance the integrity, confidentiality, and
availability of data.

We introduce a statistical framework for the prioritisation of cyber-
vulnerabilities, using robust and interpretable regression models to support
decision-making. Specifically, we take advantage of mid-quantile regression to
deal with ordinal risk assessments, and we compare it to current alternatives
for cyber-risk ranking and graded responses, identifying a novel accuracy
measure suited for rankings with partial knowledge of existing
vulnerabilities.

Our model is tested on both simulated and real data from selected databases
that support the exploitation of cyber-vulnerabilities in real contexts. The
variety of information arising from such datasets allows us to compare
multiple models based on their predictive performance, showing how accessible
information can influence perception and, hence, decision-making in
operational scenarios. Applications to threat intelligence are discussed too.

Keywords: Cyber-risk; Rank accuracy; Mid-Quantile regression; Risk perception;
MCMC simulations.

##  1  Introduction

Cyber-vulnerabilities of devices, networks, or other kinds of Information and
Communication Technologies (ICTs) can generate system failures or pave the way
for different types of cyber-attacks, including denial-of-service, malware
injection, and data exfiltration. These incidents can also be enhanced by
social engineering, with secondary cascading effects in complex ICTs (system-
of-systems, see e.g. [15]) that can compromise or interrupt service supply,
undermining the operational continuity of critical infrastructures.

New vulnerabilities are emerging from the increasing number of connections
among digital systems, now including personal devices, sensors, (computational
or storage) cloud services, and even vehicles [6], which represent an access
point to other information systems through privilege escalation; the latter
amplifies the severity of cyber-vulnerabilities and represents a weakness when
local access points may lead to violations of classified information at the
national level, as is the case of public administration [8].

Cyber-incidents lead to economic losses, risks to safety, reputational damage,
and violation of personal rights such as privacy, right-to-be-anonymous, and
proper use of personal or sensitive data. The effect of these damages is not
always measurable, due to the intangible nature of social and reputational
effects and the lack of high-quality data, which are often kept secret to
prevent additional reputational issues [17].

_Proactive_ cyber-risk assessment refers to a set of methods, standards,
approaches, and good practises aimed at informed decision-making in the
management of cyber domains, in particular, cyber-vulnerabilities, to prevent
cyber-incidents. Currently, cyber-risk assessment standards are based on
severity levels assessed by institutions, such as the National Institute of
Standards and Technology (NIST) and national Computer Security Incident
Response Teams (CSIRTs). Although NIST provides a harmonised approach to
evaluate the general impact of a cyber-vulnerability, contextual factors (e.g.
exposure to a vulnerable technology and its identifiability) may influence
exploitability both directly and in terms of attackers’ perception, thus
affecting the actual perceived risk. These factors often arise in reserved
reports, data collections, or expert evaluations that are not disclosed. In
addition to this limited knowledge, multiple cyber-vulnerabilities can be
relevant to individuals and organisations, which have to prioritise them in
order to better allocate their cybersecurity (economic, temporal,
professional) resources based on accessible information and personal criteria.

These issues prompt a deeper analysis of the way risk about cyber-
vulnerabilities is perceived and evaluated based on available information:
this leads to the following Research Questions (RQs):

  * (RQ1)

How to _estimate_ risk assessment based on a previously acquired knowledge
base and actual information on new vulnerabilities, without relying on
specific assumptions that could hardly be verified?

  * (RQ2)

How to _measure_ the accuracy of the previous assessment taking into account
the lack of complete information on the set of vulnerabilities?

To answer these questions, we propose a new statistical framework to address
the need for flexible and interpretable models relating to cyber-vulnerability
assessment and their prioritisation, in this way supporting adaptive decision-
making. Flexibility is required to allow different users to adapt the
framework based on the information they have access to, e.g. adding
explanatory variables or considering different response variables based on
their own ranking. Interpretability is needed to prompt appropriate
interventions, e.g. counteractions to fix vulnerabilities or prevent their
exploitation.

As will be detailed in the following sections, the focus on vulnerabilities,
rather than actual incidents, requires appropriate models, in terms of both
estimation procedures and accuracy measures, to deal with uncertainty and
partial information. Specifically, to address RQ1 we adopt mid-quantile
regression as a means to provide robust estimates of ordinal (quantitative and
qualitative) risk assessments of cyber-vulnerabilities depending on available
information (predictors). Regarding the research question RQ2, we introduce a
new accuracy measure that meets an invariance requirement for cyber-
vulnerability severity rankings with respect to unobserved vulnerabilities.

These proposals are tested on both simulated and real data. While the
statistical approach presented in this work is flexible enough to include
other sources of threat beyond cyber-vulnerabilities, the data we consider in
this work do not involve factors such as social engineering, insider threats,
and physical effects (e.g. overload of ICT capacities). However, it is worth
stressing that such factors may be as critical as cybervulnerabilities and may
combine with them in the execution of a cyber-attack [9].

The paper is organised as follows: the notions of cybersecurity and cyber-
vulnerabilities that are relevant for this work are described in the following
Section 2, where we also introduce the required preliminaries on the
statistical models used in the paper, with special reference to rank transform
and mid-quantile regression (Subsection 2.2). Section 3 describes the data
sources that are used for the specification and validation of the proposed
model. Our proposal is presented and motivated in Section 4, also discussing
the appropriate index to assess performance and model comparison suited to our
research questions in the cyber-risk domain. In Section 5, following a
descriptive analysis of the data, we summarise and comment on the results of
simulations and the exploration of the real dataset in terms of prioritisation
of cyber-vulnerabilities. After the discussion of the results in Section 6,
conclusions are drawn in Section 7, where we point out future work and
applications of the present proposal.

##  2 State of the art

###  2.1 Related work on cyber-risk assessments

Cyber-risk assessment is a well-recognised problem that plays a key role in
different domains, e.g. the management of critical infrastructures [31] and
industrial sectors [10]. The diffusion of IoT [32] is opening the way to new
sources of cyber-risk, and also cyber-physical systems and personal devices
require adequate solutions to ensure data protection [4]. On the other hand,
peculiar aspects of operational scenarios may cause limitations of the
efficacy of cyber-risk modelling [31] and, furthermore, require an appropriate
trade-off between the validity of the assessment and usability for decision-
making.

In cyber-risk analysis, one should distinguish between cyber-vulnerability and
cyber-incident: a vulnerability is an access point, but this does not
necessarily entail a cyber-incident, that is, actual (intentional or not)
damage to a digital system. Each known cyber-vulnerability is uniquely
identified by a Common Vulnerability Exposure (CVE) code: in the NIST
classification, the CVE acts as a primary key to retrieve both the impacts, in
terms of CIA dimensions (Confidentiality, Integrity, and Availability), and
the severity assessment of relevant intrinsic characteristics of the
vulnerability. These features are described in detail in the next section.
Focusing on cyber-risk in relation to cyber-vulnerabilities, the current
approach to properly score emerging vulnerabilities is driven by the NIST’s
methodology [33, 21].

In addition to such intrinsic features of cyber-vulnerabilities, other
extrinsic factors affect the risk of cyber-threats, in particular, the
_exposure_ of the technology, which refers to the number of exposed hosts
(devices or systems) for a given CVE, i.e. devices or systems where a given
vulnerability has been recognised. Exposure, together with the availability of
exploits and their cost, concurs to define targets and feasible attacks.

An _exploit_ is defined as a software component, a process, or any human or
physical resource that can be directly executed to perform a cyber-attack: in
this work, we primarily deal with software exploits, but related work also
addresses the role of interactions between malicious software and human
factors in the definition of new attack techniques [35]. We talk about a 0
_-day_ when the vulnerability has not been disclosed before and there are no
solutions available to patch it.

Proactive defence aims at increasing resilience at the individual and network
level (preventing criticalities), preserving individuals and community rights
in the cyber-space (privacy, GDPR compliance, right-to-be-anonymous), and
supporting efficient management of resources and ICT maintenance. In
particular, proactive defence is needed to choose appropriate counteractions
that mitigate the occurrence of cyber-incidents from cyber-vulnerabilities.
There exist several techniques to enhance cybersecurity, including
vulnerability assessment, penetration testing, and static or dynamic analysis
of applications. However, proactive defence is subject to bounded resources:
time constraints, verification costs [34], a specific effort for proprietary
software, limits to automation, and contextual security analysis in highly
connected systems. Therefore, accurate methods to support experts in risk
assessment are a relevant premise for prioritising interventions and, hence,
better use of resources. In this regard, (semi-)automatic tools and
applications based on AI, especially Deep Learning, are gaining increasing
attention as a practical support to detect malware [11]. Unfortunately, they
do not provide complete protection against malware attacks: in a recent work
[9], it is shown that CNN classification could be deceived by masking malware
with a goodware component to bypass automatic controls, also suggesting
feasible counteractions. This approach is called _polymorphism_ and is a
software property that is often used in cyber guerrilla attacks [36]. This
notwithstanding, new approaches are being investigated to take advantage of
deep learning while overcoming some of its limitations, including
explainability [22].

Different research streams are investigated to further support cybersecurity
experts through different methodological or algorithmic techniques, including
applications of fuzzy logic to intrusion detection systems [20], game-
theoretic modelling [29, 13], and stochastic processes toward a successful
attack to evaluate the resilience of a system based on Markov chains [37]. In
line with the research questions stated in the Introduction, here we focus on
interpretable statistical modelling and recently proposed applications to
promote proper cyber-risk assessment and cybersecurity analysis [17, 18],
concentrating on their extension to deal with prioritisation under partial
information on cyber-vulnerability. Two primary sources of uncertainty to take
into account in a preventive approach to cybersecurity have already been
mentioned in the previous discussion, namely, resource limitations in the
implementation of vulnerability assessments and non-disclosure policies to
avoid sharing confidential information on cyber-threats and reputational
losses. In fact, limited data accessibility has been widely recognised as a
relevant issue [17], with an economic impact on estimates [1] and consequent
effects on insurance [7].

To better define the scope of the research, in the following subsections we
introduce the required notions about the statistical models considered in this
work.

###  2.2 Preliminaries on statistical models

Before discussing the two specific models addressed in this work in the
cybersecurity domain, we briefly review the ordered logit model as a benchmark
for regression with ordinal responses [28]. As is well known, it is a
Generalised Linear Model (GLM) suited to deal with cumulative probability
distributions for ordinal responses conditioned on explanatory variables.
Specifically, let $y_{1},\dots,y_{n}$ be a sample of $n$ ordinal responses and
$\mathbf{X}$ be a vector of explanatory variables. The model aims at
describing the effect of covariates on the odds

$\log\frac{P(y\leq h|\mathbf{X})}{P(y\geq
h|\mathbf{X})}=\alpha_{h}-\beta\cdot\mathbf{X},\quad h_{1}\leq
h_{2}\Leftrightarrow\alpha_{h_{1}}\leq\alpha_{h_{2}}$ (2.1)

where $P(y\leq h|\mathbf{X})$ (respectively, $P(y\geq h|\mathbf{X})$ is the
left (respectively, right) cumulative probability associated with the $h$-th
level of the response and conditioned to the observed values $\mathbf{X}$. In
this way, the log-ratio of the odds on the left-hand side depends on the
ordinal level $h$ only through the scale coefficient $\alpha_{h}$, which does
not depend on the variables $\mathbf{X}$ (proportional odds assumption). This
model is well adapted, among all, to qualitative (ordinal) assessments,
including the risk factor of cyber-vulnerabilities, which is described in
Section 3.

We used the ordered logit model as the data generation mechanism in our
simulations, comparing it with two alternative models. Indeed, despite the
wide applicability of ordered logit or probit, more general approaches can be
envisaged to overcome limitations from the potential violation of model
assumptions, e.g. proportional odds in ordered logit.

Another motivation stimulating the research for new methodologies to deal with
ordinal responses is the reduced interpretability of parameter estimates of
GLMs with respect to simpler linear regression; this aspect is relevant in
operational scenarios, where decision-makers should be able to interpret and
quantify the impact of an explanatory variable without assuming background
knowledge on the underlying statistical model. For this reason, Giudici and
Raffinetti [17] proposed the use of a regression model with ordinal responses
in cyber-risk assessment, which we briefly present in the following.

#### Rank transform in linear regression

A recent approach in [17] involves a linear regression model for data
regarding cyber-_incidents_ and is based on the rank transform of a
$n$-dimensional ordinal variable $Y$ with $k$ levels, that is, the set of
ranks for each observation with a given prescription to handle ties [19].
Formally, we move from the ordinal response to the rank-transformed variable
$R(Y)$ defined by

$\displaystyle Y$ $\displaystyle\mapsto$ $\displaystyle
R(Y)\in\left\\{r_{1},r_{2},\dots,r_{k}\right\\},$ $\displaystyle r_{1}=1,$
$\displaystyle r_{h+1}=r_{h}+\\#Y^{(-1)}(\\{h+1\\}),\quad
h\in\\{1,\dots,k-1\\}$ (2.2)

where $\\#Y^{(-1)}(\\{h+1\\})$ denotes the cardinality of the components of
$Y$ that are the pre-image of $h+1$. The fit of the regression model

$R_{i}=\beta_{0}+\beta\cdot{\bf
X}_{i}+\varepsilon_{i},\quad\varepsilon\sim\mathcal{N}(0,\sigma^{2})$ (2.3)

where $\mathcal{N}(0,\sigma^{2})$ is the centred Normal distribution with
variance $\sigma^{2}$, is compared to the _Rank Graduation Accuracy_ (RGA)
[17]

$\mathrm{RGA}:=\sum_{i=1}^{n}\frac{n}{i}\cdot\left(\frac{1}{n\overline{y}}\cdot\sum_{j=1}^{i}y_{\hat{r}_{j}}-\frac{i}{n}\right)^{2}$
(2.4)

where test data $y$ are ranked using the estimated ranks $\hat{r}$ obtained by
fitting (2.3).

As anticipated, the choice of model (2.2)-(2.3) in [17] is argued to provide
more interpretable results supporting decision-making with respect to GLMs. On
the other hand, we note that the use of linear regression with rank transform
may not be suited to deal with cyber-vulnerabilities: contrary to cyber-
incidents, which actually happened, vulnerabilities are subject to different
types of uncertainty mentioned above, especially in the cyber guerrilla
context [36].

From the methodological perspective, this means that several assumptions
underlying the linear regression model may not be fulfilled when dealing with
cyber-vulnerabilities. In particular, linear models rely on the normality
assumption for the residuals, which may not be met in networks of digital
systems; in fact, evidence shows that some relevant features of data breach
datasets are well described by heavy-tail distributions [12]. Even the
homoscedasticity assumption may not be fulfilled and, together with class
unbalancing that we observe in Subsection 5.1, makes the linear model more
sensitive with respect to this violation, while quantile regression does not
assume homoscedasticity.

A final remark comes from a practical requirement, namely, a prioritisation
method that is based on the risk assessment of individual vulnerabilities
(i.e. it is local) to support decision-making based on available information;
the values assumed by the rank transform are informative only related to the
number of observations, which is appropriate for cyber-incidents, but might
not be representative of the full set of vulnerabilities, as argued above.

#### Quantile regression: remarks for cyber-risk assessment

Both the linear and regression models rely on assumptions that may be
unverifiable in real datasets: unbalanced classes, deviations from normality,
and lack of complete knowledge of the space of potential vulnerabilities
(unknown ones or $0$-days) can reduce the effectiveness of regression methods
the aforementioned regression methods. Regarding linear regression, even for
quantitative responses, severe deviations from the normality assumption of the
residuals invalidate the use of linear regression (see the discussion in
Subsection 5.1); as for ordered logit, in the following (see the last
paragraph in Section 6), we will see that the observed behaviour in
simulations generated from such a generalised linear model highly deviates
from the corresponding outcomes obtained from real data. In the specific
cybersecurity domain, such hypotheses may actually be not verifiable, due to
the already mentioned confidentiality and restrictions to data sharing. For
this reason, it is appropriate to consider distribution-free approaches to
make the analysis more robust against violations of statistical assumptions,
leading to quantile regression [23].

Let $Q_{\tau}:=\inf_{y}\\{y:\,\tau\leq F(y))$ be the $\tau$-th quantile of a
random variable with cumulative distribution function (CDF) $F$. Quantile
regression estimates $Q_{\tau}$ conditioning on $k$ regressors

$Q_{\tau}(y_{i}|\mathbf{X}_{i},\beta)=\mathbf{X}_{i}^{\mathtt{T}}\cdot\beta(\tau),\quad
i\in\\{1,\dots,n\\}.$ (2.5)

Estimates $\beta(\tau)$ come from the minimization of the loss function

$\displaystyle\hat{\beta}(\tau)$ $\displaystyle:=$
$\displaystyle\mathrm{argmin}_{\beta\in\mathbb{R}^{k}}\sum_{i=1}^{n}\varrho_{\tau}(y_{i}-\mathbf{X}_{i}^{\mathtt{T}}\cdot\beta),$
$\displaystyle\varrho_{\tau}(u)$ $\displaystyle:=$ $\displaystyle
u\cdot(\tau-\mathbb{I}(u<0))$ (2.6)

where $\mathbb{I}(X)$ is the characteristic function of
$X\subseteq\mathbb{R}$.

In addition to increased robustness against model misspecification, the choice
of quantile regression leads to a new parameter $\tau$ that naturally relates
to the notion of Value-at-Risk (VaR) (also see [32, 7] for a discussion of VaR
in the cybersecurity context), which is in line with the purposes of this
work.

#### Mid-quantile regression

Different estimates can arise from different choices of the quantile level,
which let us compare different rankings or prioritisations at different
quantile levels by looking at parameters associated with regressors. However,
this aspect may lead to ambiguities if it is not properly linked to risk
evaluation and decision-making, e.g. when ranking the attributes associated
with regressors [3]. This leads us to consider quantile regression, where the
response itself assesses the risk or severity of a vulnerability.

Dealing with an ordinal response, we have to extend the quantile regression
approach to discrete variables; for this purpose, we take advantage of _mid-
quantile_ (MidQR hereafter) regression methods. Recent work by Geraci and
Farcomeni [16] applies mid-quantile regression [30] to discrete data: starting
from samples $(X_{i},Y_{i})$ where $Y\sim\mathrm{cat}(p_{h},1\leq h\leq k)$ is
extracted from a categorical distribution,
$\pi_{h}=\frac{1}{2}p_{h}+\sum_{\ell=1}^{h-1}p_{\ell}$, define the _mid-
Cumulative Distribution Function_ $G_{Y}(y):=p(Y\leq y)-\frac{1}{2}p(Y=y)$ and
_mid-quantile function_

$H_{Y}(p)=\int_{0}^{1}\sum_{h=1}^{k}\left((1-\gamma)\cdot y_{h}+\gamma\cdot
y_{h+1}\right)\cdot\delta\left((1-\gamma)\cdot\pi_{h}+\gamma\cdot\pi_{h+1}-p\right)d\gamma.$
(2.7)

Setting $F(y):=p(Y\leq y)$ as before, estimators for unconditioned MidQR are
obtained naturally, i.e. by the substitution of the estimates in the
expression of the mid-quantile function. Such estimators enjoy good asymptotic
consistency and normality for the sampling distribution (see [27, 16] and
references therein).

When $H_{h(Y)|X}(p)=\mathbf{X}^{\mathtt{T}}\cdot\beta(p)$ for a given link
function $h(\cdot)$, we can estimate $\hat{G}_{Y|X}(y|x)$ through a non-
parametric estimator that encompasses both continuous and discrete predictors
[25]:

$\hat{F}_{Y|X}(y|x)=\frac{n^{-1}\cdot\sum_{i=1}^{n}\mathbb{I}(Y_{i}\leq
y)K_{\lambda}(X_{i},x)}{\hat{\delta}_{X}(x)},\quad\hat{m}_{Y|X}(z_{j}|x):=\hat{F}_{Y|X}(z_{j}|x)-\hat{F}_{Y|X}(z_{j-1}|x)$
(2.8)

where $K_{\lambda}(X_{i},x)$ is a kernel function with bandwidth $\lambda$ and
$\hat{\delta}_{X}(x)$ is the kernel estimator of the marginal density of the
explanatory variables $X$. In this way, we can obtain
$\hat{G}_{Y|X}(y|x)=\hat{F}_{Y|X}(y|x)-\frac{1}{2}\cdot\hat{m}_{Y|X}(y|x)$.
Estimates of coefficients $\beta$ follow from the minimization of the
following quadratic loss function

$\mathrm{arg\,min}\psi_{n}(\beta;p),\quad\psi_{n}(\beta;p):=n^{-1}\cdot\sum_{i=1}^{n}\left(p-\hat{G}_{Y|X}(h^{-1}(\mathbf{X}_{i}^{\mathtt{T}}\cdot\beta)\right)^{2}.$
(2.9)

The estimation and fitting procedures can be carried out using the R package
Qtools developed by the authors of [16].

MidQR is used here to deal with ordinal response variables representing risk
factor levels, beyond NIST’s assessment. Together with the statistical model,
an appropriate index should be considered to evaluate its performance and
compare it with other reference models. For this purpose, we adapt RGA (2.4)
to the specific case of cyber-vulnerability assessment, analysing the nature
of the variables involved in the model in terms of criteria to be satisfied in
the decision process. We discuss this aspect in more detail in the following
section.

##  3 Data sources for cyber-risk analysis

###  3.1 Databases

Several databases can be used to assess the cybersecurity of a digital system.
Among the most widely used by practitioners are the following ones:

  * •

the NVD database includes the assessments of vulnerabilities’ severity by the
NIST in terms of data impact dimensions (Confidentiality, Integrity,
Availability) and three additional technical features describing the
accessibility prompted by the cyber-vulnerability, namely, Access Vector (AV),
Access Complexity (AC), and Authentication (Au). The severity assessments of
these six components compose the _attack vector_
111https://nvd.nist.gov/vuln/search.

  * •

the CSIRT database222https://www.csirt.gov.it/contenuti/, which reports
relevant updates on vulnerabilities in line with the evaluation by NIST. Such
reports are communicated by the Italian CSIRT, which is established within the
National Cybersecurity Agency (ACN).

  * •

The Shodan database333https://exposure.shodan.io reports exposed hosts or IP
addresses affected by known vulnerabilities, which may represent a relevant
driver for attackers’ intervention. The Shodan database can be queried by
specifying a CVE and the Country of exposed hosts. Data are collected by the
Shodan monitor platform by combining different techniques, such as crawling,
IP lookups, and metadata analysis.

  * •

Reported exploits for CVEs can be extracted from
ExploitDB444https://www.exploit-db.com/. Information about exploits can be
further refined from VulnDB555https://vuldb.com/, a database that collects
information on the price range of exploits associated with a CVE. The fields
extracted from VulnDB include the 0-day price range, the price at the time of
querying, and the exploitability.

  * •

Tenable666https://www.tenable.com/cve/search risk factor _interprets_ CVSSs
and assigns an ordinal risk factor through threat and vulnerability analysis.
It constitutes qualitative risk information in Tenable’s Vulnerability
Priority Rating (VPR) assessment.

For all these databases, we prepared Python scripts in order to automatically
extract the required data through APIs:

  * •

We started by selecting vulnerabilities identified in Italy through Shodan
(querying for vuln: and country:IT) to obtain a base set of CVEs. Then, the
shodan API was used to extract the exposure data using the api.count function.

  * •

Subsequently, the scripts were adapted to extract the Attack Vectors
associated with these CVEs from the NVD database: for each line/CVE retrieved
in the previous step, the function requests.get on the URL
https://services.nvd.nist.gov/rest/json /cve/1.0/ combined with the CVE code.
This returns a JSON file that can be inspected to get the CVSS scores.

  * •

Then, we checked the availability of the exploits from ExploitDB and VulnDB.
For ExploitDB, we used cve_searchsploit [14] and the function
cve_searchsploit.edbid_ from_cve to obtain the exploits for the selected CVEs.

  * •

In conclusion, a dedicated script was used to obtain Tenable’s risk factor
assessment of the CVEs under consideration: also in this case, by using
requests.get on the URL https://www.tenable.com/cve/’+line+’/plugins combined
with each CVE, we can get the risk factor associated with the CVE re.search
command.

Running these Python scripts, the final dataset for model validation consists
of $n=714$ units. This data extraction procedure is graphically depicted as a
component of the overall analysis (see Figure 4.1 in the next section).

###  3.2 Data description

The above data manipulation procedure leads to a dataset with the following
variables:

  1. 1.

Ordinal regressors extracted from the NIST vulnerability assessment (attack
vector).

  2. 2.

Exposure is a discrete variable that counts exposed hosts, but the variety of
such count data lets us consider a continuous approximation of this variable.

  3. 3.

For each CVE, the existence or absence of an exploit is encoded in a Boolean
variable.

  4. 4.

Risk factors are ordinal responses conditioned by different types of
explanatory variables.

For the present investigation, we selected $p=7$ explanatory variables
returned by the procedure described above, whose interpretation is summarised
in Table 3.1. They are put in connection with an ordinal variable assessing
the risk factor of a cyber-vulnerability, here represented by Tenable’s risk
factor.

Table 3.1: Main attributes of the variables and their interpretation for
statistical modelling. For each set of variables, the data source is provided
in the leftmost column. The quantification for the ordinal assessments of the
components
$X_{\mathrm{C}},X_{\mathrm{I}},X_{\mathrm{A}},X_{\mathrm{AV}},X_{\mathrm{AC}}$
of the attack vector (rightmost column) are provided by the NIST.

Source | Variables | Type | Interpretation | Values
---|---|---|---|---
NIST | $X_{\mathrm{C}}$ |  Discrete Ordinal | Severity for Confidentiality |  $\bullet$ “none: $0$” $\bullet$ “partial: $0.275$” $\bullet$ “complete: $0.660$”
$X_{\mathrm{I}}$ | Severity for Integrity
$X_{\mathrm{A}}$ | Severity for Availability
$X_{\mathrm{AV}}$ |  | Type and severity
---
of the access vector
|  $\bullet$ “Requires local access: $0.395$
---
$\bullet$ “Local Network accessible: $0.646$”
$\bullet$ “Network accessible: $1$”
$X_{\mathrm{AC}}$ |  | Type and severity
---
of access complexity
|  $\bullet$ “high: $0.35$
---
$\bullet$ “medium: $0.61$”
$\bullet$ “low: $0.71$”
Shodan | $N_{\mathrm{exp}}$ | Count data | Number | Integers
ExploitDB | $q_{\mathrm{expl}}$ | Binary | Existence (Boolean) |  $\\{0,1\\}$ (binary)
Tenable | $Y$ |  | Discrete
---
Ordinal
| Risk factor following
---
threat/vulnerability analysis
| “Low”
---
“Medium”,
“High”,
“Critical”

##  4 Contribution and proposed methodology

###  4.1 MidQR for cyber-risk assessment

For our purposes, MidQR is used to provide estimates of the conditional
quantile given a set of regressors that includes both intrinsic vulnerability
characteristics and external variables (exposure and, additionally, exploit
availability). Tenable’s risk factor is our ordinal response variable of
interest. It is worth remarking that quantile regression is also robust
against class unbalance in discrete regressors, which is indeed observed in
real data, as we will see. This section aims at contextualising the ranking
accuracy index within the decision problem under consideration in cyber-
vulnerability assessment.

A quantitative index used in risk factor assessment should enjoy some
invariance properties with respect to different attributions of quantitative
values to each ordinal level so that the accuracy only depends on their order.
This requirement has a practical effect in regression models dealing with
estimated ranking or, more generally, distributions of ordinal variables, such
as the linear model for rank-transformed variables and MidQR. These models
estimate the _conditional_ distributions, and the estimates concern the
quantity

$F_{Y|X}(Y\leq y|\mathbf{x})=\frac{P(Y\leq
y\wedge\mathbf{X}=\mathbf{x})}{P(\mathbf{X}=\mathbf{x})}$

where we focus on regressors $\mathbf{X}$ with a non-zero point mass. This
quantity has an interesting interpretation, as it is a balance of the impact
($P(Y\leq y\wedge\mathbf{X}=\mathbf{x})$) and the rarity
($P(\mathbf{X}=\mathbf{x})$) of the event. As mentioned in the Introduction,
the latter is subject to different forms of uncertainty: among all,
subjectivity in the assessment of impact dimensions $\mathbf{X}$ and, more
importantly, the uncertainty on the representativeness of the sample due to
unknown vulnerabilities, $0$-days, and other situational factors that affect
the identification and severity of a vulnerability, and non-disclosure
policies that may underreport the occurrence of vulnerabilities. This paper
does not aim at modelling these types of measurement errors, whose efficacy
depends on the aspects of cyber-risk under consideration and the level of
investigation, as already mentioned at the beginning of Section 2. However,
the model and the performance index we propose can be discussed in relation to
the aforementioned sources of uncertainty: we postpone this discussion to
Section 6.

###  4.2 A new performance index for cyber-risk estimation

The aforementioned uncertainty about the sample spaces, with consequent
effects on the estimation of the risk factor, is a major driver that prompts
our research for a new approach to evaluate the accuracy of the assessment.

Specifically, the use of quantitative values in (2.4) should take into account
the nature of the variables in the model. The evaluation of (2.4) takes into
account an algebraic structure (formally, the semiring
$(\mathbb{N},+,\cdot,0,1)$ of natural numbers for rankings, or the ordered
field $(\mathbb{R},+,\cdot,0,1)$ from the regression) that is not necessarily
linked to the original ordinal variables assessing the risk factor of a cyber-
vulnerability. This algebraic structure is an artefact suited to the
regression model and, hence, to the estimated variables (let them be the rank
transform or the mid-quantile); the only effect derived from the ordinal
variables is the order defining the summands in (2.4). It is worth noting that
a similar observation also applies in other algebraic-statistical frameworks,
e.g. when dealing with inequivalence of micro- and macro-statistical
descriptions of a physical system [2].

Starting from the previous comments, we now introduce a novel goodness-of-fit
index more suited to accommodate the characteristics of cyber-vulnerability
data. We consider a _reverse_ RGA index, which we refer to as AGR, defined as
$\mathrm{RGA}(r_{\mathrm{tr}},r_{\mathrm{est}})$, namely, we exchange the
roles of the estimated $r_{\mathrm{est}}$ and the “true” $r_{\mathrm{tr}}$
rankings.

To better appreciate the need for appropriate use of the RGA index for
unconventional cyber-risk assessment, we consider the case of sub-sampling,
i.e. known subsets of an unknown family of cyber-vulnerabilities. This
emulates the partial information available on $0$-days.

######  Example 1.

We can consider the following $5$-dimensional rank vectors:

$c_{\mathrm{est}}:=(1,3,2,2.9,10),\quad c_{\mathrm{tr},1}:=(1,3,2,2,9),\quad
c_{\mathrm{tr},2}:=(1,5,3,3,7)$ (4.1)

where $c_{\mathrm{est}}$ derives from a given estimation procedure, while
$c_{\mathrm{tr},u}$, $u\in\\{1,2\\}$, are two “true” rankings obtained from
different knowledge about the state of a digital system and its sample space.
Although they are different, the rankings $c_{\mathrm{tr},1}$ and
$c_{\mathrm{tr},2}$ are consistent with the same attribution of ordinal
levels: for the sake of concreteness, we can assume that the components of
both $c_{\mathrm{tr},1}$ and $c_{\mathrm{tr},2}$ are generated by ranking the
same ordinal assessment (“$10$”,“$6$”,“$8$”,“$8$”,“$3$”), where risk factor
levels are ordered from “$10$” to “$1$”. In this case, the differences between
$c_{\mathrm{tr},1}$ and $c_{\mathrm{tr},2}$ can arise from the existence of
other elements in the two ranked sample spaces, beyond those associated with
the components of $c_{\mathrm{tr},1}$ and $c_{\mathrm{tr},2}$. The evaluation
of $\mathrm{RGA}(y_{\mathrm{est}},y_{\mathrm{tr},u})$ for $u\in\\{1,2\\}$
following the definition (2.4) does not satisfy invariance under changes of
rankings that are generated by the same ordinal assessment. Indeed, we have

$\mathrm{RGA}(c_{\mathrm{est}},c_{\mathrm{tr},1})=0.5161\neq
0.3232=\mathrm{RGA}(c_{\mathrm{est}},c_{\mathrm{tr},2}).$ (4.2)

On the other hand, we find

$\mathrm{AGR}(c_{\mathrm{est}},c_{\mathrm{tr},1})=\mathrm{RGA}(c_{\mathrm{tr},1},c_{\mathrm{est}})=0.5272=\mathrm{RGA}(c_{\mathrm{tr},2},c_{\mathrm{est}})=\mathrm{AGR}(c_{\mathrm{est}},c_{\mathrm{tr},2}).$
(4.3)

It is clear that the latter equality holds for all the choices of
$c_{\mathrm{est}},c_{\mathrm{tr,1}},c_{\mathrm{tr},2}$.

This shows that the AGR index resolves the lack of invariance under sub-
sampling in RGA. The favourable invariance of the AGR index under rank
transformations that are compatible with the same underlying ordinal
assessment is in line with Luce’s axiom of Independence of Irrelevant
Alternatives [26], while some algebraic conditions related to this type of
symmetry have been discussed in a physical setting in [2]. Practically, this
invariance is required when dealing with partial information about the space
of potential cyber-vulnerabilities, which is the general situation faced by a
decision-maker, due to the occurrence of unknown vulnerabilities not exploited
yet, $0$-days, and _unconventional_ cyber-attacks [36, 35].

The description of the full experimental procedure that will be carried out to
validate the proposal and investigate its scope of applicability is shown in
Figure 4.1.

Figure 4.1: Graphical description of the experiments to validate the
efficiency of mid-quantile regression for risk-factor estimates and AGR as an
accuracy index of predicted risk levels.

##  5  Experiments and results

###  5.1 Descriptive analysis of the dataset

Data extracted from the databases described in Section 3 select $n=714$ cyber-
vulnerabilities in Italy. The time span of the CVEs is 1999-2021.

We note that each variable in the attack vector is characterised by manifest
unbalancing among the different levels, as shown in Figures 1(a)-1(b).

(a) Distribution of impact dimension levels

(b) Distribution of features (AV, AC) and risk factor

Figure 5.1: Distribution of levels of variables from the cyber-vulnerability
dataset.

When the response in a regression model is (well approximated by) a continuous
variable, then unbalancing could make linear regression more sensitive to
deviations from homoscedasticity, hence quantile regression could be
favourable. This is the case when the exposure of vulnerable hosts is related
to intrinsic features of the vulnerabilities [3]: it is easily checked from
the QQ-plots in Figures 2(a)-2(b) that the residuals of the exposure
$N_{\mathrm{exp}}$ and its log-transform $\log(1+N_{\mathrm{exp}})$,
considered as responses in a linear model with regressors
$(X_{\mathrm{C}},X_{\mathrm{I}},X_{\mathrm{A}},X_{\mathrm{AV}},X_{\mathrm{AC}})$,
show strong deviations from normality.

(a) Regressors
$(X_{\mathrm{C}},X_{\mathrm{I}},X_{\mathrm{A}},X_{\mathrm{AV}},X_{\mathrm{AC}})$.

(b) Free model.

Figure 5.2: QQ-plots of the theoretical (Normal) quantiles compared to the
empirical quantiles of residuals of $y=\log(1+N_{\mathrm{exp}})$ derived from
the exposure $N_{\mathrm{exp}}$ of cyber-vulnerabilities.

This remark also entails that linear regression would not fit the distribution
assumptions when a proxy of cyber-risk such as exposure is used as the
response. We also note that even the residuals of the “free model”, i.e. the
QQ-plot of the exposure $N_{\mathrm{exp}}$ itself, violate the normality
assumption (see Figure 2(b)). The use of the transform
$N_{\mathrm{exp}}\mapsto\log(N_{\mathrm{exp}}+1)$ in the previous QQ-plots
slightly reduces the deviation from normality: more importantly, it highlights
multimodality in the distribution of exposure, as it is manifest in the
histograms depicted in Figures 3(a)-3(b).

(a) $y=N_{\mathrm{exp}}$.

(b) $y=\log(1+N_{\mathrm{exp}})$.

Figure 5.3: Histograms for the empirical distributions of the exposure
$N_{\mathrm{exp}}$ compared to $\log(1+N_{\mathrm{exp}})$. The corresponding
continuous approximations (red dashed lines) highlight multimodality.

This suggests the need to go beyond linear models for an appropriate
description of external characteristics of cyber-vulnerabilities, starting
from their intrinsic (attack vector) and extrinsic (exposure, exploits)
features as regressors.

###  5.2 Rankings and mid-quantile regression

#### Simulation study

We start by specifying the preliminary simulation study to provide a general
comparative analysis between the model presented in [17] and the MidQR.

  * •

We used $n_{tr}=320$ units for training and $n_{test}=80$ units for testing
the accuracy performance of the models. We started with a response variable
having $k=4$ levels, in line with Tenable’s risk factor that is used in the
analysis of real data. However, we also tested $k\in\\{3,6,8\\}$ to evaluate
the behaviour and performance of the different models when the number of
levels of the response variable changes.

  * •

Two continuous and two discrete explanatory variables were considered, each of
the latter having three categories. This induced $P:=2+2\cdot(3-1)=6$
regressors after moving to ANOVA variables.

  * •

Following the generation of the so-specified variables, we considered the
parameters $\alpha_{h}$, $h\in\\{1,\dots,k-1\\}$ and $\beta_{p}$,
$p\in\\{1,\dots,P\\}$ to obtain the corresponding probabilities based on the
ordered logit model (2.1).

  * •

This scheme was iterated to obtain $n_{iter}=100$ samples of the response
variable $Y$.

In this way, we got the coefficient estimates and the mean, over the
simulation runs, of the standard error (SE) estimates for each coefficient.
For MidQR, we adapted a function in Qtools to overcome computational issues in
the estimation of the conditional (mid-)CDF, which involves the kernel method
based on [24]. Specifically, we acted on the estimation of the covariance
matrix of the coefficients, making its computation compatible with cases where
the quantile level lies outside the range of the sample mid-CDF. However, the
outcomes of this procedure, which is analogous to censoring, may lead to an
overestimation of the SE obtained from the kernel method. For this reason, we
also present two additional indicators that provide information on the SE:
“Regular” Standard Error (Reg.SE) of each parameter, which is defined as the
average SE over the simulation runs where the parameter is significant at a
given level (here, $0.05$); Monte Carlo Standard Error (MCSE), that is, the
standard error calculated from the coefficient estimates. Finally, the
percentage of iteration runs where a given parameter is statistically
significant at level $0.05$ is reported (% sign.).

The analysis compares the three models under consideration, namely, the data-
generating model (ordered logit), linear regression for rank-transformed
variables, and mid-quantile regression with
$\tau\in\\{0.1,0.3,0.5,0.7,0.9\\}$. For each iteration, the RGA and AGR
indices were evaluated on the test dataset. The same analysis was subsequently
carried out with the real dataset, in order to compare the relative
performance of linear regression for rank-transformed variables and mid-
quantile regression based on real evidence.

The use of both continuous and discrete regressors mimics the occurrence of
exposure (considered continuous) and attack vector components (discrete
variables). We generated

$\mathbf{X}_{(cont)}\sim\mathcal{N}(\mu,\sigma),\quad\mathbf{X}_{(cat)}\sim
p(\pi_{1},\pi_{2})$ (5.1)

where $\mathcal{N}(\mu,\sigma)$ is the normal distribution with mean $\mu=0$
and variance $\sigma^{2}=1$; $p(\pi_{1},\pi_{2})$ is the categorical
distribution with three support points associated with probability weights
$\pi_{1},\pi_{2},1-\pi_{1}-\pi_{2}>0$, in particular, we chose
$\pi_{1}=\pi_{2}=\frac{1}{3}$. Then, the responses $Y_{i}$,
$i\in\\{1,\dots,n\\}$, were extracted from a categorical distribution with
probability derived from (2.1), i.e.

$p(Y=h|\mathbf{X})=P(Y\leq h|\mathbf{X})-P(Y\leq h-1|\mathbf{X}),\quad
h\in\\{1,\dots,k\\}.$ (5.2)

Multiple simulation runs were performed at different choices of
$\beta_{\mathrm{true}}$ with different quantile levels.

#### Simulation results

We start presenting the results of simulations where the response variable
contains $k=4$ possible levels: as mentioned above, this situation is in line
with the real dataset structure since the Tenable risk factor involves $k=4$
levels too.

In Tables 5.1-5.2 we show the outcomes from two different scenarios: the
parameters defining the theoretical distribution from the Ordered Logit model
can be tuned to obtain the uniform probability distribution on the $k$
response levels (Table 5.1) or they can be chosen generically; in the latter
case, we can get a non-uniform distribution (Table 5.2). In the tables, we
report the estimates of the model parameters (Est) and the corresponding
standard errors (SE) averaged over $100$ simulations. We also report the Monte
Carlo standard error (MCSE) in order to evaluate the stability of the
estimates over the simulations. For LinReg and MidQR, we report the percentage
of times the parameters resulted significant at 5% level (% sign.).

Table 5.1: Coefficient estimates from simulations with $k=4$ levels for the response variable; parameters in the generative model are tuned in order to get the uniform probability distribution on the $k$ possible response levels. |  | $\mathbf{X_{3}}$ | $\mathbf{X_{4}}$ | $\mathbf{X_{1}}$ | $\mathbf{X_{2}}$ | Intercept
---|---|---|---|---|---|---
|  | 1 | 2 | 1 | 2
OrdReg | Est | -3.097 | 2.094 | 1.017 | 4.141 | -2.062 | 4.227 |
SE | 0.312 | 0.244 | 0.368 | 0.530 | 0.402 | 0.540 |
MCSE | 0.032 | 0.029 | 0.033 | 0.052 | 0.042 | 0.050 |
LinReg | Est | -37.012 | 24.156 | 14.856 | 44.762 | -27.017 | 46.337 | 98.235
SE | 2.947 | 2.818 | 7.173 | 7.107 | 7.280 | 7.302 | 6.823
MCSE | 0.230 | 0.235 | 0.566 | 0.626 | 0.651 | 0.544 | 0.489
% sign. | 100.0% | 100.0% | 55.0% | 100.0% | 99.0% | 100.0% | 100.0%
MidQR($\tau_{1}$) | Est | -0.238 | 0.156 | 0.038 | 0.359 | -0.146 | 0.482 | 0.291
SE | 2.896 | 2.466 | 7.227 | 6.083 | 7.972 | 6.670 | 7.338
Reg.SE | 0.036 | 0.035 | N.D. | 0.086 | 0.092 | 0.090 | 0.089
MCSE | 0.002 | 0.002 | 0.004 | 0.007 | 0.007 | 0.007 | 0.007
% sign. | 71.0% | 71.0% | 0.0% | 70.0% | 19.0% | 71.0% | 66.0%
MidQR($\tau_{2}$) | Est | -0.274 | 0.168 | 0.058 | 0.359 | -0.184 | 0.433 | 0.563
SE | 1.283 | 1.192 | 3.365 | 2.648 | 3.563 | 3.178 | 3.150
Reg.SE | 0.025 | 0.024 | 0.061 | 0.060 | 0.066 | 0.062 | 0.061
MCSE | 0.002 | 0.002 | 0.005 | 0.006 | 0.008 | 0.006 | 0.006
% sign. | 71.0% | 71.0% | 12.0% | 71.0% | 57.0% | 71.0% | 71.0%
MidQR($\tau_{3}$) | Est | -0.270 | 0.163 | 0.046 | 0.300 | -0.188 | 0.344 | 0.827
SE | 705.709 | 340.703 | 372.360 | 1024.919 | 578.466 | 1078.914 | 520.001
Reg.SE | 0.022 | 0.021 | 0.058 | 0.056 | 0.061 | 0.056 | 0.057
MCSE | 0.002 | 0.002 | 0.004 | 0.005 | 0.007 | 0.005 | 0.006
% sign. | 54.0% | 54.0% | 7.0% | 54.0% | 48.0% | 54.0% | 54.0%
MidQR($\tau_{4}$) | Est | -0.202 | 0.117 | 0.029 | 0.193 | -0.144 | 0.213 | 1.057
SE | 1.267 | 1.148 | 2.258 | 3.299 | 2.433 | 3.350 | 2.410
Reg.SE | 0.029 | 0.027 | N.D. | 0.067 | 0.077 | 0.067 | 0.074
MCSE | 0.001 | 0.002 | 0.003 | 0.004 | 0.006 | 0.004 | 0.005
% sign. | 71.0% | 70.0% | 0.0% | 66.0% | 30.0% | 67.0% | 71.0%
MidQR($\tau_{5}$) | Est | -0.125 | 0.075 | 0.001 | 0.086 | -0.097 | 0.085 | 1.262
SE | 3.237 | 2.428 | 5.298 | 7.221 | 5.278 | 8.288 | 6.373
Reg.SE | 0.040 | 0.034 | N.D. | 0.077 | 0.094 | 0.073 | 0.100
MCSE | 0.001 | 0.001 | 0.002 | 0.003 | 0.004 | 0.003 | 0.004
% sign. | 68.0% | 46.0% | 0.0% | 2.0% | 1.0% | 1.0% | 71.0%
|  |  |  |  |  |  |  |
Table 5.2: Coefficient estimates from simulations with $k=4$ levels for the response variable; generic parameters in the generative model leading to a non-uniform probability distribution on the $k$ possible response levels. |  | $\mathbf{X_{3}}$ | $\mathbf{X_{4}}$ | $\mathbf{X_{1}}$ | $\mathbf{X_{2}}$ | Intercept
---|---|---|---|---|---|---
|  | 1 | 2 | 1 | 2
OrdReg | Est | -3.123 | 2.061 | 1.012 | 4.064 | -2.102 | 4.076 |
SE | 0.289 | 0.226 | 0.361 | 0.494 | 0.391 | 0.495 |
MCSE | 0.030 | 0.021 | 0.040 | 0.052 | 0.040 | 0.052 |
LinReg | Est | -46.974 | 28.359 | 18.304 | 59.905 | -34.439 | 54.792 | 102.372
SE | 2.901 | 2.884 | 6.938 | 7.136 | 7.185 | 7.099 | 6.381
MCSE | 0.269 | 0.225 | 0.670 | 0.612 | 0.645 | 0.609 | 0.506
% sign. | 100.0% | 100.0% | 78.0% | 100.0% | 100.0% | 100.0% | 100.0%
MidQR($\tau_{1}$) | Est | -0.311 | 0.166 | 0.083 | 0.385 | -0.140 | 0.453 | 0.288
SE | 3.032 | 2.475 | 6.167 | 6.780 | 7.080 | 6.875 | 7.375
Reg.SE | 0.036 | 0.034 | 0.079 | 0.086 | 0.086 | 0.088 | 0.084
MCSE | 0.002 | 0.002 | 0.004 | 0.006 | 0.006 | 0.006 | 0.006
% sign. | 72.0% | 72.0% | 2.0% | 72.0% | 18.0% | 72.0% | 66.0%
MidQR($\tau_{2}$) | Est | -0.316 | 0.178 | 0.064 | 0.392 | -0.172 | 0.440 | 0.552
SE | 1.214 | 1.111 | 2.664 | 2.707 | 2.776 | 2.612 | 2.566
Reg.SE | 0.023 | 0.023 | 0.057 | 0.057 | 0.061 | 0.058 | 0.056
MCSE | 0.002 | 0.002 | 0.004 | 0.005 | 0.006 | 0.006 | 0.005
% sign. | 72.0% | 72.0% | 13.0% | 72.0% | 57.0% | 72.0% | 72.0%
MidQR($\tau_{3}$) | Est | -0.285 | 0.161 | 0.055 | 0.347 | -0.188 | 0.372 | 0.797
SE | 1.303 | 1.756 | 2.397 | 2.540 | 2.926 | 2.497 | 3.089
Reg.SE | 0.021 | 0.021 | 0.052 | 0.053 | 0.057 | 0.052 | 0.053
MCSE | 0.002 | 0.002 | 0.004 | 0.004 | 0.006 | 0.005 | 0.005
% sign. | 72.0% | 72.0% | 7.0% | 72.0% | 68.0% | 72.0% | 72.0%
MidQR($\tau_{4}$) | Est | -0.202 | 0.114 | 0.038 | 0.244 | -0.147 | 0.249 | 1.023
SE | 1.413 | 1.321 | 2.591 | 3.351 | 2.722 | 3.631 | 2.508
Reg.SE | 0.027 | 0.026 | 0.065 | 0.065 | 0.073 | 0.062 | 0.067
MCSE | 0.001 | 0.001 | 0.003 | 0.003 | 0.005 | 0.004 | 0.004
% sign. | 72.0% | 72.0% | 1.0% | 72.0% | 43.0% | 71.0% | 72.0%
MidQR($\tau_{5}$) | Est | -0.113 | 0.062 | 0.022 | 0.132 | -0.114 | 0.115 | 1.231
SE | 2.867 | 2.164 | 3.590 | 5.085 | 4.094 | 7.379 | 4.220
Reg.SE | 0.038 | 0.034 | N.D. | 0.078 | 0.094 | 0.073 | 0.090
MCSE | 0.002 | 0.001 | 0.002 | 0.003 | 0.004 | 0.003 | 0.004
% sign. | 68.0% | 25.0% | 0.0% | 9.0% | 2.0% | 8.0% | 72.0%
|  |  |  |  |  |  |  |

The resulting RGA and AGR indices are reported in Table 5.3. To provide an
informative view of RGA and AGR, we present the boxplots associated with each
model in Figure 5.4.

Table 5.3: RGA and AGR from simulations with $k=4$ levels in the response variable; columns 2-5 are generated from a model tuned to produce uniform probabilities for the $k$ levels in the response. |  $k=4$, uniform |  $k=4$, non-uniform
---|---|---
RGA | AGR | RGA | AGR
Est | SD | Est | SD | Est | SD | Est | SD
OrdLog | 2.517 | 0.496 | 2.823 | 0.507 | 5.889 | 0.897 | 6.494 | 0.723
LinReg | 3.276 | 0.578 | 1.516 | 0.193 | 6.762 | 0.796 | 3.254 | 0.282
MidQR($\tau_{1}$) | 3.093 | 0.551 | 3.016 | 0.394 | 6.600 | 0.767 | 4.316 | 0.348
MidQR($\tau_{2}$) | 3.212 | 0.555 | 3.143 | 0.391 | 6.657 | 0.768 | 4.356 | 0.342
MidQR($\tau_{3}$) | 3.239 | 0.562 | 3.214 | 0.389 | 6.684 | 0.773 | 4.377 | 0.343
MidQR($\tau_{4}$) | 3.193 | 0.565 | 3.207 | 0.398 | 6.670 | 0.797 | 4.371 | 0.349
MidQR($\tau_{5}$) | 3.016 | 0.573 | 3.146 | 0.418 | 6.491 | 0.862 | 4.276 | 0.370
Self | 4.299 | 0.614 | 4.299 | 0.614 | 8.614 | 0.677 | 8.614 | 0.677
|  |  |  |  |  |  |  |

(a) RGA, $k=4$, uniform distribution.

(b) AGR, $k=4$, uniform distribution.

(c) RGA, $k=4$, non-uniform distribution.

(d) AGR, $k=4$, non-uniform distribution.

Figure 5.4: Boxplots for RGA and AGR when $k=4$; both uniform and non-uniform
probability distributions are considered starting from the data-generating
OrdLog model. Boxplots refer, from left to right of the x-axis, to OrdLog,
LinReg, MidQR with $\tau$ taking values $0.1,0.3,0.5,0.7,0.9$,  and the
reference value $\mathrm{RGA}(r_{\mathrm{true}},r_{\mathrm{true}})$.

Then, we move to different numbers of levels in order to better assess the
behaviour of the different methods in different decision scenarios. We address
this aspect first considering $k=3$: this is a typical scale in several
operational or tactical decisions, where levels are generally interpreted as
“low”, “medium”, and “high”, respectively. The outcomes of this set of
simulations are presented in Table 5.4.

Table 5.4: Coefficient estimates from simulations with $k=3$ levels for the response variable. |  | $\mathbf{X_{3}}$ | $\mathbf{X_{4}}$ | $\mathbf{X_{1}}$ | $\mathbf{X_{2}}$ | Intercept
---|---|---|---|---|---|---
|  | 1 | 2 | 1 | 2
OrdReg | Est | -3.173 | 2.083 | 1.053 | 4.249 | -2.086 | 4.193 |
SE | 0.395 | 0.298 | 0.466 | 0.745 | 0.499 | 0.755 |
MCSE | 0.038 | 0.028 | 0.050 | 0.072 | 0.042 | 0.082 |
LinReg | Est | -23.122 | 15.755 | 9.877 | 28.192 | -17.554 | 24.575 | 74.764
SE | 1.825 | 1.827 | 4.439 | 4.732 | 4.609 | 4.568 | 4.152
MCSE | 0.199 | 0.168 | 0.379 | 0.403 | 0.395 | 0.346 | 0.418
% sign. | 100.0% | 100.0% | 69.0% | 100.0% | 99.0% | 100.0% | 100.0%
MidQR($\tau_{1}$) | Est | -0.195 | 0.114 | 0.038 | 0.270 | -0.129 | 0.291 | 0.341
SE | 12.519 | 16.381 | 27.625 | 34.324 | 37.952 | 25.530 | 31.305
Reg.SE | 0.027 | 0.028 | N.D. | 0.072 | 0.072 | 0.071 | 0.070
MCSE | 0.002 | 0.002 | 0.003 | 0.004 | 0.004 | 0.005 | 0.004
% sign. | 70.0% | 69.0% | 0.0% | 70.0% | 25.0% | 70.0% | 70.0%
MidQR($\tau_{2}$) | Est | -0.218 | 0.138 | 0.047 | 0.259 | -0.133 | 0.277 | 0.550
SE | 8.409 | 6.741 | 17.770 | 18.895 | 19.943 | 18.942 | 16.774
Reg.SE | 0.019 | 0.019 | 0.049 | 0.050 | 0.054 | 0.048 | 0.049
MCSE | 0.001 | 0.002 | 0.003 | 0.004 | 0.005 | 0.004 | 0.004
% sign. | 70.0% | 70.0% | 8.0% | 70.0% | 50.0% | 70.0% | 70.0%
MidQR($\tau_{3}$) | Est | -0.206 | 0.134 | 0.056 | 0.219 | -0.133 | 0.222 | 0.765
SE | 753.120 | 344.070 | 171.833 | 819.444 | 253.610 | 970.775 | 573.024
Reg.SE | 0.019 | 0.018 | 0.046 | 0.046 | 0.050 | 0.042 | 0.045
MCSE | 0.002 | 0.002 | 0.003 | 0.004 | 0.005 | 0.004 | 0.004
% sign. | 60.0% | 60.0% | 16.0% | 60.0% | 49.0% | 60.0% | 60.0%
MidQR($\tau_{4}$) | Est | -0.129 | 0.087 | 0.040 | 0.121 | -0.086 | 0.116 | 0.924
SE | 22.573 | 11.780 | 28.125 | 42.421 | 27.902 | 57.145 | 31.146
Reg.SE | 0.028 | 0.024 | N.D. | 0.058 | 0.061 | 0.053 | 0.061
MCSE | 0.001 | 0.001 | 0.002 | 0.003 | 0.004 | 0.003 | 0.003
% sign. | 69.0% | 67.0% | 0.0% | 43.0% | 13.0% | 25.0% | 70.0%
MidQR($\tau_{5}$) | Est | -0.061 | 0.042 | 0.029 | 0.045 | -0.045 | 0.036 | 1.036
SE | 48.199 | 25.136 | 34.366 | 61.267 | 41.685 | 82.775 | 74.481
Reg.SE | 0.030 | 0.027 | N.D. | 0.060 | N.D. | N.D. | 0.119
MCSE | 0.001 | 0.001 | 0.002 | 0.002 | 0.003 | 0.001 | 0.002
% sign. | 20.0% | 10.0% | 0.0% | 1.0% | 0.0% | 0.0% | 70.0%
|  |  |  |  |  |  |  |

The corresponding RGA and AGR indices are shown in Table 5.2. Also in this
case, we provide a graphical representation of these outcomes in Figure 5.5.

Table 5.5: RGA and AGR from simulations with a low number $k=3$ of levels for
the response variable.

| RGA | AGR
---|---|---
Est | SD | Est | SD
OrdLog | 1.439 | 0.488 | 1.545 | 0.538
LinReg | 2.203 | 0.667 | 0.865 | 0.169
MidQR($\tau_{1}$) | 2.113 | 0.677 | 2.733 | 0.487
MidQR($\tau_{2}$) | 2.193 | 0.677 | 2.871 | 0.473
MidQR($\tau_{3}$) | 2.162 | 0.677 | 2.883 | 0.470
MidQR($\tau_{4}$) | 2.082 | 0.667 | 2.848 | 0.470
MidQR($\tau_{5}$) | 1.785 | 0.616 | 2.631 | 0.468
Self | 3.499 | 0.819 | 3.499 | 0.819
|  |  |  |

(a) RGA, $k=3$.

(b) AGR, $k=3$.

Figure 5.5: Boxplots for RGA and AGR when $k=3$. Boxplots refer, from left to
right of the x-axis, to OrdLog, LinReg, MidQR with $\tau$ taking values
$0.1,0.3,0.5,0.7,0.9$,  and
$\mathrm{RGA}(r_{\mathrm{true}},r_{\mathrm{true}})$.

Finally, we complete the simulation study considering more than $4$ levels in
the response variable. Specifically, we report the results at $k=6$ (Table
5.7) and $k=8$ (Table 5.8), with the associated boxplots in Figure 5.6.

Table 5.7: Coefficient estimates from simulations with $k=6$ levels for the response variable. |  | $\mathbf{X_{3}}$ | $\mathbf{X_{4}}$ | $\mathbf{X_{1}}$ | $\mathbf{X_{2}}$ | Intercept
---|---|---|---|---|---|---
|  | 1 | 2 | 1 | 2
OrdReg | Est | -3.116 | 2.064 | 1.046 | 4.120 | -2.074 | 4.094 |
SE | 0.237 | 0.179 | 0.306 | 0.407 | 0.335 | 0.394 |
MCSE | 0.024 | 0.015 | 0.029 | 0.037 | 0.035 | 0.040 |
LinReg | Est | -61.725 | 41.627 | 23.455 | 76.997 | -48.392 | 80.101 | 108.517
SE | 3.038 | 2.943 | 7.577 | 7.588 | 7.754 | 7.355 | 6.525
MCSE | 0.202 | 0.230 | 0.635 | 0.603 | 0.716 | 0.624 | 0.521
% sign. | 100.0% | 100.0% | 89.0% | 100.0% | 100.0% | 100.0% | 100.0%
MidQR($\tau_{1}$) | Est | -0.347 | 0.217 | 0.007 | 0.387 | -0.203 | 0.532 | 0.366
SE | 0.821 | 0.717 | 2.078 | 2.096 | 2.573 | 1.958 | 2.207
Reg.SE | 0.033 | 0.033 | N.D. | 0.084 | 0.090 | 0.084 | 0.078
MCSE | 0.001 | 0.002 | 0.004 | 0.006 | 0.005 | 0.006 | 0.004
% sign. | 89.0% | 89.0% | 0.0% | 89.0% | 58.0% | 89.0% | 89.0%
MidQR($\tau_{2}$) | Est | -0.342 | 0.230 | 0.075 | 0.404 | -0.254 | 0.518 | 0.597
SE | 0.388 | 0.395 | 0.984 | 0.993 | 1.054 | 0.944 | 0.935
Reg.SE | 0.023 | 0.022 | 0.056 | 0.058 | 0.063 | 0.055 | 0.051
MCSE | 0.001 | 0.002 | 0.004 | 0.005 | 0.005 | 0.005 | 0.004
% sign. | 89.0% | 89.0% | 18.0% | 89.0% | 89.0% | 89.0% | 89.0%
MidQR($\tau_{3}$) | Est | -0.314 | 0.213 | 0.089 | 0.363 | -0.230 | 0.437 | 0.830
SE | 2.967 | 2.491 | 2.748 | 5.410 | 1.992 | 6.140 | 4.738
Reg.SE | 0.023 | 0.021 | 0.053 | 0.053 | 0.062 | 0.049 | 0.052
MCSE | 0.002 | 0.002 | 0.004 | 0.005 | 0.005 | 0.004 | 0.004
% sign. | 81.0% | 80.0% | 31.0% | 81.0% | 83.0% | 80.0% | 85.0%
MidQR($\tau_{4}$) | Est | -0.253 | 0.173 | 0.080 | 0.285 | -0.192 | 0.337 | 1.077
SE | 0.320 | 0.293 | 0.689 | 0.675 | 0.695 | 0.688 | 0.555
Reg.SE | 0.026 | 0.025 | 0.066 | 0.064 | 0.075 | 0.057 | 0.064
MCSE | 0.001 | 0.002 | 0.003 | 0.004 | 0.005 | 0.003 | 0.004
% sign. | 89.0% | 89.0% | 8.0% | 89.0% | 69.0% | 89.0% | 89.0%
MidQR($\tau_{5}$) | Est | -0.185 | 0.130 | 0.059 | 0.186 | -0.131 | 0.219 | 1.347
SE | 0.500 | 0.409 | 0.962 | 1.050 | 0.936 | 1.179 | 0.846
Reg.SE | 0.036 | 0.035 | N.D. | 0.086 | 0.103 | 0.075 | 0.090
MCSE | 0.001 | 0.002 | 0.003 | 0.004 | 0.005 | 0.003 | 0.004
% sign. | 89.0% | 89.0% | 0.0% | 58.0% | 6.0% | 88.0% | 89.0%
|  |  |  |  |  |  |  |
Table 5.8: Coefficient estimates from simulations with $k=8$ levels for the response variable. |  | $\mathbf{X_{3}}$ | $\mathbf{X_{4}}$ | $\mathbf{X_{1}}$ | $\mathbf{X_{2}}$ | Intercept
---|---|---|---|---|---|---
|  | 1 | 2 | 1 | 2
OrdReg | Est | -3.062 | 2.053 | 1.008 | 4.047 | -2.045 | 4.040 |
SE | 0.217 | 0.170 | 0.289 | 0.373 | 0.305 | 0.363 |
MCSE | 0.021 | 0.019 | 0.027 | 0.033 | 0.031 | 0.037 |
LinReg | Est | -67.507 | 44.804 | 24.680 | 90.220 | -44.497 | 92.607 | 111.317
SE | 2.987 | 2.963 | 7.007 | 7.248 | 6.947 | 6.999 | 6.752
MCSE | 0.202 | 0.301 | 0.634 | 0.614 | 0.650 | 0.613 | 0.559
% sign. | 100.0% | 100.0% | 95.0% | 100.0% | 100.0% | 100.0% | 100.0%
MidQR($\tau_{1}$) | Est | -0.414 | 0.241 | 0.127 | 0.585 | -0.267 | 0.596 | 0.488
SE | 6.789 | 2.250 | 12.100 | 12.866 | 6.430 | 6.874 | 12.916
Reg.SE | 0.036 | 0.037 | 0.087 | 0.091 | 0.093 | 0.085 | 0.094
MCSE | 0.002 | 0.002 | 0.004 | 0.007 | 0.008 | 0.007 | 0.007
% sign. | 73.0% | 73.0% | 14.0% | 73.0% | 61.0% | 73.0% | 73.0%
MidQR($\tau_{2}$) | Est | -0.409 | 0.266 | 0.095 | 0.537 | -0.271 | 0.525 | 0.811
SE | 1.017 | 0.950 | 2.209 | 2.224 | 2.397 | 2.045 | 2.049
Reg.SE | 0.025 | 0.024 | 0.061 | 0.061 | 0.062 | 0.057 | 0.062
MCSE | 0.002 | 0.002 | 0.004 | 0.006 | 0.007 | 0.005 | 0.006
% sign. | 74.0% | 74.0% | 23.0% | 74.0% | 73.0% | 74.0% | 74.0%
MidQR($\tau_{3}$) | Est | -0.363 | 0.250 | 0.048 | 0.436 | -0.251 | 0.427 | 1.090
SE | 0.988 | 0.983 | 1.959 | 1.558 | 1.624 | 1.879 | 1.754
Reg.SE | 0.024 | 0.024 | 0.061 | 0.058 | 0.060 | 0.051 | 0.062
MCSE | 0.002 | 0.002 | 0.004 | 0.005 | 0.006 | 0.004 | 0.005
% sign. | 74.0% | 74.0% | 6.0% | 74.0% | 72.0% | 74.0% | 74.0%
MidQR($\tau_{4}$) | Est | -0.297 | 0.208 | 0.021 | 0.337 | -0.219 | 0.335 | 1.339
SE | 0.729 | 0.652 | 1.655 | 1.725 | 1.645 | 1.672 | 1.582
Reg.SE | 0.031 | 0.030 | N.D. | 0.070 | 0.074 | 0.059 | 0.077
MCSE | 0.002 | 0.002 | 0.003 | 0.004 | 0.006 | 0.004 | 0.005
% sign. | 74.0% | 74.0% | 0.0% | 74.0% | 68.0% | 73.0% | 74.0%
MidQR($\tau_{5}$) | Est | -0.221 | 0.154 | -0.004 | 0.220 | -0.185 | 0.212 | 1.628
SE | 1.905 | 1.125 | 1.831 | 3.658 | 2.830 | 2.480 | 2.603
Reg.SE | 0.042 | 0.041 | N.D. | 0.094 | 0.100 | 0.080 | 0.106
MCSE | 0.002 | 0.002 | 0.002 | 0.003 | 0.005 | 0.003 | 0.004
% sign. | 74.0% | 74.0% | 0.0% | 65.0% | 23.0% | 70.0% | 74.0%
|  |  |  |  |  |  |  |
Table 5.9: RGA and AGR from simulations with a higher number of levels for the response variable: $k=6$ (columns 2-5) and $k=8$ (columns 6-9). | $k=6$ | $k=8$
---|---|---
RGA | AGR | RGA | AGR
Est | SD | Est | SD | Est | SD | Est | SD
OrdLog | 7,468 | 0,679 | 8,865 | 0,717 | 6,999 | 0,603 | 8,344 | 0,644
LinReg | 8,124 | 0,652 | 5,932 | 0,426 | 7,709 | 0,494 | 6,365 | 0,303
MidQR.1 | 8,025 | 0,683 | 5,206 | 0,248 | 7,636 | 0,495 | 5,164 | 0,234
MidQR.2 | 8,064 | 0,664 | 5,268 | 0,246 | 7,682 | 0,493 | 5,222 | 0,221
MidQR.3 | 8,080 | 0,661 | 5,268 | 0,249 | 7,641 | 0,513 | 5,206 | 0,237
MidQR.4 | 8,067 | 0,657 | 5,256 | 0,253 | 7,598 | 0,510 | 5,177 | 0,241
MidQR.5 | 7,989 | 0,645 | 5,183 | 0,273 | 7,475 | 0,558 | 5,080 | 0,267
Self | 9,533 | 0,515 | 9,533 | 0,515 | 8,932 | 0,436 | 8,932 | 0,436
|  |  |  |  |  |  |  |

(a) RGA, $k=6$.

(b) AGR, $k=6$.

(c) RGA, $k=8$.

(d) AGR, $k=8$.

Figure 5.6: Boxplots for RGA and AGR when $k=6$ or $k=8$. Boxplots refer, from
left to right of the x-axis, to OrdLog, LinReg, MidQR with $\tau$ taking
values $0.1,0.3,0.5,0.7,0.9$,  and the reference value
$\mathrm{RGA}(r_{\mathrm{true}},r_{\mathrm{true}})$.

#### Real dataset analysis

In parallel with the investigation of the simulated data, we report the study
of the dataset whose construction has been described in Section 3. In
particular, we present the same type of indicators considered for the
simulations; however, here we stress that multiple datasets are constructed
from the original one through its random splitting into a training set
($n_{\mathrm{tr}}=664$) and a test set ($n_{\mathrm{test}}=50$). We generated
$10$ random extraction of test sets, whose complements return the associated
training sets, to evaluate averaged parameter estimates, standard errors, and
predictive performance indices; $16$ quantile levels equally spaced between
$0.1$ and $0.9$ are considered in this case.

We start from parameter estimates, which are shown in Table 5.10: here, the
whole set of variables described in Table 3.1 is used to implement the
regression models. Then we restrict these models by considering only technical
($X_{\mathrm{AC}}$, $X_{\mathrm{AV}}$) and contextual (exposure, exploit)
variables; the corresponding outcomes are presented in Table 5.11.

Table 5.10: Parameter estimates from data regarding real cyber-
vulnerabilities. All the variables have been used as regressors.

Exposure C I A AV AC Exploit Intercept(s) L Q L Q L Q L Q L Q 1|2 2|3 3|4
OrdReg Est -0.002 -0.398 -0.413 1.086 0.289 0.649 -0.199 0.175 0.499 -0.063
0.114 0.202 -2.499 1.191 3.333 SE 0.009 0.837 0.505 0.884 0.535 0.435 0.270
0.483 0.287 0.289 0.213 0.218 0.435 0.424 0.474 MCSE 0.001 0.055 0.041 0.045
0.034 0.029 0.015 0.084 0.050 0.031 0.026 0.020 0.063 0.059 0.053 LinReg Est
-1.995 -113.680 -60.712 145.483 61.944 24.812 -1.404 -30.366 82.213 -13.484
17.780 15.138 299.034 SE 0.809 84.827 50.777 87.973 52.871 38.624 24.041
44.040 26.250 26.260 19.493 20.182 37.839 MCSE 0.064 5.617 3.592 3.786 2.826
1.999 0.741 6.825 4.016 2.403 1.768 1.723 4.214 % sign. 90.0% 0.0% 0.0% 0.0%
0.0% 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 100.0% MidQR($\tau_{1}$) Est 0.002
0.027 -0.023 0.053 -0.015 0.046 -0.018 0.031 0.021 0.007 -0.010 0.006 0.080 SE
0.034 2.450 1.520 2.621 1.631 1.421 0.910 1.618 0.914 1.012 0.739 0.771 1.559
MCSE 0.000 0.004 0.003 0.004 0.003 0.001 0.001 0.002 0.002 0.001 0.001 0.001
0.003 MidQR($\tau_{4}$) Est 4 0.000 0.008 -0.049 0.070 -0.007 0.071 -0.021
0.053 0.034 0.016 -0.006 0.006 0.409 SE 4 0.031 2.431 1.500 2.719 1.685 1.399
0.883 1.895 1.119 1.009 0.769 0.829 1.492 MCSE 4 0.000 0.005 0.004 0.005 0.003
0.003 0.001 0.004 0.002 0.002 0.001 0.001 0.005 MidQR($\tau_{7}$) Est 7 -0.001
-0.007 -0.044 0.070 0.000 0.057 -0.015 0.041 0.037 0.019 -0.005 0.007 0.670 SE
7 0.025 2.654 1.615 2.854 1.739 1.453 0.898 2.215 1.283 0.976 0.731 0.719
1.409 MCSE 7 0.000 0.004 0.003 0.003 0.002 0.003 0.001 0.005 0.003 0.001 0.001
0.001 0.005 MidQR($\tau_{10}$) Est 10 -0.001 -0.011 -0.033 0.060 0.003 0.043
-0.009 0.031 0.034 0.018 -0.004 0.008 0.868 SE 10 0.019 1.976 1.203 2.183
1.334 0.992 0.619 1.725 0.996 0.749 0.562 0.553 1.072 MCSE 10 0.000 0.003
0.003 0.002 0.002 0.002 0.001 0.004 0.002 0.001 0.001 0.001 0.004
MidQR($\tau_{13}$) Est 13 -0.002 -0.017 -0.024 0.051 0.008 0.026 -0.003 0.024
0.032 0.016 -0.002 0.008 1.036 SE 13 0.016 1.775 1.073 1.956 1.189 0.838 0.517
1.281 0.740 0.638 0.491 0.462 0.852 MCSE 13 0.000 0.002 0.002 0.001 0.001
0.002 0.001 0.004 0.002 0.001 0.001 0.001 0.004 MidQR($\tau_{16}$) Est 16
-0.004 -0.058 -0.032 0.072 0.014 0.004 -0.002 0.006 0.046 0.020 0.003 0.005
1.285 SE 16 0.019 1.442 0.902 1.559 0.972 0.833 0.530 1.009 0.592 0.632 0.485
0.523 0.854 MCSE 16 0.000 0.003 0.002 0.001 0.001 0.002 0.001 0.004 0.002
0.001 0.001 0.001 0.005

Table 5.11: Parameter estimates from data regarding real cyber-vulnerabilities. Only technical and contextual variables have been used as regressors. |  | Exposure | AV | AC | Exploit | Intercept(s)
---|---|---|---|---|---|---
|  | L | Q | L | Q | 1|2 | 2|3 | 3|4
OrdReg | Est | -0.009 | -0.064 | 0.636 | -0.023 | 0.136 | 0.190 | -2.566 | 0.959 | 3.124
SE | 0.009 | 0.464 | 0.278 | 0.279 | 0.210 | 0.218 | 0.422 | 0.407 | 0.461
MCSE | 0.001 | 0.068 | 0.037 | 0.038 | 0.016 | 0.009 | 0.065 | 0.059 | 0.068
LinReg | Est | -2.078 | -43.265 | 90.509 | -16.164 | 19.103 | 19.928 | 295.752 |  |
SE | 0.794 | 43.024 | 25.670 | 25.757 | 19.419 | 20.164 | 36.936 |  |
MCSE | 0.094 | 5.794 | 3.062 | 3.227 | 1.133 | 0.678 | 4.905 |  |
% sign. | 90.0% | 0.0% | 100.0% | 0.0% | 0.0% | 0.0% | 100.0% |  |
MidQR($\tau_{1}$) | Est | 0.001 | 0.045 | 0.007 | 0.009 | -0.007 | -0.004 | 0.052 |  |
SE | 0.032 | 1.795 | 1.036 | 1.061 | 0.782 | 0.781 | 1.508 |  |
MCSE | 0.000 | 0.003 | 0.002 | 0.001 | 0.001 | 0.001 | 0.003 |  |
MidQR($\tau_{4}$) | Est | 0.000 | 0.044 | 0.034 | 0.023 | -0.003 | -0.007 | 0.414 |  |
SE | 0.026 | 2.439 | 1.427 | 0.977 | 0.764 | 0.763 | 1.511 |  |
MCSE | 0.000 | 0.004 | 0.002 | 0.002 | 0.001 | 0.001 | 0.003 |  |
MidQR($\tau_{7}$) | Est | -0.001 | 0.033 | 0.037 | 0.024 | -0.003 | -0.004 | 0.674 |  |
SE | 0.022 | 2.245 | 1.312 | 0.904 | 0.701 | 0.701 | 1.358 |  |
MCSE | 0.000 | 0.004 | 0.002 | 0.001 | 0.001 | 0.001 | 0.003 |  |
MidQR($\tau_{10}$) | Est | -0.002 | 0.021 | 0.034 | 0.022 | -0.002 | -0.001 | 0.870 |  |
SE | 0.015 | 1.553 | 0.906 | 0.590 | 0.474 | 0.483 | 0.930 |  |
MCSE | 0.000 | 0.003 | 0.001 | 0.001 | 0.001 | 0.000 | 0.003 |  |
MidQR($\tau_{13}$) | Est | -0.002 | 0.011 | 0.031 | 0.018 | -0.001 | 0.000 | 1.032 |  |
SE | 0.014 | 1.331 | 0.777 | 0.515 | 0.413 | 0.424 | 0.805 |  |
MCSE | 0.000 | 0.002 | 0.001 | 0.001 | 0.001 | 0.000 | 0.002 |  |
MidQR($\tau_{16}$) | Est | -0.004 | 0.011 | 0.048 | 0.029 | 0.001 | 0.004 | 1.270 |  |
SE | 0.017 | 1.310 | 0.761 | 0.631 | 0.516 | 0.492 | 0.892 |  |
MCSE | 0.000 | 0.003 | 0.001 | 0.002 | 0.001 | 0.001 | 0.005 |
|  |  |  |  |  |  |  |  |  |

Moving to the performance indices, both RGA and AGR for all the regression
models under examination are reported in Table 5.12. In addition, we provide
two graphical representations regarding the behavior of the predictive
performance at different quantile levels: the boxplots in Figure 5.7 and the
plots of average RGA and AGR for all the $16$ quantile levels in Figure 5.8.

Table 5.12: RGA and AGR indices from real data analysis. Columns 2-5 refer to models with the full set of regressors; columns 6-9 follow from the restriction to technical (AV, AC) and contextual (exposure, exploit) variables as regressors. | Full set of regressors | Only technical regressors
---|---|---
RGA | AGR | RGA | AGR
Est | SD | Est | SD | Est | SD | Est | SD
OrdLog | 0.542 | 0.248 | 0.000 | 0.000 | 0.716 | 0.565 | 0.000 | 0.000
LinReg | 1.006 | 0.839 | 0.042 | 0.025 | 1.006 | 1.005 | 0.042 | 0.028
MidQR($\tau_{1}$) | 0.982 | 0.724 | 0.358 | 0.172 | 0.825 | 0.764 | 0.130 | 0.059
MidQR($\tau_{2}$) | 0.929 | 0.837 | 0.328 | 0.150 | 0.786 | 0.649 | 0.127 | 0.065
MidQR($\tau_{3}$) | 0.926 | 0.786 | 0.283 | 0.121 | 0.733 | 0.389 | 0.129 | 0.062
MidQR($\tau_{4}$) | 0.916 | 0.735 | 0.253 | 0.117 | 0.786 | 0.559 | 0.153 | 0.102
MidQR($\tau_{5}$) | 0.909 | 0.728 | 0.207 | 0.116 | 0.812 | 0.475 | 0.162 | 0.084
MidQR($\tau_{6}$) | 0.950 | 0.795 | 0.208 | 0.122 | 0.839 | 0.479 | 0.159 | 0.084
MidQR($\tau_{7}$) | 1.008 | 0.821 | 0.215 | 0.142 | 0.881 | 0.579 | 0.171 | 0.099
MidQR($\tau_{8}$) | 1.034 | 0.845 | 0.204 | 0.139 | 0.899 | 0.625 | 0.175 | 0.107
MidQR($\tau_{9}$) | 1.082 | 0.865 | 0.203 | 0.147 | 0.932 | 0.659 | 0.177 | 0.111
MidQR($\tau_{10}$) | 1.085 | 0.894 | 0.197 | 0.147 | 0.945 | 0.734 | 0.182 | 0.116
MidQR($\tau_{11}$) | 1.163 | 1.001 | 0.199 | 0.160 | 0.937 | 0.762 | 0.180 | 0.117
MidQR($\tau_{12}$) | 1.147 | 0.979 | 0.181 | 0.137 | 0.966 | 0.824 | 0.181 | 0.114
MidQR($\tau_{13}$) | 1.191 | 1.008 | 0.174 | 0.125 | 0.974 | 0.829 | 0.175 | 0.108
MidQR($\tau_{14}$) | 1.159 | 1.050 | 0.159 | 0.106 | 0.989 | 0.844 | 0.179 | 0.111
MidQR($\tau_{15}$) | 1.132 | 1.094 | 0.151 | 0.096 | 1.005 | 0.830 | 0.185 | 0.117
MidQR($\tau_{16}$) | 0.905 | 0.798 | 0.146 | 0.086 | 1.017 | 0.824 | 0.185 | 0.107
Self | 6.120 | 0.880 | 6.120 | 0.880 | 5.991 | 1.182 | 5.991 | 1.182
|  |  |  |  |  |  |  |

(a) RGA from real data,
full model.

(b) AGR from real data,
full model.

(c) RGA from real data,
partial model.

(d) AGR from real data,
partial model.

Figure 5.7: Boxplots of RGA and AGR for real data. Boxplots refer, from left
to right of the x-axis, to OrdLog, LinReg, and MidQR with $\tau$ taking values
$0.1,0.26,0.42,0.58,0.74,0.9$.

(a) RGA.

(b) AGR.

Figure 5.8: Behaviour of average RGA and AGR for real data and the $16$
quantile levels $\tau$ under consideration. Circles and triangles denote the
index estimates for the full and partial models, respectively. The
$y$-intercepts of the dotted and dot-dashed lines represent the value of the
index from the ordered logit and the linear regression on rank-transformed
variables, respectively.

##  6 Discussion

This work proposed MidQR to enhance the flexibility, interpretability, and
robustness of cyber-risk assessments: a quantile-based approach can extract
relevant information beyond means to prevent rare events, which is a primary
need for the continuity of a network or critical infrastructure. Then, we
introduced the AGR index to evaluate predictive performance without relying on
a quantitative structure on the ordinal responses.

Here, we discuss the outcomes of the analysis of synthetic and real data.

##### AGR as an appropriate measure of predictive accuracy

From simulations, we see that the data-generating models are generally
associated with a higher AGR value, while its RGA is often worse than other
models (see Figures 5.4, 5.5, and 5.6).

It is plausible that the specific model underlying the data generation process
provides better predictive performance compared to other models. This
criterion identifies AGR as a more appropriate performance index for our
purposes, since it better distinguishes the data-generating model in terms of
predictive capacity, as it is manifest from the above-mentioned figures.

In addition, AGR enjoys the invariance property under sub-sampling, as
discussed in Section 4, which is desirable since the relative order between
two vulnerabilities is not affected by other vulnerabilities in the sample. In
this way, we can better prioritise the relative risk factors for the
vulnerabilities under consideration, without incurring order reversal due to
new vulnerabilities not previously detected. From a different perspective,
such new information may be needed to update individual risk factors and adapt
to the dynamic behaviour of the cyber-space, as is discussed in the following
paragraph.

##### Vulnerabilities vs. incidents: dynamical and subjective cyber-risk
assessment

We already pointed out the distinction between cyber-incidents and cyber-
vulnerability, recalling that the analysis in [17] focuses on the former. This
distinction is relevant for decision-makers, namely, cybersecurity experts and
ICT managers, Security Operational Centres, National Agencies, etc. The
analysis of cyber-incidents is fundamental for cyber-forensic activities, but
the prevention of _new_ cyber-incidents in operational scenarios should use
all fungible information to better manage security resources and take
appropriate counteractions.

This condition underlies a dynamic environment characterising cyber-risk
assessment, which affects both regressors and responses. Indeed, for each
statistical unit identified by a CVE, both intrinsic characteristics (NIST’s
evaluation of the attack vector) and external variables (exposure, exploits)
can change in time; furthermore, risk factors may vary due to internal
priorities in the organisation and evolution of the overall digital system
(new products, legislation, etc.).

Direct consequences of the dynamics of cyber-risk assessment concern the
prioritisation of fixing activities and the specification of subjective cyber-
risk. Fixing being resource-expensive, decision-makers have to allocate their
efforts based on their current state of knowledge. The driver of such choices
is the individual _risk perception_. While the present work uses Tenable’s
risk factor for the analysis, each decision-maker can customise the present
risk assessment model (as well as the quantile level), adapting it in time to
get new estimates and quantile effects, or comparing different risk factors
attributions (e.g. derived from different criteria) in terms of their
predictive power.

These observations are mainly related to cybersecurity data and their
usefulness for distinct decision-making stages, which led us to select the
databases described in Section 3. Information granularity in data from cyber-
incidents does not often suffice to extract useful insights on the current
threats: as recalled in [17], the type of data that is likely to be accessed
when analysing cyber-incidents is rarely disclosed. This leads to data
aggregation and censoring that could not allow cybersecurity operational
experts to prioritise the current vulnerabilities, as is the case in the
classification of attack technique reported in [17] where multiple types of
attack are grouped together (e.g. SQL injection is a particular attack model
upon which malware can be based, and malware can exploit one or more 0-days).
This has to be interpreted as complementarity between the analyses on cyber-
incidents, like the one conducted in [17], and the present one: they serve
different phases (strategic, tactic, or operative) of a process with a common
objective, and each phase should identify appropriate data for its scope.

##### Implications of MidQR on secure information disclosure

As a consequence of the observations in the last paragraph, we draw attention
to the information the individual decision-maker has, uses, and communicates
about cyber-risk.

Agencies such as NIST share their evaluation through dedicated information
channels: however, this information can also be acquired by potential
attackers, who can use them to prioritise their own objectives. Indeed,
resources are needed also by attackers (e.g. costs for exploit acquisition,
time and effort for detection of vulnerable hosts, integration of multiple
components to avoid countermeasures), and information on risk factors from
different organisations can be useful to suggest relevant criticalities.

Our proposal addresses this issue in two ways: first, as already recalled,
MidQR enhances robustness against violation of assumptions in parametric
methods; censored data on cyber-vulnerabilities, due to no-disclosure
policies, may limit or distort the verification of such assumptions, which
results in misleading analysis and results.

Second, AGR better identifies the added information content provided by the
variables for the cyber-vulnerability data: referring to Table 5.12, two
different models are considered, the full one (all the relevant variables in
the dataset derived from Table 3.1 are involved) and a restricted one, where
the “CIA” components of attack vectors are excluded. We note that LinReg does
not distinguish these two models (up to the third significant digit), while
MidQR always does (except for MidQR($\tau_{12}$) using AGR).

Furthermore, restricting our attention to MidQR, we see that AGR is more
sensitive than RGA with respect to the choice of the quantile level in terms
of model discrimination: formally, let us consider the ratios

$\varrho_{\mathrm{RGA}}:=\frac{\mathrm{RGA}_{\mathrm{tech}}}{\mathrm{RGA}_{\mathrm{full}}},\quad\varrho_{\mathrm{AGR}}:=\frac{\mathrm{AGR}_{\mathrm{tech}}}{\mathrm{AGR}_{\mathrm{full}}}$
(6.1)

of RGA and AGR evaluated for the technical and the full models, respectively.
Excluding $\tau_{16}$, for the remaining MidQR models we can observe
$\varrho_{\mathrm{RGA}}\in[0.792;0.893]$, while
$\varrho_{\mathrm{AGR}}\in[0.363;1.225]$.

These observations confirm the utility of the combined use of RGA and AGR to
get a more representative view of cyber-risk models and, in this specific
case, of the potential value of information communicated when sharing risk
assessment.

##### Dependence of MidQR performance on $k$

A final remark regarding the comparison of RGA and AGR involves the number of
chosen levels of the response variable: MidQR performs better when $k$ is
small (less than $6$), as can be seen comparing 5.4-5.5 with 5.6. In the
latter, AGR highlights a divergence between the data-generating model (OrdLog)
and alternative models (LinReg or MidQR); on the other hand, RGA returns a
performance comparable to that of LinReg and MidQR.

##### SE of the estimates

As remarked in the previous section, an arbitrary choice of the quantile level
may lead to overestimating the parameter SE through the kernel approach
adopted in [16] and based on [25]; this is confirmed by the outputs of the
simulations. When this overestimation happens, the remaining indices (i.e. the
regular SE and the MCSE) provide a more informative picture of the sampling
distribution.

##### Real data and implications for cyber-threat intelligence

While the different models considered in this work are comparable in terms of
RGA performance on real data, using AGR, we can see that OrdLog performs
poorly since the predicted values are restricted to the set $\\{1,\dots,k\\}$;
when the dataset has low variability, the estimated values collapse to a
typical value, which contains no information and drastically reduces
predictive performance. This also suggests a severe deviation from the OrdLog
model assumptions in the present cyber-vulnerability dataset.

Another indirect test of the deviation of real data from the OrdLog model
comes from the relative magnitude of RGA and AGR: in Tables 5.1, 5.2, 5.4,
5.7, and 5.8, which refer to data simulated starting from the ordered logit
model, AGR is comparable with RGA (i.e. with the same order of magnitude). At
low values of $k$, especially at $k=3$, AGR is larger than RGA when we focus
on MidQR and on the data-generating model. On the other hand, real data lead
to a different behaviour, where the ratios $\mathrm{AGR}/\mathrm{RGA}$ lie in
$[0.1334;0.365]$ for the full model and in $[0.088;0.238]$ for the “technical”
model. While these ratios are useful as an additional check of the deviation
from the OrdLog model used in simulations, AGR and RGA indices for the same
model should not be compared, as they measure different performance aspects of
a given model.

As a final observation, we stress that the choice of the quantile level plays
an important role: this choice can be driven by the empirical distribution of
risk factor levels, but it can also be seen as a latent trait to be estimated
through Bayesian approaches.

##  7 Conclusion and Future Work

This work investigates statistical modelling for threat intelligence, with
particular attention to the information resources regarding cyber-
vulnerabilities and to the effects of risk acceptance/aversion. The
statistical model and the index proposed for cyber-vulnerability assessment
are complementary to other approaches developed in cyber-risk literature: in
general, multiple models should be considered in parallel, to highlight
distinct aspects of relevance to the decision-makers.

The actual realisation of cyber-attacks relies on several information sources
that can enhance or inhibit cyber-attacks. It is plausible that access to
information plays a more important role than expected: cybersecurity data are
subject to limited disclosure and underestimation, as recalled in [17], both
to adhere to security standards and to prevent reputational loss. On the other
hand, even data communicated by organisations to prevent cyber-incidents can
guide cyber-attackers, as discussed in Section 6. The present work opens the
way to further applications supporting secure information disclosure on cyber-
vulnerabilities, since the advantages of the framework discussed in the
previous sections can highlight the effects of both information sources (in
terms of available regressors) and cyber-risk perception or severity
assessments (e.g. a suited data-generating model). A more accurate evaluation
of such effects is a necessary premise to avoid indirect and unintended
communication of information.

In addition to the methodological interest, the present contribution and its
extensions may integrate the functionalities of current tools for the
development of secure digital products [5]. In this regard, future work will
explore specific models for the quantitative assessment of latent traits in
cyber-risk perception, including Bayesian approaches, e.g. global-local priors
to shrink weak signals and highlight relevant risk responses. A deeper
investigation is needed for the emergence of multiple prioritisations due to
different decision criteria and uncertainty sources, possibly involving
information-theoretic methods to quantify their compatibility or, dually,
their divergence.

## References

  * [1] R. Anderson, C. Barton, R. Böhme, R. Clayton, M. J. Van Eeten, M. Levi, T. Moore, and S. Savage.  Measuring the cost of cybercrime.  The economics of information security and privacy, pages 265–300, 2013.
  * [2] M. Angelelli.  Tropical limit and a micro-macro correspondence in statistical physics.  J. Phys. A-Math. Theor., 50(41):415202, 2017.
  * [3] M. Angelelli and C. Catalano.  A quantile regression ranking for cyber-risk assessment.  In N. Torelli, R. Bellio, and V. Muggeo, editors, Proceedings of the 36th International Workshop on Statistical Modelling. Trieste: EUT Edizioni, 2022.
  * [4] M. Angelelli, C. Catalano, D. Hill, H. Koshutanski, C. Pascarelli, and J. Rafferty.  A reference architecture proposal for secure data management in mobile health.  In 2022 7th International Conference on Smart and Sustainable Technologies (SpliTech), pages 1–6. IEEE, 2022.
  * [5] M. T. Baldassarre, V. S. Barletta, D. Caivano, and A. Piccinno.  A visual tool for supporting decision-making in privacy oriented software development.  In AVI ’20: Proceedings of the International Conference on Advanced Visual Interfaces, pages 1–5, 2020.
  * [6] V. S. Barletta, D. Caivano, M. D. Vincentiis, A. Ragone, M. Scalera, and M. Á. S. Martín.  V-SOC4AS: A Vehicle-SOC for Improving Automotive Security.  Algorithms, 16(2):112, 2023.
  * [7] M. Carfora, F. Martinelli, F. Mercaldo, and A. Orlando.  Cyber risk management: An actuarial point of view.  Journal of Operational Risk, 14(4), 2019.
  * [8] C. Catalano, P. Afrune, M. Angelelli, G. Maglio, F. Striani, and F. Tommasi.  Security Testing Reuse Enhancing Active Cyber Defence in Public Administration.  In ITASEC, pages 120–132, 2021.
  * [9] C. Catalano, A. Chezzi, M. Angelelli, and F. Tommasi.  Deceiving AI-based malware detection through polymorphic attacks.  Comput. Ind., 143:103751, 2022.
  * [10] A. Corallo, M. Lazoi, and M. Lezzi.  Cybersecurity in the context of industry 4.0: A structured classification of critical assets and business impacts.  Comput. Ind., 114:103165, 2020.
  * [11] Z. Cui, F. Xue, X. Cai, Y. Cao, G.-g. Wang, and J. Chen.  Detection of malicious code variants based on deep learning.  IEEE T. Ind. Inform., 14(7):3187–3196, 2018.
  * [12] B. Edwards, S. Hofmeyr, and S. Forrest.  Hype and heavy tails: A closer look at data breaches.  J. Cybersecur., 2(1):3–14, 2016.
  * [13] A. Fielder, E. Panaousis, P. Malacaria, C. Hankin, and F. Smeraldi.  Decision support approaches for cyber security investment.  Decision support systems, 86:13–23, 2016.
  * [14] A. Fioraldi.  CVE SearchSploit.  GitHub, June 2017.
  * [15] G. Fortino, C. Savaglio, G. Spezzano, and M. Zhou.  Internet of things as system of systems: A review of methodologies, frameworks, platforms, and tools.  IEEE T. Syst. Man Cy.-S., 51(1):223–236, 2020.
  * [16] M. Geraci and A. Farcomeni.  Mid-quantile regression for discrete responses.  Stat. Methods Med. Res., 31(5):821–838, 2022.
  * [17] P. Giudici and E. Raffinetti.  Cyber risk ordering with rank-based statistical models.  ASTA-Adv. Stat. Anal., 105(3):469–484, 2021.
  * [18] P. Giudici and E. Raffinetti.  Explainable AI methods in cyber risk management.  Qual. Reliab. Eng. Int., 38(3):1318–1326, 2022.
  * [19] R. L. Iman and W. J. Conover.  The use of the rank transform in regression.  Technometrics, 21(4):499–509, 1979.
  * [20] D. Javaheri, S. Gorgin, J.-A. Lee, and M. Masdari.  Fuzzy logic-based ddos attacks and network traffic anomaly detection methods: Classification, overview, and future perspectives.  Information Sciences, 2023.
  * [21] B. Jung, Y. Li, and T. Bechor.  Cavp: A context-aware vulnerability prioritization model.  Comput. Secur., 116:102639, 2022.
  * [22] M. Keshk, N. Koroniotis, N. Pham, N. Moustafa, B. Turnbull, and A. Y. Zomaya.  An explainable deep learning-enabled intrusion detection framework in iot networks.  Information Sciences, page 119000, 2023.
  * [23] R. Koenker and K. F. Hallock.  Quantile regression.  J. Econ. Perspect., 15(4):143–156, 2001.
  * [24] Q. Li, J. Lin, and J. S. Racine.  Optimal bandwidth selection for nonparametric conditional distribution and quantile functions.  J. Bus. Econ. Stat., 31(1):57–65, 2013.
  * [25] Q. Li and J. S. Racine.  Nonparametric estimation of conditional CDF and quantile functions with mixed categorical and continuous data.  J. Bus. Econ. Stat., 26(4):423–434, 2008.
  * [26] R. D. Luce.  Individual choice behavior: A theoretical analysis.  Dover Publications, 2005.
  * [27] Y. Ma, M. G. Genton, and E. Parzen.  Asymptotic properties of sample quantiles of discrete distributions.  Ann. I. Stat. Math., 63(2):227–243, 2011.
  * [28] P. McCullagh.  Regression models for ordinal data.  J. Roy. Stat. Soc. B Met., 42(2):109–127, 1980.
  * [29] P. Paruchuri, J. P. Pearce, J. Marecki, M. Tambe, F. Ordonez, and S. Kraus.  Playing games for security: An efficient exact algorithm for solving bayesian stackelberg games.  In Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volume 2, pages 895–902, 2008.
  * [30] E. Parzen.  Quantile probability and statistical data modeling.  Stat. Sci., 19(4):652–662, 2004.
  * [31] M.-E. Paté-Cornell, M. Kuypers, M. Smith, and P. Keller.  Cyber risk management for critical infrastructure: a risk analysis model and three case studies.  Risk Anal., 38(2):226–241, 2018.
  * [32] P. Radanliev, D. C. De Roure, R. Nicolescu, M. Huth, R. M. Montalvo, S. Cannady, and P. Burnap.  Future developments in cyber risk assessment for the internet of things.  Comput. Ind., 102:14–22, 2018.
  * [33] R. Sharma and R. Singh.  An improved scoring system for software vulnerability prioritization.  In Quality, IT and Business Operations: Modeling and Optimization, pages 33–43. Springer, 2018.
  * [34] B. Srinidhi, J. Yan, and G. K. Tayi.  Allocation of resources to cyber-security: The effect of misalignment of interest between managers and investors.  Decis. Support Syst., 75:49–62, 2015.
  * [35] F. Tommasi, C. Catalano, U. Corvaglia, and I. Taurino.  MinerAlert: an hybrid approach for web mining detection.  J. Comput. Virol. Hacking Tech., pages 1–14, 2022.
  * [36] J. Van Haaster, R. Gevers, and M. Sprengers.  Cyber guerilla.  Syngress, 2016.
  * [37] Y. Zhang and P. Malacaria.  Optimization-time analysis for cybersecurity.  IEEE Transactions on Dependable and Secure Computing, 19(4):2365–2383, 2021.