# LIVABLE: Exploring Long-Tailed Classification of Software Vulnerability
Types

Xin-Cheng Wen2, Cuiyun Gao123, Feng Luo2, Haoyu Wang4, Ge Li5, and Qing Liao2
Harbin Institute of Technology, Shenzhen, China
3 Peng Cheng Laboratory, Shenzhen, China
4 Huazhong University of Science and Technology, Wuhan, China
5 Peking University, Beijing, China
* corresponding author. 2<EMAIL_ADDRESS>{gaocuiyun<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Prior studies generally focus on software vulnerability detection and have
demonstrated the effectiveness of Graph Neural Network (GNN)-based approaches
for the task. Considering the various types of software vulnerabilities and
the associated different degrees of severity, it is also beneficial to
determine the type of each vulnerable code for developers. In this paper, we
observe that the distribution of vulnerability type is long-tailed in
practice, where a small portion of classes have massive samples (i.e., head
classes) but the others contain only a few samples (i.e., tail classes).
Directly adopting previous vulnerability detection approaches tends to result
in poor detection performance, mainly due to two reasons. First, it is
difficult to effectively learn the vulnerability representation due to the
over-smoothing issue of GNNs. Second, vulnerability types in tails are hard to
be predicted due to the extremely few associated samples.

To alleviate these issues, we propose a Long-taIled software VulnerABiLity
typE classification approach, called LIVABLE. LIVABLE mainly consists of two
modules, including (1) vulnerability representation learning module, which
improves the propagation steps in GNN to distinguish node representations by a
differentiated propagation method. A sequence-to-sequence model is also
involved to enhance the vulnerability representations. (2) adaptive re-
weighting module, which adjusts the learning weights for different types
according to the training epochs and numbers of associated samples by a novel
training loss. We verify the effectiveness of LIVABLE in both type
classification and vulnerability detection tasks. For vulnerability type
classification, the experiments on the Fan et al. dataset show that LIVABLE
outperforms the state-of-the-art methods by 24.18% in terms of the accuracy
metric, and also improves the performance in predicting tail classes by 7.7%.
To evaluate the efficacy of the vulnerability representation learning module
in LIVABLE, we further compare it with the recent vulnerability detection
approaches on three benchmark datasets, which shows that the proposed
representation learning module improves the best baselines by 4.03% on average
in terms of accuracy.

###### Index Terms:

Software Vulnerability; Deep Learning; Graph Neural Network

##  1 Introduction

Software vulnerabilities are important and common security threats in software
systems. These vulnerabilities can be easily exploited by attackers and have
the potential to cause irreparable damage to software systems [1]. For
example, buffer overflow issues [2] allow attackers to exploit vulnerabilities
to tamper with memory data or gain control of the system. Vulnerabilities are
inevitable for many reasons, e.g. the complexity of software and the steady
growth in the size of the Internet [3]. Vulnerability detection has received
intensive attention in the software community recently.

With the development of deep learning (DL) techniques, various DL-based
vulnerability detection methods have been proposed [4, 5, 6, 7, 8, 9], which
aim to detect whether a code function is vulnerable or not. For example,
SyseVR [6] combines multiple sequence features to generate code slices and
uses a bidirectional Recursive Neural Network (RNN) [10] to detect
vulnerabilities. Recent studies [7, 8, 9] have demonstrated that Graph Neural
Networks (GNNs) [11] are effective in vulnerability detection by capturing the
structural information of source code. For example, Devign [7] applies the
Gated Graph Neural Network (GGNN) [12] to learn the representation of code
structure graph for vulnerability detection, where the code structure graph is
the combination of Abstract Syntax Tree (AST), Control Flow Graph (CFG), Data
Flow Graph (DFG) and Natural Code Sequence (NCS).

Figure 1: The label distribution (bar chart) and the corresponding CVSS score
(line chart) of National Vulnerability Database (NVD) data [13] during the
period of 2012 to 2022.The head, medium, and tail classes respectively contain
27, 45 and 253 classes in NVD respectively.

Despite that the previous studies [7, 8, 9] can inform developers of the
vulnerable functions in source code, the developers may still feel difficult
to fix the vulnerabilities. Considering the various types of vulnerabilities
and the associated different degrees of severity [14, 15], predicting the
vulnerability types can assist developers in arranging the maintenance
priority and localizing the vulnerable cause. Recently, Zou et al. [16]
leverage function call information to conduct multi-class vulnerability
detection, which requires project-level information and can hardly be applied
to predict the vulnerability type of source code in function-level. To our
best knowledge, no prior research has explored the type classification problem
for vulnerable functions.

For facilitating analysis, we first study the collected National Vulnerability
Database (NVD) [13] data during the period of 2012 to 2022, as illustrated in
Figure 1. We observe that the sample size of different vulnerability types
generally presents a long-tailed distribution. Specifically, only a small
portion of classes have massive samples (i.e., head classes) while the others
contain extremely few samples (i.e., tail classes). Although the vulnerable
code in tails is limited in number, the degree of severity would be high.
According to _Common Vulnerability Scoring System (CVSS)_ [17], which
characterizes the degree of severity into a score of 1-10, the average CVSS
score 111In this paper, we use the _Common Vulnerability Scoring System V3
Score[18]_.  of all the classes in tails is 7.01, indicating a high-level
severity [19]. Some of the most threatening vulnerability types in tails such
as CWE-507 and CWE-912 are with a CVSS score of 9.8. For example, the CWE-507
(Trojan Horse) [20] is a virus that is added to supply chain software by
malware, leading to serious security risks. Therefore, effectively classifying
the vulnerable types including the tail classes is important for software
security.

Directly applying the existing vulnerable detection methods [7, 8] is one
possible solution. However, they tend to fail in the long-tailed scenario,
because: (1) They are difficult to effectively learn the vulnerability
representation due to the over-smoothing issue [21] of GNNs. Prior studies
[22] show that the performance of GNNs will degrade as the number of layers
increases, leading to similar representations for different nodes. However,
few layers in GNNs will make the model hard to capture the structural
information of vulnerable code, due to the deep levels in the code structure
graph [23, 24]. (2) It is hard to predict the vulnerability types in tails
which are associated with extremely few samples. The serious imbalance between
the numbers of different vulnerability types renders the models biased towards
head classes, leading to poor representations of tail classes. The models are
prone to producing head classes while ignoring the tails.

To mitigate the above challenges, we propose a Long-taIled software
VulnerABiLity typE classification approach, called LIVABLE. LIVABLE mainly
consists of two modules: (1) a vulnerability representation learning module,
which involves a differentiated propagation-based GNN for alleviating the
over-smoothing problem. For further enhancing the vulnerability representation
learning, a sequence-to-sequence model is also involved to capture the
semantic information. (2) an adaptive re-weighting module, which adaptively
updates the learning weights according to the vulnerable types and training
epochs. The module is designed to well learn the representations of tail
classes based on the limited samples.

The experimental evaluation is performed on both vulnerability type
classification and vulnerability detection tasks. For vulnerability type
classification, we prepare the evaluation dataset by extracting the vulnerable
samples from Fan et al. [25]. The results demonstrate that the LIVABLE
outperforms the state-of-the-art methods by 24.18% in terms of the accuracy
metric, by improving the performance in predicting medium and tail class by
14.7% and 7.7%, respectively. To evaluate the efficacy of the vulnerability
representation learning module in LIVABLE, we further compare with the recent
vulnerability detection approaches on FFMPeg+Qemu [7], Reveal [8], and Fan et
al. [25] datasets, which shows that the proposed representation learning
module improves the best-performing baselines by 4.03% on average in terms of
accuracy.

The major contributions of this paper are as follows:

  1. 1.

We are the first to approach to explore the long-tailed classification of
vulnerable functions.

  2. 2.

We propose LIVABLE, a long-tailed software vulnerability type classification
approach, including: 1) a vulnerability representation learning module for
alleviating the over-smoothing issue of GNNs and enhancing the vulnerability
representations; and 2) an adaptive re-weighting module that involves a novel
training objective for balancing the weights of different types.

  3. 3.

Extensive experiments show the effectiveness of LIVABLE in vulnerability type
classification and the efficacy of vulnerability representation learning
module in vulnerability detection.

The rest of this paper is organized as follows. Section 2 describes the
background. Section 3 details the two components in the proposed framework of
LIVABLE, including the vulnerability representation learning module and
adaptive re-weighting module. Section 4 describes the evaluation methods,
including the datasets, baselines, implementation and metrics. Section 5
presents the experimental results. Section 6 discusses some cases and threats
to validity. Section 8 concludes the paper.

##  2 Background

###  2.1 Graph Neutral Networks

Graph Neural Networks (GNNs) have demonstrated superior performance at
capturing the structural information of source code in the software
vulnerability detection task [7, 8, 9]. The widely-used GNN methods include
Graph Convolutional Network (GCN) [26], Graph Attention Network (GAT) [27],
GGNN. The two-layer of GCN model can calculate as:

$H=softmax(\hat{\tilde{A}}ReLU(\hat{\tilde{A}}XW_{0})W_{1})$ (1)

where $\hat{\tilde{A}}$ denotes the adjacency matrix and $W_{0},W_{1}$ are
trainable weight matrices. $X$ and $H$ denote the initial and output node
representations respectively. With the two layers, only neighbors in the two-
hop neighborhood are considered for the node representation learning. However,
due to the deep levels in the code structure graph [23, 24], a two-layer GNN
is hard to well capture the code node representations. Thus, the GNNs tend to
learn similar node representations during the propagation steps [28], i.e.,
the over-smoothing issue [22], failing to capture the vulnerability patterns
for different types. To mitigate the over-smoothing issue and distinguish the
node representations, a vulnerability representation learning module is
proposed by improving the node feature propagation method.

###  2.2 Long-tailed Learning Methods

In real-world scenarios, data typically exhibit a long-tailed distribution
[29], where a small portion of classes contain massive samples while the
others are associated with only a few samples. Long-tailed learning methods
have been widely studied in the computer vision field [30, 29, 31]. There are
some popular methods to alleviate long-tailed problems, such as class-balanced
strategies [32, 33] and smoothing strategies [30, 34]. For example, one
popular method Focal Loss [33] adjusts the standard cross-entropy loss [35] by
reducing the learning weight for well-classified samples and focuses more on
misclassified samples during model training, calculated as:

${\mathcal{L}}_{FL}=-\sum_{i=1}^{n}(1-\hat{y_{i}})^{\gamma}y_{i}log(\hat{y_{i}})$
(2)

where $y_{i}$ and $\hat{y_{i}}$ denote the label distribution in the ground
truth and the predicted output, respectively. $\gamma$ is the modulating
factor for focusing on misclassified samples.

Label smooth cross-entropy loss [30] is also a widely used long-tailed
solution. It uses smoothing strategies to encourage the model to be less
confident in head classes:

${\mathcal{L}}_{LSCE}=-\sum_{i=1}^{n}log(\hat{y_{i}})\left((1-\epsilon)y_{i}+\epsilon\delta_{i}\right)$
(3)

where $\epsilon$ denotes a smoothing parameter, and $\delta_{i}$ denotes the
uniform distribution to smooth the ground-truth distribution $y_{i}$.

Although the existing methods alleviate the problem of long-tailed
distributions in computer vision, no studies have explored the performance of
these methods in software vulnerability classification. To fill the gap, we
experimentally evaluate the popular long-tailed learning methods, and propose
a novel adaptive learning method for vulnerability type classification.

##  3 Proposed Framework

Figure 2: The architecture of LIVABLE, which mainly contains two components:
(A) a vulnerability representation learning module, and (B) an adaptive re-
weighting module.

In this section, we first formulate the long-tailed problem and then describe
the overall framework of the proposed LIVABLE. As shown in Figure 2, LIVABLE
consists of two main components: (1) a vulnerability representation learning
module for enhancing the vulnerability representations; and (2) an adaptive
re-weighting module that involves a novel training objective for balancing the
weights of different types.

###  3.1 Problem Formulation

The problem of the long-tailed vulnerability type classification is formulated
below. We denote the unbalanced dataset with training instances collected from
the real world as $D=\left\\{x_{i},y_{i}\right\\}$, where $x_{i}$ denotes a
vulnerable function in raw source code and $y_{i}$ denotes its corresponding
vulnerability type. Assume that all classes are ordered by cardinality, i.e.,
if $N_{a}\geq N_{b}$ when the class index $a<b$, where $N_{a}$ indicates the
number of training samples for the class $a$. According to the typical
division ratio in long-tailed scenario [34, 36], we classify the vulnerable
types into three classes: head (¿200 samples per type), medium (50-200 samples
per type), and tail (¡50 samples per type). As a result, the head, medium, and
tail classes contain 12, 11, and 46 types, respectively. The goal of LIVABLE
is to learn a mapping, $f:x_{i}\mapsto y_{i}$, $y_{i}$ denotes the label
distribution, to predict the class of vulnerabilities function. Typically, we
use the Cross-Entropy (CE) loss [35] function as follows:

${\mathcal{L}}_{CE}=-\sum_{i=1}^{n}y_{i}log(\hat{y_{i}})$ (4)

where $n$ denotes the number of categories and $\hat{y_{i}}$ denotes the
output predicted by the model.

###  3.2 Vulnerability Representation Learning Module

In this section, we elaborate on the proposed vulnerability representation
learning module, which involves differentiated propagation-based GNN for
capturing the structural information of source code, and combines the
sequence-to-sequence approach for further enhancing the vulnerability
representations.

####  3.2.1 Differentiated propagation-based GNN

Following the prior studies [7], we extract the same code structure graph of
source code for vulnerability representation. To alleviate the over-smoothing
issue of GNN, we design a differentiated propagation method to distinguish
node representations.

We denote the code structure graph $\mathcal{G}(\mathcal{V},\mathcal{E})$ as
the input, where $\mathcal{V}$ denotes the set of nodes $v$ and $\mathcal{E}$
denotes the set of edges $e$ in the graph. The differentiated propagation-
based GNN is decoupled into two steps, including node feature transformation
and propagation. The representation for each node $v$ is first initialized as
a 128-dimensional vector by the Word2Vec [37] model. To enhance the
representation of each node vector, the feature transformation step then
encodes the vector feature based on a Gated Recurrent Unit (GRU) layer:

$H^{(0)}={\mathcal{F}}\left(X\right)$ (5)

where $X$ is the initial node representation of node $v$, and $\mathcal{F}$
denotes a GRU layer.

In the node feature propagation step, considering that the node
representations in GNNs tend to converge to a similar during the propagation,
we propose to involve the initial node representations. The node
representation $H^{l}$ at the $l$-th layer is defined as:

$H^{l}=\frac{1}{l}\sum_{i=0}^{l-1}\left(\left(1-\alpha\right)\left({\widetilde{D}^{-\frac{1}{2}}}{\widetilde{A}}{\widetilde{D}^{-\frac{1}{2}}}\right)^{i}H^{0}+\alpha
H^{(0)}\right)$ (6)

where $H^{(0)}$ is the transformed feature representation computed by Equation
(5). $\widetilde{A}$ is the adjacency matrix of the code structure graph with
self-connections and $\widetilde{D}$ is the degree matrix of $\widetilde{A}$.
$\alpha$ denotes a teleport hyperparameter, which determines the importance of
the initial node features during the propagation. Finally, the graph
representation $G_{output}$ of the code structure graph $\mathcal{G}$ is
calculated through a hybrid pooling layer:

$G_{output}=\mathcal{C}\left(AvgPool(H_{v})+MaxPool(H_{v})\right),\forall{v}\in\mathcal{V}$
(7)

where $AvgPool(\cdot)$ and $MaxPool(\cdot)$ indicate the average pooling
operation [38] and maximum pooling operation [39], for better capturing the
local information and global information of the code structure graph [40],
respectively. $\mathcal{C}(\cdot)$ denotes two Multi-Layer Perceptron (MLP)
layers.

####  3.2.2 Sequence-to-sequence model

To further enhance the vulnerability representations, we involve a sequence-
to-sequence model to capture the semantics information. Specifically, given a
code snippet $S$ and its composed code token sequence
$S=\\{t_{1},...,t_{i},...,t_{n}\\}$, where $t_{i}$ denotes the $i$-th token
and $n$ denotes the total number of tokens, the sequence representation
$S_{output}$ is calculated as follows:

$S_{output}=\mathcal{C}\left(AvgPool(s_{t_{i}})+MaxPool(s_{t_{i}})\right),\forall{t_{i}\in
S}$ (8)
$s_{t_{1}},...,,s_{t_{n}}=\left(\overrightarrow{LSTM}\left(t_{1},...,t_{n}\right)\right)||\left({\overleftarrow{LSTM}\left(t_{1},...,t_{n}\right)}\right)$
(9)

where the symbol $||$ is the concatenation operation. $\overrightarrow{}$ and
$\overleftarrow{}$ denote the Bi-directional Long Short-Term Memory (LSTM)
[41] operations respectively.

Finally, the representation for each vulnerable code is computed by combining
the two outputs:

${\mathcal{O}}=G_{output}+S_{output}.$ (10)

###  3.3 Adaptive Re-weighting Module

The adaptive re-weighting module is proposed to better learn the
representations for vulnerability types with different numbers of samples. A
novel training objective is designed to adjust the learning weights for
different types of vulnerabilities according to the training epochs and a
number of associated samples.

Before the module design, we first investigate the popular long-tailed
learning methods in vulnerability type classification, such as class-balanced
strategies [32, 33] and smoothing strategies [30, 34]. We experimentally
evaluate the performance of these methods. And the results are illustrated in
Table II and will be detailed in Section 5.1. We find that focal loss [33]
adds a modulating factor to focus more on tail samples. The label smooth CE
loss [30] uses the smoothing strategies to reduce the focus on head classes.
The experiment results also demonstrate that focal loss performs better for
classifying tails, while the label smooth CE loss helps to improve the
performance of head classes. Based on the observation, we propose a novel
training objective that involves two training branches, i.e., a “tail-focused
branch” for tail classification and “head-focused branch” for head
classification.

Specifically, these two branches take
$\mathcal{O}=\\{\hat{y_{i}}|i=1,2,..,n\\}$ as input, where $n$ denotes the
number of types and $\hat{y_{i}}$ denotes the output predicted by the model.
The adaptive re-weighting method $\mathcal{L}$ is calculated as follows:

$\mathcal{L}=T\cdot\mathcal{L}_{FL}\left({\mathcal{O}}\right)+(1-T)\cdot\mathcal{L}_{LSCE}\left({\mathcal{O}}\right)$
(11)

where $\mathcal{L}_{FL}$ denotes the focal loss which focuses more on learning
the representations of tail classes. $\mathcal{L}_{LSCE}$ denotes the label
smooth CE loss focuses more on learning the representations of head classes.
$T$ denotes a learning weight for the two branches, which are calculated
according to the training epochs:

$T=1-\left(\frac{E_{now}}{E_{max}}\right)^{2}$ (12)

where $E_{now}$ and $E_{max}$ denote the current training epoch and the total
number of training epochs, respectively. The design of $T$ is based on the
assumption that the representations of head classes are more easily learnt in
the long-tailed scenario [42]. In the early training stage (i.e, with a
smaller $E_{now}$), the design will enable the model to focus on learning the
representations of tail classes. As the training epoch increases, $T$ will
gradually decrease, shifting the model’ learning focus to head classes. $T$
ensures that both branches are updated continuously throughout the training
process, avoiding learning conflict between the two branches.

##  4 EXPERIMENTAL Setup

TABLE I: The specific vulnerability types and their corresponding proportion and group of the types of vulnerabilities in this paper. Classes with a sample size of less than 20 are grouped together in the Remain class. “None type” means the vulnerability is not classified into any class. As these vulnerabilities exist in the real world as well, they are also considered to be a vulnerability type. Types | Ratio | Group | Types | Ratio | Group
---|---|---|---|---|---
CWE-119 | 19.94% | Head | CWE-415 | 0.76% | Medium
None type | 19.85% | CWE-732 | 0.62%
CWE-20 | 10.71% | CWE-404 | 0.58%
CWE-399 | 6.90% | CWE-79 | 0.52%
CWE-125 | 5.86% | CWE-19 | 0.52%
CWE-264 | 4.76% | CWE-59 | 0.49%
CWE-200 | 4.72% | CWE-17 | 0.48%
CWE-189 | 3.16% | CWE-400 | 0.45% | Tail
CWE-416 | 3.09% | CWE-772 | 0.43%
CWE-190 | 2.88% | CWE-269 | 0.36%
CWE-362 | 2.61% | CWE-22 | 0.33%
CWE-476 | 2.02% | CWE-369 | 0.32%
CWE-787 | 1.86% | Medium | CWE-18 | 0.32%
CWE-284 | 1.66% | CWE-835 | 0.32%
CWE-254 | 1.15% | Remain class | 1.57%
CWE-310 | 0.88% |  |  |

###  4.1 Research Questions

In order to evaluate LIVABLE, we answer the following research questions:

  1. RQ1:

How well does LIVABLE perform in classifying vulnerability types under the
long-tailed distribution?

  2. RQ2:

How effective is the vulnerability representation learning module in
vulnerability detection?

  3. RQ3:

What is the impact of different components on the type classification
performance of LIVABLE?

  4. RQ4:

What is the influence of hyper-parameters on the performance of LIVABLE?

###  4.2 Datasets

Vulnerability type classification. To obtain the labels of vulnerabilities for
type classification, we extract a new dataset from Fan et al. [25], which
consists of different types of vulnerabilities from 2002 to 2019 and provides
the _Common Weakness Enumeration IDentifier_ (CWE ID) [43] for each vulnerable
function. We obtain a total of 10667 vulnerable functions with 91 different
kinds of CWE-ID. Specifically, as shown in Figure I, we demonstrate the group
and the proportion of head, medium, and tail classes in our dataset. The
percentage of samples in the head, medium, and tail classes amount to 86.50%,
9.52%, and 4.00% respectively. Moreover, we group categories with a sample
size of less than 20 as a new class named _Remain class_.

Vulnerability detection. Our study employs three representative vulnerability
datasets: FFMPeg+Qemu [7], Reveal [8], and Fan et al. [25]. The FFMPeg+Qemu
dataset, previously utilized in Devign [7] with +22K data, exhibits a
vulnerability rate of 45.0% among all instances. The Reveal dataset consists
of +18K instances, and the proportion of vulnerable to non-vulnerable data is
1:9.9. The Fan et al. dataset includes a total of 188,636 C/C++ function
samples, among which 5.7% are vulnerable. We do not use FFMPeg+Qemu [7] and
Reveal [8] datasets for vulnerability type classification since these two
datasets do not contain type information of vulnerability.

###  4.3 Baseline Methods

In vulnerability detection task, we compare LIVABLE with three sequence-based
methods [4, 5, 6] and three state-of-the-art graph-based methods [7, 8, 9].

  1. 1.

VulDeePecker [4]: VulDeePecker embeds the data flow dependency to construct
the code slices. Then, it adopts the BiLSTM to detect buffer error
vulnerabilities and resource management error vulnerabilities.

  2. 2.

Russell et al. [5]: Russell et al. involves the Convolutional Neural Network
(CNN), integrated learning and random forest to vulnerability detection.

  3. 3.

SySeVR [6]: SySeVR constructs the program slices by combining control flow
dependency, data flow dependency, and code statement. It embeds program slices
as input and uses Recursive Neural Networks (RNN).

  4. 4.

Devign [7]: Devign constructs code structure graph from functions and
leverages Gated Graph Neural Network (GGNN) for classification.

  5. 5.

Reveal [8]: Reveal leverages code property graph (CPG) as input and also
adopts GGNN for extracting features. Then it utilizes a multi-layer perceptron
for vulnerability detection.

  6. 6.

IVDetect [9]: IVDetect constructs Program Dependency Graph (PDG) and designs
the feature attention GCN for vulnerability detection.

In vulnerability type classification, we compare LIVABLE with graph-based
methods: Devign and Reveal. These methods are designed for the software
vulnerability detection task and achieve the best-performing results. We also
use the focal loss [33] and label smooth CE (LSCE) loss [30] to replace CE
loss, which has been introduced in Section 3.3. Furthermore, we compare with
label aware smooth loss [34], class-balanced loss [32], and class-balanced
focal loss [32] that mitigates the imbalance of long-tailed distribution.
Label aware smooth loss adds a probability distribution factor based on LSCE,
which considers the predicted probability distributions of different classes.
Class-balanced loss and class-balanced focal loss are modified from CE loss
and focal loss, respectively. They add a weight inversely related to the
number of class samples to tackle long-tailed distribution.

###  4.4 Implementation Details

In the experiment section, we randomly split the dataset into training,
validation, and testing sets by the number of classes in a ratio of 8:1:1. To
ensure the fairness of the experiments, we use the same data split in all
experiments. We train the vulnerability detection model for 100 epochs and the
vulnerability type classification model for 50 epochs on a server with an
NVIDIA GeForce RTX 3090.

We leverage Word2Vec [44] to initialize the node representation in the graph
branch and the token representation in the sequence branch, where both vectors
have a dimension of $d=128$. The number of layers in the graph branch is
$L=16$ and the limitation and the source code length in the sequence branch is
$n=512$. The dimension of the hidden vector in the sequence branch is set as
512.

###  4.5 Performance Metrics

We use the following four widely-used performance metrics in our evaluation:

Accuracy: $Accuracy=\frac{TP+TN}{TP+FP+TN+FN}$. Accuracy is the proportion of
correctly classified instances to all instances. _TP_ is the number of true
positives, _TN_ is the number of true negatives. and $TP+FP+TN+FN$ represents
the number of all instances. To evaluate the classification performance of
different classes of vulnerabilities, we use $Head$, $Medium$, $Tail$ to
denote the proportion of the head, medium, and tail CWE samples that are
detected correctly, respectively.

Precision: $Precision=\frac{TP}{TP+FP}$. Precision is the proportion of
relevant instances among those retrieved. _TP_ is the number of true positives
and _FP_ is the number of false positives.

Recall: $Recall=\frac{TP}{TP+FN}$. Recall is the proportion of relevant
instances retrieved. _TP_ is the number of true positives and _FN_ is the
number of false negatives.

F1 score: $F1score=2*\frac{Precision*Recall}{Precision+Recall}$. F1 score is
the geometric mean of precision and recall and indicates the balance between
them.

##  5 Experimental Results

###  5.1 RQ1: Evaluation on Vulnerability Types Classification

TABLE II: Under the long-tail data distribution, comparison results between
LIVABLE and the baselines in vulnerability type classification. VRL module
denotes the vulnerability representation learning module.

Metrics(%) Baseline | Devign | Reveal | VRL module in LIVABLE
---|---|---|---
Re-weighting | Head | Medium | Tail | Accuracy | Head | Medium | Tail | Accuracy | Head | Medium | Tail | Accuracy
CE loss | 38.26 | 7.35 | 23.08 | 34.99 | 39.39 | 42.65 | 46.15 | 39.83 | 60.61 | 51.47 | 38.46 | 59.32
Label aware smooth loss | 37.14 | 25.00 | 23.08 | 35.70 | 38.26 | 42.65 | 46.15 | 38.83 | 63.50 | 52.94 | 46.15 | 62.16
Label smooth loss | 37.78 | 4.41 | 23.08 | 34.28 | 39.07 | 42.65 | 46.15 | 39.54 | 63.50 | 55.88 | 46.15 | 62.45
Class-balanced loss | 27.97 | 36.76 | 38.46 | 29.02 | 37.30 | 44.12 | 46.15 | 38.12 | 60.45 | 54.41 | 53.85 | 59.74
Class-balanced focal loss | 24.44 | 7.35 | 23.08 | 22.76 | 27.17 | 42.65 | 46.15 | 29.02 | 54.82 | 54.41 | 53.85 | 54.77
Focal loss | 38.91 | 14.71 | 38.46 | 37.27 | 38.59 | 41.18 | 46.15 | 38.98 | 61.09 | 52.94 | 53.85 | 60.17
Adaptive re-weighting module | 39.27 | 37.29 | 50.00 | 39.39 | 41.00 | 47.06 | 46.15 | 41.68 | 64.79 | 58.82 | 53.85 | 64.01

To answer this research question, we compare our approach with the previous
methods of vulnerability type classification under long-tailed distribution.

####  5.1.1 Compared with baselines

As shown in Table II, the proposed LIVABLE consistently outperforms all the
baseline methods in the long-tailed distribution. Our approach achieves an
accuracy of 64.01%, which is an absolute improvement of 26.74% and 24.18% over
Devign and Reveal, respectively. Compared with the baseline methods, we focus
on modeling vulnerability patterns from two perspectives based on the graph
branch and the sequence branch. The results show that our approach can learn
vulnerabilities more efficiently from different perspectives. For the head,
medium and tail classes, we can see that LIVABLE achieves the best performance
compared to the previous methods, with the improvement of 25.40%, 16.17% and
7.7% respectively. It demonstrates that our approach is better at capturing
the differences between the different vulnerability types.

Figure 3 shows the results of LIVABLE with Devign and Reveal in terms of
vulnerability type classification, with respect to each vulnerability type. In
the head class, LIVABLE performs better in the detection of unknown types
(i.e. None type) of vulnerabilities. This may be due to that the vulnerability
representation learning module captures more information from the code
snippets and obtains a more discriminative vulnerability representation. In
the medium and tail classes, LIVABLE also performs better than the previous
method, especially for CWE-787, CWE-284 and CWE-25, CWE-22, etc. This is due
to the fact that LIVABLE gives higher weighted attention to the tail class
samples. In general, LIVABLE outperforms baselines for classifying
vulnerability types.

In addition, we note that the performance of Reveal on the tail data is not
affected by the loss functions used. This may be due to the way that the
learned vulnerability representations are constrained by over-smoothing and
the model has difficulty learning the samples of tail class.

####  5.1.2 Compared with re-weighting methods

In this subsection, we experiment with different re-weighting methods to solve
the long-tailed problem of vulnerability type classification.

Cross-entropy Loss: When using the CE-loss, our LIVABLE is more accurate in
identifying head classes than tail classes. CE-Loss encourages the whole model
to be over-confident in the head classes for massive data. The experiment
shows that the head class samples performed 9.14% and 22.15% better than the
medium and tail classes, respectively, in terms of accuracy.

Smoothing strategies: Smoothing strategies include label smooth loss and
label-aware smooth loss. They are another regularization technique that
encourages the model to be less over-confident in the head classes. We use
these methods to replace CE loss and they both improve the accuracy of the
head class by 2.89% in LIVABLE. The use of the smoothing strategies in other
methods did not work, which is probably due to the poor vulnerability
representation they captured.

Class-balanced strategies: The class-balanced strategies consist of the class-
balanced loss and class-balanced focal loss, which assigns different weights
for classes and instances. They lead to an average increase in accuracy of
2.94% and 15.39% for the medium and tail classes respectively, but an average
decrease of 2.98% for the head classes. It shows that class-balanced
strategies focus more on the tail data but will cause a decrease in the
accuracy of the head classes.

Figure 3: The accuracy of each vulnerability type in LIVABLE. The X-axis
denotes the vulnerability type. The Y-axis on the left and right indicate the
number of vulnerabilities and accuracy respectively. The lines in yellow,
purple, and red represent the accuracy of Devign, Reveal, and LIVABLE
respectively.

Adaptive re-weighting module: Compared with other re-weighting methods, the
adaptive re-weighting module achieves the highest accuracy in all four
metrics. Specifically, the adaptive re-weighting module achieves an absolute
improvement of 4.24% in overall accuracy.The tail-focused branch leverages the
advantages of class-balanced strategies, which assign higher weights to the
more challenging tail samples in the early stages of training. It improves the
average accuracy in medium and tail classes by 5.15% and 5.13% respectively.
In the later phase of training, the head-focused branch involves the smoothing
strategy, which calculates the loss of the soft value of the label. It is
effective as a means of coping with label noise [45, 46] in the vulnerability
types classification. The shifting weight ensures that the head and tail
classes can be constantly learned in the whole training process, which avoids
conflicts with each other. In summary, the adaptive re-weighting module is
able to learn representations of tail data efficiently based on a limited
number of samples.

Answer to RQ1: LIVABLE outperforms all the baseline methods in terms of
accuracy for head classes, medium classes, and tail classes, as well as for
all samples. In particular, the adaptive re-weighting module achieves better
performance than existing re-weighting methods.

###  5.2 RQ2: Performance on Vulnerability Detection

To answer this research question, we explore the performance of the
vulnerability representation learning (VRL) module in LIVABLE and compare it
with other baseline methods. Table III shows the overall results.

The proposed VRL module outperform all the baseline methods: Overall, VRL
module achieves better results on the FFMPeg+Qemu, Reveal and Fan et al.
datasets. When considering all the performance metrics regarding the three
datasets (12 combination cases), VRL module obtains the best performance in 11
out of 12 cases. On all three datasets, VRL module outperforms all baseline
methods in terms of accuracy and F1 score metrics. Especially in terms of F1
score, the relative improvements are 8.36%, 23.74%, and 68.78%, respectively.
Compared to the best-performing baseline method Reveal, VRL module achieves an
average improvement of 9.88%, 67.49%, 2.09%, and 33.63% on the four metrics
across all datasets. This indicates that the vulnerability representation
learning module performs better on vulnerability detection than previous
methods, obtaining a more discriminative code representation.

The combination of the graph-based branch and the token-based branch can
achieve better performance: The experimental results also show that the graph-
based methods perform better in most cases. The token-based methods perform
well on some metrics. For example, in Fan et al., the token-based methods
obtain better performance than graph-based methods on precision metrics
overall. The reason may be attributed to that token-based methods are better
at capturing semantic information in the code, whereas graph-based methods are
more attentive to the structure information in the code. Thus, VRL module
combines the advantages of both graph-based and token-based methods to capture
more information and obtain a more discriminating code representation in
vulnerability detection.

TABLE III:  Comparison results between vulnerability representation learning
(VRL) module and the baselines on the three datasets in vulnerability
detection. “-” means that the baseline does not apply to the dataset in this
scenario. The best result for each metric is highlighted in bold. The shaded
cells represent the performance of the top-3 best methods in each metric.
Darker cells represent better performance.

Metrics(%) Dataset | FFMPeg+Qemu [7] | Reveal [8] | Fan et al.. [25]
---|---|---|---
Baseline | Accuracy | Precision | Recall | F1 score | Accuracy | Precision | Recall | F1 score | Accuracy | Precision | Recall | F1 score
VulDeePecker | 49.61 | 46.05 | 32.55 | 38.14 | 76.37 | 21.13 | 13.10 | 16.17 | 81.19 | 38.44 | 12.75 | 19.15
Russell et al.. | 57.60 | 54.76 | 40.72 | 46.71 | 68.51 | 16.21 | 52.68 | 24.79 | 86.85 | 14.86 | 26.97 | 19.17
SySeVR | 47.85 | 46.06 | 58.81 | 51.66 | 74.33 | 40.07 | 24.94 | 30.74 | 90.10 | 30.91 | 14.08 | 19.34
Devign | 56.89 | 52.50 | 64.67 | 57.95 | 87.49 | 31.55 | 36.65 | 33.91 | 92.78 | 30.61 | 15.96 | 20.98
Reveal | 61.07 | 55.50 | 70.70 | 62.19 | 81.77 | 31.55 | 61.14 | 41.62 | 87.14 | 17.22 | 34.04 | 22.87
IVDetect | 57.26 | 52.37 | 57.55 | 54.84 | - | - | - | - | - | - | - | -
VRL module | 64.84 | 57.87 | 80.67 | 67.39 | 93.53 | 52.27 | 50.55 | 51.50 | 95.05 | 40.04 | 37.27 | 38.60

Answer to RQ2: The vulnerability representation learning module outperforms
all baseline methods in terms of accuracy and F1 scores in vulnerability
detection. In particular, it improves the accuracy of the three datasets by
3.77%, 6.04% and 2.27% respectively over the best baseline method.

###  5.3 RQ3: Ablation Study

To answer this research question, we explore the effect of vulnerability
representation learning module and adaptive re-weighting module on the
performance of LIVABLE by performing an ablation study on vulnerability type
classification.

####  5.3.1 Vulnerability representation learning module

To explore the contribution of the vulnerability representation learning
module, we construct the following two variants in the module for comparison:
(1) only using differentiated propagation-based GNN (denoted as w/o sequence
representation) to validate the impact of the sequence-to-sequence model (2)
only using sequence-to-sequence model (denoted as w/o graph representation) to
verify the effectiveness of differentiated propagation-based GNN.

As shown in Table IV, all variations contribute positively to the overall
performance. This indicates that the combination of differentiated
propagation-based GNN and sequence-to-sequence model can better help the model
to capture more discriminative representations from the source code. Overall,
the impact of combining sequence-to-sequence model is greater, which achieves
a 14.65% improvement while combining differentiated propagation-based GNN
gives only a 5.41% performance improvement.

####  5.3.2 Adaptive re-weighting module

Then we explore the contribution of the adaptive re-weighting module to the
performance of vulnerable type classification. We construct three variations,
including (1) using the cross-entropy loss instead of the adaptive re-
weighting module (denoted as w/o adaptive) (2) removing the tail-focused
branch while keeping the head-focused branch (denoted as w/o tail-focused
branch) (3) removing head-focused branch while keeping tail-focused branch
(denoted as w/o head-focused branch).

The results of ablation studies are shown in Table IV. We find that the
performance of all the variants is lower than LIVABLE, which indicates that
all the components contribute to the overall performance of LIVABLE. The head-
focused branch has more influence on the differentiated propagation-based GNN,
while the tail-focused branch enables the sequence-to-sequence model to have a
greater performance improvement.

TABLE IV: Results of ablation study in vulnerability type classification.

Metrics(%) Methods | w/o sequence representation | w/o graph representation | LIVABLE
---|---|---|---
Methods | Head | Medium | Tail | Accuracy | Head | Medium | Tail | Accuracy | Head | Medium | Tail | Accuracy
w/o adaptive | 46.95 | 25.00 | 38.46 | 44.67 | 56.59 | 32.35 | 38.46 | 53.91 | 60.61 | 51.47 | 38.46 | 59.32
w/o tail-focused branch | 49.20 | 44.12 | 30.77 | 48.36 | 60.29 | 47.06 | 38.46 | 58.61 | 63.50 | 55.88 | 46.15 | 62.45
w/o head-focused branch | 48.71 | 33.82 | 30.77 | 46.94 | 62.38 | 51.47 | 46.15 | 61.02 | 61.41 | 45.59 | 38.46 | 59.46
LIVABLE | - | - | - | - | - | - | - | - | 64.79 | 58.82 | 53.85 | 64.01

Answer to RQ3: The vulnerability representation learning module combines the
differentiated propagation-based GNN and sequence-to-sequence model, in which
both contributes significantly to the performance of LIVABLE, with an
improvement of 5.41% and 14.65% respectively in terms of accuracy. The
adaptive re-weighting module also has a positive effect on model performance.

###  5.4 RQ4: Parameter Analysis

TABLE V: The impact of the number of GNN layers in the graph branch and hidden size in the sequence branch on the performance of LIVABLE.  Graph Branch | Sequence Branch
---|---
Layer number | Accuracy(%) | Hidden size | Accuracy(%)
12 | 62.07 | 128 | 56.53
14 | 62.78 | 256 | 59.23
16 | 64.01 | 512 | 64.01
18 | 62.92 | 768 | 62.07
20 | 61.50 | 1024 | 62.93

To answer this research question, we explore the impact of hyper-parameters in
the LIVABLE, including the layer number of the differentiated propagation-
based GNN, and the hidden size in the sequence-to-sequence model.

####  5.4.1 Number of GNN layers

Table V shows the accuracy of LIVABLE with different layers of GNN on the
vulnerability type classification. In the previous work, GNNs usually achieve
the best performance at 2 layers. However, with the number of layers set to
16, LIVABLE achieves a best performance of 64.01%. This indicates that LIVABLE
can learn the relationships between more distant nodes than traditional
methods, effectively alleviating the problem of over-smoothing. In addition,
due to the limited number of nodes in the code structure graph, the number of
layers of the GNN model cannot be increased indefinitely, and the accuracy
rate starts to decrease when the number of layers equals 18. In summary, the
model benefits from the use of more layers to capture relationships between
distant nodes, enhancing the model’s ability to capture information about the
graph structure.

####  5.4.2 Size of hidden layers

We also experiment with MLPs with different sizes of hidden neurons. We use
the same setup in the previous experiments. We experiment with the task of
vulnerability type classification. The results are shown in Table V and show
that the LIVABLE performs best when using a hidden size of 512. This may be
due to the fact that the task requires the detection of a larger number of
vulnerability types. When more dimensions are used (more than 512), the
performance starts to drop, which may be due to the over-fitting problem.

Answer to RQ4: The hyper-parameter settings of GNN layer number and hidden
size can impact the performance of LIVABLE. The proposed differentiated
propagation-based GNN can use more layers for learning to distinguish node
representations.

##  6 Discussion

###  6.1 Why Does LIVABLE Work?

In this section, we identify the following two advantages of LIVABLE, which
can explain its effectiveness in vulnerability type classification. We
visualize the vulnerability representations learnt by LIVABLE and the best
baseline Reveal via the popular T-SNE technique [47], as shown in Figure 4 (a)
and (b), respectively.

(1) The ability to learn the vulnerability representation. The proposed
vulnerability representation learning module helps LIVABLE to learn the
vulnerability representation. More specifically, the proposed differentiated
propagation GNN branch alleviates the over-smoothing problem and the sequence
branch enhances the model’s ability for capturing semantics information. As
shown in Figure 4 (a) and (b), compared with the Reveal, LIVABLE can enhance
the discriminability of representation on the type classification, in which
head and tail classes are clustered more tightly.

(2) The ability to capture the representation of the tail classes. Although
Reveal also identifies some samples from the tail class, they are usually
mixed and clustered with samples from other classes. In comparison, LIVABLE is
easier to detect samples of the tail class. We choose CWE-22 in tail classes
as an example, as shown in Figure 4 (c). The example code attempts to validate
a given input path by checking it against an allowlist and once validated
delete the given file. But the “../” sequence in line 3 will cause the program
to delete the important file in the parent directory. As can be seen in Figure
4 (b), the CWE-22 sample in the red box can be identified because its distance
with the other class samples are clearer.

Figure 4: The case study of LIVABLE, which mainly contains three parts: (a)
T-SNE visualization of Reveal, (b) T-SNE visualization of LIVABLE, and (c) a
CWE-22 [48] example in tail class. In (a) and (b), the blue dots represent
head samples, the green dots represent mid samples and the pink dots represent
tail samples. In (c), the red-colored code is vulnerable code.

###  6.2 Threats to Validity

One threat to validity comes from the size of our constructed dataset.
Following the previous methods, we use a C/C++ dataset built for vulnerability
detection. In the task of vulnerability type classification, we extract 91
types of vulnerability functions from this dataset and construct the new
dataset. However, in the real world, there are more than 300 types of
vulnerabilities. In the future, we will collect a more realistic benchmark for
evaluation.

The second threat to validity is that our proposed LIVABLE has only
experimented on the C/C++ dataset. Experiments have not been conducted on
datasets from other programming languages, such as Java and Python. In the
future, we will select more programming languages and corresponding datasets
to further evaluate the effectiveness of LIVABLE.

Another threat comes from the implantation of baselines. Since Devign [7] does
not publish their implementation and hyper-parameters. So we reproduce Devign
based on the Reveal’s [8] implementation and re-implement the method to the
best of our abilities.

##  7 Related Work

###  7.1 Learning-Based Vulnerability Detection

In recent years, learning-based methods have been widely used for
vulnerability detection tasks. The researcher first uses the Machine Learning
(ML)-based method [49, 50, 51, 52, 53] in vulnerability detection. For
example, Neuhauset al. [54] extract the dependency information matrix and
vulnerability vectors from the source code, and use the support vector machine
to detect them. Griecoet al. [55] abstract features from the C-standard
library and use multiple machine learning models, such as logistic regression
and random forest.

ML-based methods rely heavily on manual feature extraction, which is time-
consuming and may require much effort. Therefore, Deep Learning (DL)-based
methods [56, 57, 58, 59, 60] learn the input representation from the source
code, which can better capture the vulnerability patterns. Depending on the
input generated from source code and training model types, DL-based approaches
can group into two different types: sequence-based and graph-based methods.
Sequence-based approaches utilize source code tokens as their model inputs.
VulDeePecker [4] extracts data flow information from source code and adapts
BiLSTM to detect buffer error vulnerabilities. SySeVR [6] combines both
control flow and data flow to generate program slices and utilizes a
bidirectional RNN for code vulnerability detection. Graph-based methods
capture more structural information than sequence-based methods, which achieve
better performance on vulnerability detection tasks. Devign [7] builds graphs
combining AST, CFG, DDG and NCS edges and uses the GGNN model for
vulnerability detection. IVDetect [9] constructs the Program Dependency Graph
(PDG) and proposes feature attention GCN to learn the graph representation.

However, directly adopting these approaches tends to result in poor
performance in vulnerability type classification, which is limited by class-
imbalanced and over-smoothing problems in GNNs. In this paper, we propose a
vulnerability representation learning module to boost the model to learn the
vulnerability representation and capture vulnerability types.

###  7.2 Class Re-weighting Strategies

Long-tailed classification has attracted increasing attention due to the
prevalence of imbalanced data in real-world applications [42, 30, 29, 31, 61].
It leads to a small portion of classes having massive sample points but the
others contain only a few samples, which makes the model ignore the
identification of tail classes. Recent studies have mainly pursued re-
weighting strategies. For example, Lin et al. propose Focal Loss [33], which
adjusts the standard CE-loss to reduce the relative loss for well-classified
samples and focus more on rare samples that are misclassified during model
training. Cui et al. [32] design the class-balanced loss, which adds a weight
related to the number of samples in the class to make the model focus on the
tail class. Another effective approach is to reduce the model’s excessive
focus on head classes. Zhong et al. [34] use a label smoothing strategy and
combines the probability distributions of different classes to mitigate the
overweighting of the head classes. In this paper, we propose an adaptive re-
weighting module to learn the vulnerability representation in the tail
classes.

##  8 Conclusion

In this paper, we show the distribution of vulnerability types in the real
world belongs to the long-tailed distribution and the importance of the tail
class vulnerabilities. We propose LIVABLE, a long-tailed software
vulnerability type classification approach, which involves the vulnerability
representation learning module and adaptive re-weighting module. It can
distinguish node representation and enhance vulnerability representations. It
can also predict vulnerability types in the long-tailed distribution. Our
experimental results on vulnerability type classification validate the
effectiveness of LIVABLE, and the results in vulnerability detection and the
ablation studies further demonstrate the advantages of LIVABLE. The future
works include combining the vulnerability repair with type and generating
human-readable or explainable reports due to the vulnerability type.

## Data availability

Our source code as well as experimental data are available at:
https://github.com/LIVABLE01/LIVABLE.

## References

  * [1] Google., “Key statistics of the google bug bounty program,” 2022. [Online]. Available: https://bughunters.google.com/about/key-stats
  * [2] “Common weakness enumeration,” [n.d.]. [Online]. Available: https://cwe.mitre.org/data/definitions/119.html
  * [3] Microsoft. (2021) Microsoft bug bounty programs year in review: $13.6m in rewards. [Online]. Available: https://msrc-blog.microsoft.com/2021/07/08/microsoft-bug-bounty-programs-year-in-review-13-6m-in-rewards/
  * [4] Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” in _25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-21, 2018_. The Internet Society, 2018.
  * [5] R. L. Russell, L. Y. Kim, L. H. Hamilton, T. Lazovich, J. Harer, O. Ozdemir, P. M. Ellingwood, and M. W. McConley, “Automated vulnerability detection in source code using deep representation learning,” in _17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018, Orlando, FL, USA, December 17-20, 2018_ , M. A. Wani, M. M. Kantardzic, M. S. Mouchaweh, J. Gama, and E. Lughofer, Eds. IEEE, 2018, pp. 757–762.
  * [6] Z. Li, D. Zou, S. Xu, H. Jin, Y. Zhu, and Z. Chen, “Sysevr: A framework for using deep learning to detect software vulnerabilities,” _IEEE Trans. Dependable Secur. Comput._ , vol. 19, no. 4, pp. 2244–2258, 2022.
  * [7] Y. Zhou, S. Liu, J. K. Siow, X. Du, and Y. Liu, “Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks,” in _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, Eds., 2019, pp. 10 197–10 207.
  * [8] S. Chakraborty, R. Krishna, Y. Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet?” _IEEE Trans. Software Eng._ , vol. 48, no. 9, pp. 3280–3296, 2022.
  * [9] Y. Li, S. Wang, and T. N. Nguyen, “Vulnerability detection with fine-grained interpretations,” in _ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021_ , D. Spinellis, G. Gousios, M. Chechik, and M. D. Penta, Eds. ACM, 2021, pp. 292–303.
  * [10] J. L. Elman, “Finding structure in time,” _Cogn. Sci._ , vol. 14, no. 2, pp. 179–211, 1990.
  * [11] M. Gori, G. Monfardini, and F. Scarselli, “A new model for learning in graph domains,” in _Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005._ , vol. 2, 2005, pp. 729–734 vol. 2.
  * [12] Y. Li, D. Tarlow, M. Brockschmidt, and R. S. Zemel, “Gated graph sequence neural networks,” in _4th International Conference on Learning Representations, ICLR 2016_ , 2016.
  * [13] “National vulnerability database,” [n.d.]. [Online]. Available: https://nvd.nist.gov/
  * [14] B. Shuai, H. Li, M. Li, Q. Zhang, and C. Tang, “Automatic classification for vulnerability based on machine learning,” in _IEEE International Conference on Information and Automation, ICIA 2013, Yinchuan, China, August 26-28, 2013_. IEEE, 2013, pp. 312–318.
  * [15] S. Na, T. Kim, and H. Kim, “A study on the classification of common vulnerabilities and exposures using naïve bayes,” in _Proceedings of the 11th International Conference On Broad-Band Wireless Computing, Communication and Applications_ , ser. Lecture Notes on Data Engineering and Communications Technologies, vol. 2. Springer, 2016, pp. 657–662.
  * [16] D. Zou, S. Wang, S. Xu, Z. Li, and H. Jin, “$\mu$vuldeepecker: A deep learning-based system for multiclass vulnerability detection,” _IEEE Trans. Dependable Secur. Comput._ , vol. 18, no. 5, pp. 2224–2236, 2021.
  * [17] “Common vulnerability scoring system sig,” [n.d.]. [Online]. Available: https://www.first.org/cvss/
  * [18] “Common vulnerability scoring system version 3.0,” [n.d.]. [Online]. Available: https://www.first.org/cvss/v3-0/
  * [19] “Common vulnerability scoring system version 3.0 qualitative-severity-rating-scale,” [n.d.]. [Online]. Available: https://www.first.org/cvss/v3.0/specification-document#Qualitative-Severity-Rating-Scale
  * [20] “Common weakness enumeration,” [n.d.]. [Online]. Available: https://cwe.mitre.org/data/definitions/507.html
  * [21] D. Lukovnikov and A. Fischer, “Improving breadth-wise backpropagation in graph neural networks helps learning long-range dependencies,” in _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 2021, pp. 7180–7191.
  * [22] U. Alon and E. Yahav, “On the bottleneck of graph neural networks and its practical implications,” in _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * [23] D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. B. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, “Graphcodebert: Pre-training code representations with data flow,” in _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * [24] J. Zhang, X. Wang, H. Zhang, H. Sun, K. Wang, and X. Liu, “A novel neural source code representation based on abstract syntax tree,” in _Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019_ , J. M. Atlee, T. Bultan, and J. Whittle, Eds. IEEE / ACM, 2019, pp. 783–794.
  * [25] J. Fan, Y. Li, S. Wang, and T. N. Nguyen, “A C/C++ code vulnerability dataset with code changes and CVE summaries,” in _MSR ’20: 17th International Conference on Mining Software Repositories, Seoul, Republic of Korea, 29-30 June, 2020_ , S. Kim, G. Gousios, S. Nadi, and J. Hejderup, Eds. ACM, 2020, pp. 508–512.
  * [26] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017\.
  * [27] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” _CoRR_ , vol. abs/1710.10903, 2017\.
  * [28] K. Xu, C. Li, Y. Tian, T. Sonobe, K. Kawarabayashi, and S. Jegelka, “Representation learning on graphs with jumping knowledge networks,” in _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , ser. Proceedings of Machine Learning Research, J. G. Dy and A. Krause, Eds., vol. 80. PMLR, 2018, pp. 5449–5458.
  * [29] Y. Zhang, B. Kang, B. Hooi, S. Yan, and J. Feng, “Deep long-tailed learning: A survey,” _CoRR_ , vol. abs/2110.04596, 2021.
  * [30] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_. IEEE Computer Society, 2016, pp. 2818–2826.
  * [31] J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and J. Yan, “Equalization loss for long-tailed object recognition,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_. Computer Vision Foundation / IEEE, 2020, pp. 11 659–11 668.
  * [32] Y. Cui, M. Jia, T. Lin, Y. Song, and S. J. Belongie, “Class-balanced loss based on effective number of samples,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, 2019, pp. 9268–9277.
  * [33] T. Lin, P. Goyal, R. B. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_. IEEE Computer Society, 2017, pp. 2999–3007.
  * [34] Z. Zhong, J. Cui, S. Liu, and J. Jia, “Improving calibration for long-tailed recognition,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 2021, pp. 16 489–16 498.
  * [35] Z. Zhang and M. R. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” in _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., 2018, pp. 8792–8802.
  * [36] H. Guo and S. Wang, “Long-tailed multi-label visual recognition by collaborative training on uniform and re-balanced samplings,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 2021, pp. 15 089–15 098. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2021/html/Guo\\_Long-Tailed\\_Multi-Label\\_Visual\\_Recognition\\_by\\_Collaborative\\_Training\\_on\\_Uniform\\_and\\_CVPR\\_2021\\_paper.html
  * [37] K. W. Church, “Word2vec,” _Nat. Lang. Eng._ , vol. 23, no. 1, pp. 155–162, 2017.
  * [38] Y. Boureau, J. Ponce, and Y. LeCun, “A theoretical analysis of feature pooling in visual recognition,” in _Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel_ , J. Fürnkranz and T. Joachims, Eds. Omnipress, 2010, pp. 111–118. [Online]. Available: https://icml.cc/Conferences/2010/papers/638.pdf
  * [39] K. Yue, F. Xu, and J. Yu, “Shallow and wide fractional max-pooling network for image classification,” _Neural Comput. Appl._ , vol. 31, no. 2, pp. 409–419, 2019.
  * [40] S. Woo, J. Park, J. Lee, and I. S. Kweon, “CBAM: convolutional block attention module,” in _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII_ , ser. Lecture Notes in Computer Science, V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds., vol. 11211. Springer, 2018, pp. 3–19.
  * [41] P. Zhou, W. Shi, J. Tian, Z. Qi, B. Li, H. Hao, and B. Xu, “Attention-based bidirectional long short-term memory networks for relation classification,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 2: Short Papers_. The Association for Computer Linguistics, 2016.
  * [42] B. Zhou, Q. Cui, X. Wei, and Z. Chen, “BBN: bilateral-branch network with cumulative learning for long-tailed visual recognition,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_. Computer Vision Foundation / IEEE, 2020, pp. 9716–9725.
  * [43] “Common weakness enumeration,” [n.d.]. [Online]. Available: http://cwe.mitre.org/
  * [44] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” in _1st International Conference on Learning Representations, ICLR 2013_ , 2013.
  * [45] M. Lukasik, S. Bhojanapalli, A. K. Menon, and S. Kumar, “Does label smoothing mitigate label noise?” in _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 6448–6458.
  * [46] R. Croft, M. A. Babar, and M. M. Kholoosi, “Data quality for software vulnerability datasets,” _CoRR_ , vol. abs/2301.05456, 2023.
  * [47] V. der Maaten, Laurens, and H. Geoffrey, “Visualizing data using t-sne.” _Journal of machine learning research_ , vol. 9, no. 11, 2008.
  * [48] “Common weakness enumeration,” [n.d.]. [Online]. Available: https://cwe.mitre.org/data/definitions/22.html
  * [49] S. Forrest, S. A. Hofmeyr, A. Somayaji, and T. A. Longstaff, “A sense of self for unix processes,” in _1996 IEEE Symposium on Security and Privacy, May 6-8, 1996, Oakland, CA, USA_. IEEE Computer Society, 1996, pp. 120–128.
  * [50] F. Yamaguchi, F. F. Lindner, and K. Rieck, “Vulnerability extrapolation: Assisted discovery of vulnerabilities using machine learning,” in _5th USENIX Workshop on Offensive Technologies, WOOT’11, August 8, 2011, San Francisco, CA, USA, Proceedings_ , D. Brumley and M. Zalewski, Eds. USENIX Association, 2011, pp. 118–127.
  * [51] I. Santos, J. Devesa, F. Brezo, J. Nieves, and P. G. Bringas, “OPEM: A static-dynamic approach for machine-learning-based malware detection,” in _International Joint Conference CISIS’12-ICEUTE’12-SOCO’12 Special Sessions, Ostrava, Czech Republic, September 5th-7th, 2012_ , ser. Advances in Intelligent Systems and Computing, Á. Herrero, V. Snásel, A. Abraham, I. Zelinka, B. Baruque, H. Quintián-Pardo, J. L. Calvo-Rolle, J. Sedano, and E. Corchado, Eds., vol. 189. Springer, 2012, pp. 271–280.
  * [52] S. Neuhaus, T. Zimmermann, C. Holler, and A. Zeller, “Predicting vulnerable software components,” in _Proceedings of the 2007 ACM Conference on Computer and Communications Security, CCS 2007, Alexandria, Virginia, USA, October 28-31, 2007_ , P. Ning, S. D. C. di Vimercati, and P. F. Syverson, Eds. ACM, 2007, pp. 529–540.
  * [53] Y. Shin, A. Meneely, L. A. Williams, and J. A. Osborne, “Evaluating complexity, code churn, and developer activity metrics as indicators of software vulnerabilities,” _IEEE Trans. Software Eng._ , vol. 37, no. 6, pp. 772–787, 2011.
  * [54] S. Neuhaus and T. Zimmermann, “The beauty and the beast: Vulnerabilities in red hat’s packages,” in _2009 USENIX Annual Technical Conference, San Diego, CA, USA, June 14-19, 2009_ , G. M. Voelker and A. Wolman, Eds. USENIX Association, 2009.
  * [55] G. Grieco, G. L. Grinblat, L. C. Uzal, S. Rawat, J. Feist, and L. Mounier, “Toward large-scale vulnerability discovery using machine learning,” in _Proceedings of the Sixth ACM on Conference on Data and Application Security and Privacy, CODASPY 2016, New Orleans, LA, USA, March 9-11, 2016_ , E. Bertino, R. S. Sandhu, and A. Pretschner, Eds. ACM, 2016, pp. 85–96.
  * [56] G. Lin, J. Zhang, W. Luo, L. Pan, and Y. Xiang, “POSTER: vulnerability discovery with function representation learning from unlabeled projects,” in _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, Dallas, TX, USA, October 30 - November 03, 2017_ , B. Thuraisingham, D. Evans, T. Malkin, and D. Xu, Eds. ACM, 2017, pp. 2539–2541.
  * [57] J. Li, P. He, J. Zhu, and M. R. Lyu, “Software defect prediction via convolutional neural network,” in _2017 IEEE International Conference on Software Quality, Reliability and Security, QRS 2017, Prague, Czech Republic, July 25-29, 2017_. IEEE, 2017, pp. 318–328.
  * [58] J. Harer, O. Ozdemir, T. Lazovich, C. P. Reale, R. L. Russell, L. Y. Kim, and P. Chin, “Learning to repair software vulnerabilities with generative adversarial networks,” in _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., 2018, pp. 7944–7954.
  * [59] S. Wang, T. Liu, and L. Tan, “Automatically learning semantic features for defect prediction,” in _Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016_ , L. K. Dillon, W. Visser, and L. A. Williams, Eds. ACM, 2016, pp. 297–308.
  * [60] M. White, M. Tufano, C. Vendome, and D. Poshyvanyk, “Deep learning code fragments for code clone detection,” in _Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, ASE 2016, Singapore, September 3-7, 2016_ , D. Lo, S. Apel, and S. Khurshid, Eds. ACM, 2016, pp. 87–98.
  * [61] S. Park, J. Lim, Y. Jeon, and J. Y. Choi, “Influence-balanced loss for imbalanced visual classification,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 715–724.