# Ranking Warnings of Static Analysis Tools Using Representation Learning

Kien-Tuan Ngo, Dinh-Truong Do, Thu-Trang Nguyen, Hieu Dinh Vo  Faculty of
Information Technology, VNU University of Engineering and Technology, Vietnam
Email:{tuanngokien, 17021090, trang.nguyen<EMAIL_ADDRESS>
###### Abstract

Static analysis tools are frequently used to detect potential vulnerabilities
in software systems. However, an inevitable problem of these tools is their
large number of warnings with a high false positive rate, which consumes time
and effort for investigating. In this paper, we present DeFP, a novel method
for ranking static analysis warnings. Based on the intuition that warnings
which have similar contexts tend to have similar labels (true positive or
false positive), DeFP is built with two BiLSTM models to capture the patterns
associated with the contexts of labeled warnings. After that, for a set of new
warnings, DeFP can calculate and rank them according to their likelihoods to
be true positives (i.e., actual vulnerabilities). Our experimental results on
a dataset of 10 real-world projects show that using DeFP, by investigating
only 60% of the warnings, developers can find +90% of actual vulnerabilities.
Moreover, DeFP improves the state-of-the-art approach 30% in both Precision
and Recall.

###### Index Terms:

static analysis warnings, actual vulnerability, false positive, warning
context, representation learning

##  I introduction

In order to guarantee the quality of software, many techniques such as code
review, automatic static analysis, and testing, etc. have been applied during
the software development life cycle. Especially, static analysis [1, 2] plays
an important role in detecting vulnerabilities at the early phases. Without
executing programs, static analysis (SA) tools analyze the source code to
identify the violations of the pre-defined rules and recommendations. These
rules and recommendations are often defined by coding standards such as SEI
CERT Coding Rule [3] or MISRA [4].

However, SA tools often report a large number of warnings (a.k.a. alarms) [5].
In particular, a warning indicates the statement containing the potential
vulnerability, the vulnerability type, and often the additional meta-
information [6]. In practice, developers have to manually inspect all the
reported warnings and address them if necessary. However, due to the
conservative over-approximation of program behaviors/properties of SA, many
warnings are incorrectly reported by tools (i.e., false positive warnings).
Among the reported warnings, true positive warnings or true positives (TPs)
are actual vulnerabilities, while false positive warnings or false positives
(FPs) are the positions which indeed do not violate the checking rules.
Investigating FPs consumes time and effort but does not bring any benefit,
therefore, the high FP rate reduces the productivity of developers [7, 8] and
is bad for the usability of SA tools [7, 9]. In consequence, it is necessary
to reduce the number of FPs that developers need to verify.

In previous studies, sophisticated techniques such as model checking and
symbolic execution have been applied to eliminate FPs [10, 11, 12]. Although
these approaches obtain high precision in identifying FPs, they generally face
non-scalability and time consuming issues because of their state space
problems [13].

With the growth in size and complexity of the source code, recently machine
learning (ML) techniques are leveraged to build models for discovering
patterns associated with TPs/FPs. In general, detecting TPs/FPs among SA
warnings can be considered as a standard binary classification problem. To
build a SA waning classification model, there are two main methods for
extracting features from the warnings, one uses a pre-defined set of features
and the other encodes the features by ML models. Then, from the extracted
features, the ML models need to predict whether the SA warning is a TP or FP.

In several existing studies [14, 6, 15], the fixed sets of the hand-engineered
features based on static code metrics and warning information have been
derived from the source code and then fed to classifiers. The effectiveness of
these approaches depends on the quality of the selected set of features.
Moreover, the features in these approaches are manually defined by experts for
certain kinds of warnings, therefore it is difficult to extend for handling
different ones.

Meanwhile, instead of using a fixed pre-defined set of features, Lee et al.
[16] proposed a model to learn the lexical patterns of the statements around
the warnings at the source code token level. They used Word2vec [17] to embed
code tokens into the vector form and then trained a Convolutional Neural
Networks (CNN) classifier. Specially, by investigating their projects, they
define the number of surrounding statements that need to be extracted to
reflect the contexts of the warnings. These numbers of extracted statements
are different for their six proposed checkers. However, it is challenging to
apply this approach for different projects and/or different kinds of warnings
since it requires expert knowledge and carefully manual investigation to
decide how many statements in each function are enough to capture the contexts
of the checking warnings. Moreover, not every statement surrounding a warning
is all related to its violation and equally important for TP/FP detection.
Also, warning unrelated statements would negatively affect the performance of
the ML models.

In this paper, we propose DeFP, a novel method to prioritize SA warnings.
Instead of classifying SA warnings, DeFP predicts their likelihoods to be TPs
and then ranks them with the top entries are more likely to be TPs (actual
vulnerabilities), and the last entries are more likely to be FPs. Indeed,
ranking SA warnings rather than classifying them gives us three following
benefits.

First, for developers, DeFP helps shorten the cycle of developing and
releasing products, especially for critical systems. The reason is that
critical systems are highly required to be safe, secure, and reliable.
Therefore, any potential vulnerabilities (warnings) are all needed to be
addressed. In other words, if a warning is eliminated due to being falsely
classified as an FP, it would cause the system to be dangerously explored in
the future. Instead of directly eliminating any warnings classified as FP,
DeFP ranks SA warnings according to their vulnerable likelihoods. With the
pressure to release the high-quality software on time, focusing on the top-
ranked warnings first helps developers to find more actual vulnerabilities in
a fixed duration. Then, they can spend time and effort on the low-ranked
warnings later.

Second, for SA tool builders, DeFP suggests case studies that they can examine
to improve the quality of their tools. Indeed, to better serve the market, SA
tool builders need to frequently improve their tools by not only increasing
the TP rate, but also decreasing the FP rate. Among a huge number of reported
warnings in various projects, DeFP suggests an effective order for
investigation. Specially, SA tool builders can directly concentrate on
addressing warnings which are more likely to be FPs, i.e., warnings at the
last of the resulting lists, to find the patterns which tends to be
incorrectly reported.

Third, for researchers, to build the datasets of real-world warnings, DeFP
helps the data collection process be more efficient. In practice, in this
field, it still lacks public datasets for evaluating approaches and
researchers often have to manually investigate to label warnings. This process
is extremely time-consuming. From the ranked lists of DeFP, researchers can
effectively collect warnings by selectively labeling top-ranked and last-
ranked warnings.

Our key idea is based on the intuition that warnings which have similar
contexts tend to have similar labels (TP or FP). For each warning, DeFP
exploits both syntax and semantics from the context of the warning and then
determines its likelihood to be TP. In order to capture the context of a
warning, DeFP extracts all of the statements in the program which impact and
are impacted by the statement containing the warning (the reported statement).
After that, to better represent the general patterns of warnings, identifiers
and literals, which are specific for functions/files/projects and could make
the ML models be biased by the training source code, are replaced by abstract
names. Next, the reported statements and their contexts are vectorized and
used to train neural network models. One of the models represents the
information specifically included in the reported statements, while the other
extracts critical information in the warning contexts. Then, the high-level
features encoded from these models are utilized to estimate the likelihoods to
be TPs of the corresponding warnings. Lastly, SA warnings are ranked according
to their predicted scores.

To the best of our knowledge, it still lacks a public real-world dataset for
widely evaluating approaches post-handling SA warnings. In existing studies
[14, 6, 15], ML models are often trained and tested on synthetic datasets,
such as Juliet [18] and SARD [19]. However, Chakraborty et al. [20] has
demonstrated that these datasets are quite simple for estimating the
performance of ML models on real-world data. Thus, to address the limitation
of data shortage, we propose a dataset containing 6,620 warnings of 10 real-
world projects.

Our experiments show that about 60% of actual vulnerabilities are ranked by
DeFP in Top-20% of warnings. Moreover, +90% of actual vulnerabilities can be
found by investigating only 60% of the total warnings. Meanwhile, by using the
state-of-the-art approach [16], with the same number of examined warnings,
developers can find only 46% and 82% TPs.

In summary, our contributions in this paper are:

  * •

A novel approach to rank SA warnings, which does not require feature
engineering and could be flexible to extend for different kinds of warnings.

  * •

A public dataset of 6,620 warnings collected from 10 real-world projects,
which can be used as a benchmark for evaluating related work.

  * •

An extensive experimental evaluation showing the performance of DeFP over the
state-of-the-art approach [16].

##  II Motivating example and guiding principles

###  II-A Motivating Example

Figure 1: An FP reported by Flawfinder at line $24$

Fig. 1 shows a simplified version of function aoc_s_event in project
Asterisk111https://github.com/asterisk/asterisk, the complete version can be
found on our website [21]. In this example, a warning related to Buffer
Overflow (BO) is reported at line $24$ by Flawfinder [22], a static analysis
tool. The reason is that strcat appends the string pointed by rate_str to the
end of prefix. This could cause the size of the resulting string stored in
prefix to be greater than $32$ which is the prefix’s size of allocated memory
(line 15).

However, via inter-procedure analysis, we can conclude that this warning is an
FP. Specially, at line $19$ in Fig. 1, prefix is set to be R(i) where i is the
index of the loop, and the maximum size of prefix after this statement is
$13$, in the case of $i=INT\\_MAX$ (i.e., 2,147,483,647). At line $23$, prefix
is appended a character (i.e., "/"), and then at line $24$ it is appended a
string pointed by rate_str, which has length $8$ (rate_str = "Duration", line
$20$ and line $6$). As a result, after line $24$, the maximum length of prefix
is $22$, which is still much smaller than $32$. Therefore, in order to
determine whether a warning is a TP or FP, it need to conduct not only intra-
procedure analysis but also inter-procedure analysis. In other words, simply
approximating the context of a warning by its surrounding statements or by its
containing function could be ineffective.

###  II-B Guiding Principles

In order to determine whether a SA warning is a TP or FP, analyzing only the
reported statement is not enough. It requires investigating the context of the
warning as well. For example, to conclude the warning at line $24$ in Fig. 1
is an FP, not only that statement but also the related statements such as
lines 18, 19, 20, etc. need to be examined. Therefore, to build an ML model
which can effectively predict the likelihoods to be TP/FP of the warnings, for
each warning, we need to extract its appropriate context in the program. From
the extracted contexts, the model can capture patterns associated with the
warnings. Also, statements unrelated to the warning, which might cause noises
and negatively affect the performance of the model, should be excluded from
the warning context. In this paper, we propose the following principles for
the problem of ranking SA warnings by representation learning.

P1. The warning contexts can be semantically captured by the statements in the
program which can impact and be impacted by the reported statements. In
practice, to determine whether a warning indicates an actual violation of a
specific vulnerability type or not, it is necessary to investigate all of the
feasible execution paths containing the reported statement. In other words, we
need to examine the control flows and data flows of the program which contain
the reported statement. Besides, functions in a program do not work
independently, they often execute with the invocations of the others. Thus,
inter-procedural analysis is essential to effectively capture the contexts of
the warnings. Simply, all the statements in the functions/programs can be
considered as the warning contexts, however, this method can cause
unnecessarily large contexts and negatively affect the ML model’s predictive
performance. Also, not all of the statements in the functions/programs are
actually relevant to the warnings. For example, the statement at line 17 in
Fig. 1 does not affect the decision about TP or FP of the warning at line 24.
Including such irrelevant statements may cause the ML model to falsely learn
the actual patterns associated with TPs/FPs. Therefore, inter-procedural
slicing techniques [23] can be applied to effectively extract the warning
contexts by statements semantically related to the reported statements and
eliminate the irrelevant statements.

P2. The reported statements should be highlighted compared to the other
statements in the warning contexts. Intuitively, not all the statements in the
program slices are equally important regarding the considering warnings. The
reported statements are where the vulnerabilities are potentially explored,
thus, they should be highlighted compared to the other statements in their
contexts. For example in Fig. 1, the statements at lines $19$, $23$, and $24$,
which all modify the value of prefix, are all necessary for investigating the
warning. However, according to the report of Flawfinder, the BO vulnerability
is potentially explored at line $24$, which appends prefix by an unknown size
string, i.e., a string is returned by another function. Intuitively, the
statement at line $24$ should be emphasized compared to statements at lines
$19$ and $23$. Moreover, in practice, a program slice could contain several
warnings, therefore to distinguish warnings, not only the program slices (i.e.
context of the warnings) but also the reported statements need to be
featurized.

P3. Identifiers should be abstracted. The reason is that identifiers such as
variables, function names, constants are project-specific (or even file-
specific/function-specific) and considerably vary regarding developers’ coding
style. By learning such specific information, the ML model could not capture
general patterns of the warnings. Also, this could make the models simply
learn the connections between specific identifiers and warning labels (TP/FP).
Consequently, the models accurately predict the warnings of several training
programs but their performance might decrease dramatically on the different
programs. Therefore, to build a general ML model which can work well and
stably across programs, the identifiers should be abstracted into symbolic
names, for example, VAR1, FUNC1, etc. Moreover, without abstraction, the
number of identifiers could be virtually infinite, so ML model could have to
deal with the vocabulary explosion problem.

##  III Static analysis warning ranking with representation learning

Figure 2: Our proposed approach for ranking SA warnings

Fig. 2 illustrates our SA warning ranking approach. Particularly, from the
source code and the set of warnings of the analyzed program, we extract the
reported statements and their program slices associated with warnings. For
each warning, the reported statement and the corresponding program slice are
converted into vectors and then fed to the BiLSTM models to predict its
likelihood to be TP. After that, all of the warnings of the program are ranked
according to their predicted scores.

###  III-A Program Slice Extraction

In this work, to capture the context of each warning, we extract all the
statements in the program which impact and are impacted by the corresponding
reported statement. Specially, starting from the reported statement, we employ
Joern [24] to conduct both backward and forward inter-procedural slicing in
the program. For instance, the context of the warning at line 24 in Fig. 1 is
captured by the program slice shown in Fig. 3. Therefore, by this approach,
not only a large number of irrelevant statements in the program are removed,
but also the warning contexts are precisely captured via control/data
dependencies relationship throughout the program.

###  III-B Input Vectorization

Program slices and reported statements are lexical source code tokens.
Meanwhile, neural network models require their inputs to be formalized as
numeric vectors. Therefore, we need to represent the model input data by a
suitable data structure. In this section, we show three steps to represent the
program slice and the reported statement of each warning: identifier
abstraction, tokenization, and vectorization.

####  III-B1 Identifier Abstraction

In general, programs often contain a huge number of identifiers, also their
naming conventions and styles are diverse. It could cause difficulties for the
ML models to capture the general patterns of the warnings [20]. Besides, ML
models would simply learn characteristics of identifiers in certain projects,
as well as simply map specific identifiers with corresponding warning labels.
In order to avoid this problem, DeFP abstracts all the identifiers before
feeding them to the models. In particular, variables, function names, and
constants in the extracted program slices are replaced by common symbolic
names. For example, function name aoc_s_event is replaced by FUNC1, the array
prefix is replaced by VAR5, and its allocated size $32$ is replaced by
NUMBER_LIT. The details of our rules for abstracting identifiers can be found
on our website [21].

Figure 3: The program slice of the warning at line $24$ in Fig. 1

####  III-B2 Tokenization

To represent in numeric vectors for feeding to the neural network models, both
the extracted program slices and the reported statements are tokenized into
sequences of tokens. In this work, we use lexical analysis to break down each
code statement into code tokens, which including identifiers, keywords,
punctuation marks, and operators. For instance, the statement at line $24$ in
Fig. 1, strcat(prefix, rate_str);, is abstracted as strcat(VAR8, VAR11); and
then it is tokenized into seven separated tokens: “strcat”, “(”, “VAR8”, “,”,
“VAR11”, “)” and “;”.

####  III-B3 Padding and Truncation

In practice, the number of sequence tokens in different slices could be
significantly different. For example, in our experiment, the sequence lengths
can vary from 5 to 9,566 code tokens. Therefore, to ensure that all the
sequences have the same length, $L$, before inputting to the neural network,
we use padding and truncation techniques. To achieve the best performance, the
fixed length $L$ is carefully selected via multiple experiments.

Particularly, for sequences having lengths smaller than $L$, we add one or
more special tokens ($<$pad$>$) at the end of these sequences. For the
sequences whose lengths are greater than $L$, we truncate them to fit the
fixed length.

In practice, the positions of the reported statements in their corresponding
slices significantly vary. They can appear at the beginning of the slices or
at the end of the slices. Thus, truncating from either the beginning or the
end of the sequences could lead to the cases that the statements containing
warnings are missed in the truncated sequences. Therefore, in order to
guarantee that the truncated sequences always contain the reported statements,
we take these statements as the center for truncating. Specially, from the
reported statements, we extend to both sides of the sequences. until reach the
fixed length $L$. Importantly, to capture the correct semantics of each code
statement, we ensure that a statement will be fully included in the truncated
sequences. It means that when only some tokens of a statement are included in
the truncated sequences and the remaining is left due to the length
limitation, we will replace all of the tokens of that statement in the
truncated sequence by ($<$pad$>$) token.

Figure 4: The proposed LSTM-based representation learning model for ranking SA
warnings

####  III-B4 Vectorization

In this step, the token sequences are embedded into numeric fixed-length
vectors. In practice, besides the structural information of the sequences
which need to be encoded, relationships between the tokens are also important.
The reason is that code tokens have to appear together in a certain order to
make the program grammatically and syntactically correct [25]. For this
purpose, in the vectorization step, we use Word2vec model [17] with Continuous
Bag-of-Words (CBOW) architecture.

###  III-C Representation Learning and Warning Ranking

The DeFP’s model architecture is shown in Fig. 4. Especially, to effectively
learn contextual information that is crucial for revealing TP/FP code
patterns, two Bidirectional Long Short-Term Memory networks (BiLSTM) [26] are
employed to train on the embedding vectors of the program slices and the
reported statements. Afterwards, DeFP extracts the meaningful characteristics
of related warnings by concatenating the outputs from these BiLSTM models and
feed into Fully Connected layers. In particular, we consider the final layer’s
output of the model as the likelihood to be TP/FP of each input warning. All
these scores are finally gathered by DeFP and ranked accordingly.

####  III-C1 Representation Learning

In this work, the warning contexts (i.e. program slices) and the reported
statements are encoded by two BiLSTM networks. In the case of program slices
containing multiple statements distributed across functions, LSTM architecture
might essentially apprehend the relationships between code tokens.
Additionally, by utilizing a gated mechanism, LSTM can handle long-term
dependencies and also focus on the most significant parts of the sequences.

However, the information in LSTM is expressed one way through continuous time
steps in sequential order. Meanwhile, the occurrence of a code token is
usually related to either the previous or the subsequent tokens, or even both.
Thus, additionally applying the bidirectional implementation of LSTM can
assist the model to build dependencies in both forward and backward
directions, which efficiently captures the general pattern of warnings.

DeFP also employs the Global Max Pooling (GMP) layer to accumulate the output
of each BiLSTM network. Especially, GMP layer computes maximum values over
LSTM’s time steps, which help to reduce output dimension and only keep the
most important elements from LSTM cells. As a result, DeFP has two GMP layers
following each BiLSTM network, and then they are concatenated into a unified
one to represent a whole feature map (Fig. 4).

####  III-C2 Warning Ranking

After obtaining the learned representations of the warnings, DeFP
distinguishes their patterns by feeding them into three Fully Connected
(Dense) layers behind. Particularly, the final layer has only two hidden units
activated by the Softmax function, which produces two scores whose total is
1.0. These two values correspond to the likelihoods of each warning to be TP
and FP, respectively.

In the training phase, DeFP’s neural network enhances its predictions by
finding the best hidden weights through estimating its errors. In other words,
an objective function, cross-entropy, is applied to calculate the model’s loss
and update the weights towards minimizing this error value. Consequently, in
the case of a TP warning, the model tends to converge its TP score towards 1.0
and its FP score closes to 0.0, and vice versa for an FP warning. In the
ranking phase, by inputting a list of warnings, DeFP directly calculates their
TP scores and sorts them in descending order.

##  IV Empirical Methodology

In order to evaluate DeFP, we seek to answer the following research questions:

  * •

RQ1: How accurate is DeFP in ranking SA warnings? and how is it compared to
the state-of-the-art approach [16]?

  * •

RQ2: How does the extracted warning context affect DeFP’s performance? (P1)

  * •

RQ3: How does the highlighting reported statement impact the performance of
DeFP? (P2)

  * •

RQ4: How does the identifier abstraction component impact the performance of
DeFP? (P3)

###  IV-A Dataset

In order to train and evaluate an ML model ranking SA warnings, we need a set
of warnings labeled to be TPs or FPs. Currently, most of the approaches are
trained and evaluated by synthetic datasets such as Juliet [18] and SARD [19].
However, they only contain simple examples which are artificially created from
known vulnerable patterns. Thus, the patterns which the ML models capture from
these datasets could not reflect the real-world scenarios [20]. To evaluate
our solution and the others on real-world data, we construct a dataset
containing 6,620 warnings in 10 open-source projects [27, 28]. Table I shows
the overview of our dataset.

In these 10 real-world projects, functions are previously manually labeled as
vulnerable and non-vulnerable [27, 28]. Then, our dataset is constructed by
the following steps:

  1. 1.

Collecting warnings: We pass the studied projects through three open-source SA
tools, Flawfinder [22], CppCheck [29], RATS [30] to collect a set of warnings.
In practice, this set contains warnings in multiple kinds of vulnerabilities.
However, we only collect the warnings related to Buffer Overflow (BO) and Null
Pointer Dereference (NPD) since for the other kinds, the number of reported
warnings are too small for training and evaluating an ML model.

  2. 2.

Labeling warnings in the non-vulnerable functions: Since, these functions are
already marked as clean regarding BO and/or NPD, thus all the corresponding
warnings in these functions are annotated as FPs.

  3. 3.

Labeling warnings in the vulnerable functions: Although these functions are
marked containing BO and/or NPD vulnerabilities, we do not know exactly how
many vulnerabilities each function contains and the positions of the
vulnerabilities in the source code. Therefore, for each of the warnings in
these functions, we manually investigate to label whether it is a TP or FP.

TABLE I: Overview of DeFP’s dataset

No. | Project | Buffer Overflow | Null Pointer
---|---|---|---
Dereference
#W | #TP | #FP | #W | #TP | #FP
1 | Asterisk | 2049 | 63 | 1986 | 133 | 0 | 133
2 | FFmpeg | 1139 | 387 | 752 | 105 | 37 | 68
3 | Qemu | 882 | 396 | 486 | 72 | 39 | 33
4 | OpenSSL | 595 | 53 | 542 | 9 | 2 | 7
5 | Xen | 388 | 15 | 373 | 23 | 6 | 17
6 | VLC | 288 | 20 | 268 | 16 | 2 | 14
7 | Httpd | 250 | 45 | 205 | 17 | 0 | 17
8 | Pidgin | 250 | 13 | 237 | 242 | 0 | 242
9 | LibPNG | 83 | 9 | 74 | 2 | 0 | 2
10 | LibTIFF | 74 | 9 | 65 | 3 | 3 | 0
# | Total | 5998 | 1010 | 4988 | 622 | 89 | 533

  * •

#W, #TP and #FP are total warnings, true positives and false positives.

###  IV-B Evaluation Setup, Procedure, and Metrics

####  IV-B1 Experimental Setup

We implemented neural network models using Keras together with TensorFlow
backend (version 2.5.0). The tokenizer was built upon NLTK library (version
3.6.2) and Word2vec embedding model was provided by the gensim package
(version 3.6.0). All experiments were computed by a server running Ubuntu
18.04 with an NVIDIA Tesla P100 GPU.

We adopt cross-validation to train several neural networks and select the best
parameter values corresponding to the effectiveness of predicting likelihoods
to be TP warnings in the proposed dataset. Specially, for DeFP, embedding size
are set to 96, the maximum length of each slice and reported statement is
fixed to 600 and 40, respectively, and they are learned by two BiLSTM networks
which each has 256 hidden nodes. During the training phase, the dropout, batch
size, and number of epochs is set to 0.1, 64 and 60, respectively. Also, the
minibatch stochastic gradient descent ADAMAX optimizer is selected with the
learning rate of 0.002.

Besides, the data is sampled into stratified 5 folds, while 4 folds are picked
for training and 1 remaining fold for testing (ratio of 8:2). We then run 5
different experiments on 5 pairs of training and testing data and aggregate
average results for the final assessment of the corresponding experiment.

TABLE II: Performance of DeFP and CNN model proposed by Lee et al. [16] in
ranking SA warnings

WN | Project | Method | # TP warnings found in top-k% warnings
---|---|---|---
Top-5% | Top-10% | Top-20% | Top-50% | Top-60%
Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall
BO | Qemu | CNN | 71.11% | 8.09% | 53.33% | 12.13% | 46.86% | 20.72% | 44.32% | 49.25% | 43.02% | 57.57%
DeFP | 82.22% | 9.34% | 67.78% | 15.40% | 65.14% | 28.78% | 52.27% | 58.08% | 50.38% | 67.43%
FFmpeg | CNN | 30.91% | 4.40% | 31.30% | 9.30% | 33.24% | 19.64% | 32.46% | 47.80% | 33.04% | 58.39%
DeFP | 67.27% | 9.56% | 61.74% | 18.34% | 52.43% | 31.00% | 38.95% | 57.37% | 37.72% | 66.66%
Asterisk | CNN | 11.00% | 17.56% | 8.78% | 28.59% | 7.56% | 49.36% | 4.49% | 72.95% | 3.82% | 74.49%
DeFP | 34.00% | 53.97% | 18.54% | 60.26% | 10.73% | 70.00% | 5.18% | 84.10% | 4.56% | 88.97%
COMBINED | CNN | 43.00% | 12.77% | 39.67% | 23.56% | 34.25% | 40.69% | 25.40% | 75.45% | 23.46% | 83.56%
DeFP | 66.00% | 19.60% | 56.00% | 33.27% | 43.92% | 52.18% | 27.50% | 81.68% | 24.82% | 88.42%
NPD | COMBINED | CNN | 63.33% | 21.37% | 43.33% | 29.15% | 38.40% | 53.99% | 21.29% | 74.25% | 19.62% | 82.09%
DeFP | 80.00% | 26.93% | 65.00% | 43.66% | 47.20% | 66.14% | 25.81% | 89.74% | 22.58% | 94.25%

####  IV-B2 Empirical Procedure

RQ1. We compare the performance of DeFP and the CNN model proposed by Lee et
al. [16] for ranking warnings in the proposed dataset.

RQ2. We study the impact of the warning contexts on the performance of DeFP.
Specially, we compare the performance of DeFP in four scenarios of the warning
contexts: (1) the raw code of the program, (2) the program slices on control
dependencies, (3) the program slices on data dependencies, and (4) the program
slices on both control and data dependencies.

RQ3. We study the impact of highlighting the reported statements on DeFP’s
performance. We compare the ranking results of DeFP in two scenarios when the
reported statements are and are not encoded for training the BiLSTM model.

RQ4. We study the impact of the identifier abstraction component by comparing
the performance of DeFP when the inputs are embedded with and without this
component.

For evaluation, we have two experimental settings as widely adopted in related
studies [31, 28, 27]: within-project setting and combined-project setting.
First, in within-project setting, warnings from the same project are split
into training and testing sets. Second, in combined-project setting, we
collect the warnings from all 10 projects and then split them into training
and testing sets. In practice, in several projects, the number of warnings is
quite small for training and testing an ML model, and it could cause
overfitting or underfitting problems. Thus, we only select three projects
which have the largest number of BO warnings for evaluating RQ1 in the
corresponding vulnerability type in the within-project setting. RQ1 in the NPD
vulnerability and the other research questions are only evaluated in the
combined-project setting.

####  IV-B3 Evaluation Metrics

In order to evaluate DeFP and compare its performance with the state-of-the-
art approach, we applied Top-$k$% Precision (P@K) and Top-$k$% Recall (R@K).
These two metrics are widely used in related studies [28, 32], especially when
the dataset is severely imbalanced. In this paper, P@K denotes the proportion
of actual TP warnings in the Top-$k$% of warnings ranked by the model, and R@K
refers to the proportion of correctly predicted TP warnings in Top-$k$% among
the total actual TPs warnings. In particular, P@K and R@K are calculated using
the following formulas, where $\\{Actual\>TPs\\}$ is the set of actual TP
warnings, $\\{Predicted\>TPs\\}@K$ is the list of Top-$k$% of warnings ranked
first by the model.

$P@K=\frac{|\\{Actual\>TPs\\}\cap\\{Predicted\>TPs\\}@K|}{|\\{Predicted\>TPs\\}@K|}$

$R@K=\frac{|\\{Actual\>TPs\\}\cap\\{Predicted\>TPs\\}@K|}{|\\{Actual\>TPs\\}|}$

##  V Experimental results

###  V-A Performance Comparison (RQ1)

Table II shows the performance of DeFP and the CNN model proposed by Lee et
al. [16] in Top-5%–Top-60% warnings of the ranked lists. Overall, DeFP
improves their model by nearly 30% in Precision and Recall for both BO and NPD
warnings. For example, in FFmpeg, Qemu, and Asterisk with the Top-20% of
warnings returned by DeFP, developers can find 23/79, 24/77, and 9/13 actual
vulnerabilities. Meanwhile, by using the CNN model [16], the corresponding
figures are only 16/79, 15/77, and 6/13, respectively. When the warnings of
all the projects are combined, by investing 20% of the warnings in the top of
the ranked list, 105/202 actual BO vulnerabilities and 12/18 actual NPD
vulnerabilities can be found by DeFP, while these figures for the CNN model
are only 82/202 and 10/18, respectively. Interestingly, with the results of
DeFP, developers can find +90% of actual vulnerabilities by investigating only
60% of the total warnings, which is 8% better than the CNN model.

Indeed, DeFP obtains better results because it concentrates on statements
which semantically describe the contexts of the warnings and DeFP is not
negatively affected by the statements which are unrelated to the warnings. For
example, the warning in Fig. 1, DeFP captures its context by the statements
which impact and are impacted by the reported statement at line 24 as shown in
Fig 3. These statements are essential for semantically capturing the context
of the warning because they show when and how the value of prefix, which is
reported to potentially overflow, is changed. Additionally, unlike the CNN
model [16], DeFP ignores statements at lines 17 and 26, which do not play any
role in reflecting the violation of the reported statement, although they are
near it. Therefore, they could cause noises if they are encoded as the context
of the warning.

In addition, by inter-procedural analysis, DeFP does not miss important
information to validate the violation of the reported statements. Specially,
there are statements, which are essential for validating the warnings, could
be in multiple functions. For example in Fig 1, the statement at line 6
specifying the concatenated string to prefix is extremely important to
determine whether the BO vulnerability could occur at line 24 or not. However,
this statement is not in the same function with the reported statement, yet in
another function aoc_rate_type_str. This statement is captured by DeFP, but it
will be missed if only intra-procedural analysis is considered.

Interestingly, among the studied projects, DeFP achieved the highest results
in Qemu and the lowest results in Asterisk. Specially, for Top-20% warnings of
Qemu, DeFP obtained 65.14% in Precision, whereas this figure of Asterisk is
only 10.73%. The reason is that the models are impacted by the imbalance of
the dataset. For instance, the numbers of TPs and FPs in Qemu are quite
balanced, while they are greatly imbalanced in Asterisk. Moreover, Asterisk
only contains 63 TPs, which is extremely small compared to its 1986 FPs.

###  V-B Impact of the Warning Context (RQ2)

Figure 5: Impact of the extracted warning contexts on DeFP’s performance. RAW,
CD, DD, and CD && DD denote the warning contexts which are captured by raw
source code, program slices on control dependencies, program slices on data
dependencies, and program slices on both control and data dependencies,
respectively.

Fig. 5 shows the performance of DeFP when the contexts of the warnings are
captured by different kinds of dependencies. DeFP obtains the best performance
when the warning contexts are captured by both control and data dependencies
on the PDG.  The reason is that, by using slicing techniques on both of these
dependencies, unrelated statements are removed from the warnings’ contexts and
only related statements are encoded and fed to the BiLSTM models. Therefore,
the models can better capture the patterns of warnings without being affected
by noises caused by the statements which are semantically unrelated to the
warnings. In particular, when program slices are conducted on both control and
data dependencies, the performance of DeFP in two studied vulnerability types
is 16% and 19% better than when the warning contexts captured by the raw code
of the containing functions. Interestingly, by slicing on both control and
data dependencies, the performance of DeFP is significantly improved in the
warnings which are ranked at the top of the lists. Specially, compare to the
raw code, for the Top-1%–Top-50% of the ranked warnings, DeFP’s performance
with this kind of program slices is improved 42% for NPD warnings and 34% for
BO warnings. In other words, among Top-20% of warnings (243 warnings for BO
and 25 warnings for NPD) in the resulting lists, DeFP correctly ranks 105/202
and 12/18 actual BO and NPD vulnerabilities. Meanwhile, when the warning
context is captured by raw code of the whole functions, these figures are only
79/202 and 9/18, respectively.

Importantly, for both BO and NPD vulnerabilities, program slices on only data
dependencies capture the warning contexts better than the raw code of
functions, however, the program slices on only control dependencies do worse.
The reason is that for these two kinds of vulnerabilities, the information
about data dependencies, which illustrates how the values of the variables are
propagated, is more informative for reasoning the warnings. For example in
Fig. 1, to determine whether the warning (line 24) is an FP, it is essential
to analyze the statements which have data-dependent on, such as lines 6, 15,
19, etc. Although the raw code may contain noises and unrelated statements, it
still contains all of this information. However, this important information is
missed in the program slices on control dependencies only. Therefore, the
performance of DeFP with raw code is worse than the program slices on data
dependencies, yet better than the program slices on control dependencies.
Specifically for Top-1%–Top-60% of warnings, compared to the raw code, DeFP’s
results with program slices on the data dependencies is 7% and 29% better for
BO and NPD. Also, compared to the program slices on the control dependencies,
these figures are 8% and 46%, respectively.

In practice, for different kinds of vulnerabilities, it could require control
or data dependencies or both of these two kinds of information for validating
the warnings. Therefore, to guarantee the best performance of DeFP, program
slices on both control and data dependencies should be leveraged to capture
the warning contexts.

###  V-C Impact of the Reported Statements (RQ3)

Figure 6: Impact of highlighting the reported statements on the performance of
DeFP

As seen in Fig 6, the performance of DeFP is slightly improved by 4% and 7%
for BO and NPD vulnerabilities when the reported statements are highlighted by
being encoded as an input of the BiLSTM model. For example, in Top-20% of
ranked warnings, by encoding the reported statements, developers can find 7
more actual BO vulnerabilities and 1 more actual NPD vulnerabilities. More
details about the performance of DeFP on P@K can be found on our website [21].

Indeed, highlighting the reported statements can help the neural network model
not only capture the patterns associated with the warning contexts, but also
explicitly emphasize the positions of warnings. Consequently, this would be
considerably helpful when several warnings having similar contexts but labeled
(TP and FP) differently. However, our dataset is built from the set functions
which are already classified as vulnerable or non-vulnerable. Thus, most of
the warnings in a vulnerable function tend to have the same TP labels. Also,
all of the warnings in a non-vulnerable function are labeled as FPs. That is
the reason why the DeFP’s performance is just slightly improved when the
reported statements are encoded as an input of the representation model.

###  V-D Impact of the Identifier Abstraction Component (RQ4)

Figure 7: Impact of identifier abstraction on DeFP’s performance

Fig. 7 shows that by abstracting identifiers, DeFP can capture the general
patterns associated with the warnings better. Specially, with identifier
abstraction, DeFP achieves about 7% and 12% better in two kinds of studied
vulnerabilities for Top-1%-Top-60%. For instance, in Top-20% of warnings, DeFP
can find 105 actual BO vulnerabilities and 12 actual NPD vulnerabilities,
which is about 52% and 66% of their total actual vulnerabilities. Meanwhile,
without identifier abstraction, these numbers are only 96 and 11
vulnerabilities, respectively. More details about the performance of DeFP on
P@K can be found on our website [21].

Moreover, identifier abstraction decreases Word2vec vocabulary size on BO and
NPD datasets from 37,170 to 512 and 4,094 to 259 tokens, respectively. This
helps the model deal with the vocabulary explosion problem, better generalize
rare identifiers, and avoid out-of-vocabulary. As well, Word2vec might also
beneficially reduce vector dimension to represent each code token, thus
improve memory usage and shorten the training/prediction time.

##  VI Threats to Validity

There are three main threats to validity in this paper, they are external
validity, internal validity, and construct validity, respectively, which are
illustrated as follows.

####  VI-1 External validity

Our dataset contains 10 open-source programs and warnings in only two
vulnerability types. Therefore, our results may not be general for all
software projects and other kinds of vulnerabilities. To reduce the threat, we
chose the programs which are widely used in the related work [27, 28] and the
top most popular vulnerabilities [33]. Also, we plan to collect more data for
the future work.

####  VI-2 Internal validity

For this paper, the internal validity mainly lies in the data used for
learning process. We manually labeled for the warnings based on the labels of
the functions, which are assigned by Zhou et al. [27] and Lin et al. [28]. The
threat may come from their incorrect labels at function level or our
misleading labels at warning level. To minimize this threat, we carefully
investigate to label the warnings.

####  VI-3 Construct validity

In this study, we adopt P@K and R@K for evaluating the performance of the
ranking models. However, with the problem of handling SA warnings, evaluation
in terms of other metrics may also require in practice. We will conduct
experiments using more evaluation measures in our future work.

##  VII Related Work

There are various approaches have been applied to detect source code
vulnerabilities in the early phases of the software development process.
Specially, using SA tools is an automatic and simple way to detect various
kinds of vulnerabilities without executing the programs [1, 2]. Baca et al.
[34] has demonstrated that SA tools is better than average developers in
detecting warnings, especially the security ones. However, the generated
warnings of SA tools often contain a high number of FP rate [7, 8]. Therefore,
developers still need to waste a lot of time and effort for investing such FP
warnings.

In order to improve the precision of SA tools, sophisticated program
verification techniques such as model checking, symbolic execution, or
deductive verification, etc., have been applied to reduce the number of FPs
[35, 36, 12, 10, 11]. For instance, Muske et al. [35, 36] uses model checking
to eliminate FPs. Specially, for each warning, they generate appropriate
assertions and then use model checking to verify whether those assertions
hold. Nguyen et al. [12] also generate proper annotations to describe the
verified properties of the warnings and then prove them by deductive
verification. These approaches can precisely discard a number of FPs. However,
not all of the generated warnings can be formally proved to be FPs or TPs by
these approaches. Additional, model checking approaches also suffer from the
enormous states space, which affect their performance and lead them to be non-
scalable.

In addition, several studies applied ML models to address SA warnings.
Specially, some research [14, 6, 15] propose sets of features about statistic
information of the warnings and then build a model which learn these features
to classify SA warnings. However, these features are manually defined based on
the dataset and the used SA tools. This process is error-prone even for
experts. Meanwhile, instead of using a fixed set of features, Lee et al. [16]
trained a CNN model classifying warnings based on features which are learned
from lexical patterns in source code. However, they manually defined different
contexts for different kinds of warnings based on their dataset. This limits
the adaptation of their approach for other kinds of vulnerabilities and
different dataset. In this paper, we propose an approach which can be fully
automated and easily to adapt for handling different warnings in different
projects. Specially, our models are trained to capture the patterns associated
with the warnings in their corresponding contexts, which are extracted by
inter-procedural slicing techniques.

Moreover, ML are also actively adopted in vulnerabilities detection.
Particularly, to leverage the syntax and semantics information presented in
the Abstract Syntax Tree, Dam et al. [31] proposed a deep learning tree based
model to predict whether a source file is clean or defective. Besides, there
are multiple studies also propose token-based models [37, 38] or graph-based
models [27, 20] to predict whether a function containing vulnerabilities.
However, these research focuses on detecting vulnerabilities at the file level
or function level, which are quite coarse-grained in granularity. Developers
still need to investigate the whole source code in the detected files or
functions to localize the vulnerabilities. In this research, our objective is
more fine-grained in granularity. We focus on ranking the warnings which are
reported by SA tools. With the resulting lists, developers can decide which
vulnerabilities should be investigated and fixed in a given a amount of time.

##  VIII Conclusion

SA tools have demonstrated their usefulness in detecting potential
vulnerabilities. However, these tools often report a large number of warnings
containing both TPs and FPs, which causes time-consuming for post-handling
warnings and affects the productivity of developers. In this paper, we
introduce DeFP, a novel method for ranking SA warnings. Based on the reported
statements and the corresponding warning contexts, we train two BiLSTM models
to capture the patterns associated with the TPs and FPs. After that, for a set
of new warnings, DeFP can predict the likelihood to be TP of each warning and
then rank them according to the predicted scores. By using DeFP, more actual
vulnerabilities can be found in a given time. In order to evaluate the
effectiveness of DeFP, we conducted experiments on 6,620 warnings in 10 real-
world projects. Our experimental results show that using DeFP, developers can
find +90% of actual vulnerabilities by investigating only 60% of the total
warnings.

## Acknowledgment

This work has been supported by VNU University of Engineering and Technology
under project number CN20.26.

In this work, Kien-Tuan Ngo was funded by Vingroup Joint Stock Company and
supported by the Domestic Master/ PhD Scholarship Programme of Vingroup
Innovation Foundation (VINIF), Vingroup Big Data Institute (VINBIGDATA), code
VINIF.2020.ThS.04.

## References

  * [1] N. Ayewah, W. Pugh, D. Hovemeyer, J. D. Morgenthaler, and J. Penix, “Using static analysis to find bugs,” _IEEE software_ , vol. 25, no. 5, pp. 22–29, 2008.
  * [2] N. Nagappan and T. Ball, “Static analysis tools as early indicators of pre-release defect density,” in _Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005._ IEEE, 2005, pp. 580–586.
  * [3] C. S. C. Group, “SEI CERT Coding Standards (wiki).” [Online]. Available: https://wiki.sei.cmu.edu/confluence/display/seccode
  * [4] M. The Motor Industry Software Reliability Association, _Guidelines for the Use of the C Language in Critical Systems_ , 03 2012.
  * [5] M. Beller, R. Bholanath, S. McIntosh, and A. Zaidman, “Analyzing the state of static analysis: A large-scale evaluation in open source software,” in _2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)_ , vol. 1. IEEE, 2016, pp. 470–481.
  * [6] L. Flynn, W. Snavely, D. Svoboda, N. VanHoudnos, R. Qin, J. Burns, D. Zubrow, R. Stoddard, and G. Marce-Santurio, “Prioritizing alerts from multiple static analysis tools, using classification models,” in _2018 IEEE/ACM 1st International Workshop on Software Qualities and their Dependencies (SQUADE)_. IEEE, 2018, pp. 13–20.
  * [7] B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge, “Why don’t software developers use static analysis tools to find bugs?” in _2013 35th International Conference on Software Engineering (ICSE)_. IEEE, 2013, pp. 672–681.
  * [8] U. Koc, S. Wei, J. S. Foster, M. Carpuat, and A. A. Porter, “An empirical assessment of machine learning approaches for triaging reports of a java static analysis tool,” in _2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)_. IEEE, 2019, pp. 288–299.
  * [9] J. Ruthruff, J. Penix, J. Morgenthaler, S. Elbaum, and G. Rothermel, “Predicting accurate and actionable static analysis warnings,” in _2008 ACM/IEEE 30th International Conference on Software Engineering_. IEEE, 2008, pp. 341–350.
  * [10] H. Post, C. Sinz, A. Kaiser, and T. Gorges, “Reducing false positives by combining abstract interpretation and bounded model checking,” in _2008 23rd IEEE/ACM International Conference on Automated Software Engineering_. IEEE, 2008, pp. 188–197.
  * [11] H. Li, T. Kim, M. Bat-Erdene, and H. Lee, “Software vulnerability detection using backward trace analysis and symbolic execution,” in _2013 International Conference on Availability, Reliability and Security_. IEEE, 2013, pp. 446–454.
  * [12] T. T. Nguyen, P. Maleehuan, T. Aoki, T. Tomita, and I. Yamada, “Reducing false positives of static analysis for sei cert c coding standard,” in _2019 IEEE/ACM Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop on Software Engineering Research and Industrial Practice (SER &IP)_. IEEE, 2019, pp. 41–48.
  * [13] T. Muske and A. Serebrenik, “Survey of approaches for handling static analysis alarms,” in _2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM)_. IEEE, 2016, pp. 157–166.
  * [14] U. Yüksel, H. Sözer, and M. Şensoy, “Trust-based fusion of classifiers for static code analysis,” in _17th International Conference on Information Fusion (FUSION)_. IEEE, 2014, pp. 1–6.
  * [15] M. Berman, S. Adams, T. Sherburne, C. Fleming, and P. Beling, “Active learning to improve static analysis,” in _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_. IEEE, 2019, pp. 1322–1327.
  * [16] S. Lee, S. Hong, J. Yi, T. Kim, C.-J. Kim, and S. Yoo, “Classifying false positive static checker alarms in continuous integration using convolutional neural networks,” in _2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)_. IEEE, 2019, pp. 391–401.
  * [17] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” in _ICLR_ , 2013.
  * [18] V. Okun, A. Delaitre, P. E. Black _et al._ , “Report on the static analysis tool exposition (sate) iv,” _NIST Special Publication_ , vol. 500, p. 297, 2013.
  * [19] N. I. of Standards and Technology, “Software assurance reference dataset.” [Online]. Available: https://samate.nist.gov/SRD/index.php
  * [20] S. Chakraborty, R. Krishna, Y. Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet,” _IEEE Transactions on Software Engineering_ , 2021.
  * [21] “DeFP.” [Online]. Available: https://tuanngokien.github.io/DeFP/
  * [22] “Flawfinder.” [Online]. Available: https://dwheeler.com/flawfinder/
  * [23] S. Horwitz, T. Reps, and D. Binkley, “Interprocedural slicing using dependence graphs,” _ACM Transactions on Programming Languages and Systems (TOPLAS)_ , vol. 12, no. 1, pp. 26–60, 1990.
  * [24] “Joern.” [Online]. Available: https://docs.joern.io/home
  * [25] G. Lin, J. Zhang, W. Luo, L. Pan, O. De Vel, P. Montague, and Y. Xiang, “Software vulnerability discovery via learning multi-domain knowledge bases,” _IEEE Transactions on Dependable and Secure Computing_ , 2019.
  * [26] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [27] Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu, “Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks,” _arXiv preprint arXiv:1909.03496_ , 2019.
  * [28] G. Lin, W. Xiao, J. Zhang, and Y. Xiang, “Deep learning-based vulnerable function detection: A benchmark,” in _International Conference on Information and Communications Security_. Springer, 2019, pp. 219–232.
  * [29] “CppCheck.” [Online]. Available: http://cppcheck.sourceforge.net
  * [30] “RATS - Rough Auditing Tool for Security.” [Online]. Available: https://github.com/andrew-d/rough-auditing-tool-for-security
  * [31] H. K. Dam, T. Pham, S. W. Ng, T. Tran, J. Grundy, A. Ghose, T. Kim, and C.-J. Kim, “Lessons learned from using a deep tree-based model for software defect prediction in practice,” in _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 2019, pp. 46–57.
  * [32] H. N. Nguyen, S. Teerakanok, A. Inomata, and T. Uehara, “The comparison of word embedding techniques in rnns for vulnerability detection.” in _International Conference on Information Systems Security and Privacy (ICISSP)_ , 2021, pp. 109–120.
  * [33] Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” _arXiv preprint arXiv:1801.01681_ , 2018.
  * [34] D. Baca, K. Petersen, B. Carlsson, and L. Lundberg, “Static code analysis to detect software security vulnerabilities-does experience matter?” in _2009 International Conference on Availability, Reliability and Security_. IEEE, 2009, pp. 804–810.
  * [35] T. Muske, A. Datar, M. Khanzode, and K. Madhukar, “Efficient elimination of false positives using bounded model checking,” in _ISSRE_ , vol. 15, 2013, pp. 2–5.
  * [36] T. Muske and U. P. Khedker, “Efficient elimination of false positives using static analysis,” in _2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE)_. IEEE, 2015, pp. 270–280.
  * [37] R. Russell, L. Kim, L. Hamilton, T. Lazovich, J. Harer, O. Ozdemir, P. Ellingwood, and M. McConley, “Automated vulnerability detection in source code using deep representation learning,” in _2018 17th IEEE international conference on machine learning and applications (ICMLA)_. IEEE, 2018, pp. 757–762.
  * [38] Z. Li, D. Zou, S. Xu, H. Jin, Y. Zhu, and Z. Chen, “Sysevr: A framework for using deep learning to detect software vulnerabilities,” _IEEE Transactions on Dependable and Secure Computing_ , 2021.


# Experimental observation of a moving mirror

Morgan H. Lynch<EMAIL_ADDRESS>Center for Theoretical Physics, Seoul
National University,
Seoul 08826, Korea  Michael R. R. Good<EMAIL_ADDRESS>Department of
Physics $\&$ Energetic Cosmos Lab, Nazarbayev University,
Astana 010000, Qazaqstan  Leung Center for Cosmology $\&$ Particle
Astrophysics, National Taiwan University,
Taipei 10617, Taiwan

###### Abstract

In this manuscript, we examine the radiative beta decay of free neutrons
measured by the RDK II collaboration. The infrared portion of the emitted
photon spectrum is shown to corroborate a thermal distribution consistent with
the dynamical Casimir effect (DCE). We find the system is thermalized at the
local Fulling-Davies-Unruh temperature predicted by the DCE. Substantiated by
an excellent chi-squared statistic, we present the evidence for the
observation of thermal radiation emitted during the beta decay process in
accordance with an accelerated moving mirror.

moving mirrors, beta decay, black hole evaporation, acceleration radiation

###### pacs:

41.60.-m (Radiation by moving charges), 04.70.Dy (Quantum aspects of black
holes)

##  I Introduction

Thermodynamic particle production, see e.g. [1, 2, 3], discovered in the
framework of quantum field theory in curved spacetime [4, 5, 6] is an emerging
experimental science in both analogue and fundamental systems. In analogue
settings, e.g. [7], experimental signatures are present for the major particle
production mechanisms, e.g. the Parker [8], Hawking [9], and Fulling-Davies-
Unruh effects [10], as well as the dynamic Casimir effect (DCE) [11]. The
surprising robustness of these phenomena even apply to seemingly disparate
systems such as rear end collisions in traffic jams [12]. Moreover, quantum
metrology is now on the cusp of bringing some of the most precise measurement
apparatus online utilizing these very tenets in their operation [13]. The
widespread ubiquity of thermodynamic quantum fluctuations, and their
applications, also spreads beyond the analogue regimes where their properties
are being fully unraveled in a systematic fashion. Fundamental systems also
provide an avenue to explore such phenomena and thus solidify their
thermodynamic interpretation as an intrinsic property of spacetime [14, 15].

High energy channeling radiation experiments at CERN-NA63 [16] have indeed
already opened the door to explore the Unruh effect and establish it as a
fundamental property of spacetime [17]. There, the extremely large
accelerations produced by the radiation reaction, or recoil, have given us
confirmation that for accelerated observers, the quantum vacuum appears as an
effervescent thermal bath at the Fulling-Davies-Unruh (FDU) temperature [18,
19, 3]. It also appears that stimulated Hawking radiation from astrophysical
black holes may be on the cusp of current detector sensitivities [20].
Gravitational wave echoes could provide a unique window into this phenomenon
by stimulating the emission of Hawking radiation as well as probe the nature
of event horizons [21]. Finally, precision cosmology may even offer an avenue
to explore the Parker effect [22]. In fact, multiple avenues are already being
explored such as anisotropies in the cosmic microwave background [23].

One closely related fundamental phenomenon, which yields thermalized particle
production under acceleration, is known as the DCE [24, 25]. This phenomenon,
when viewed as an analogue of black hole evaporation, yields the now
celebrated “moving mirror model” [26, 27]. Here, a rapidly accelerating
perfect reflector emits particles which can be Planck distributed at the FDU
temperature [19]. Like the Unruh effect, which is characterized by
interactions with thermalized quantum fluctuations [28], the accelerating
boundary of the DCE brings those thermalized quantum fluctuations on shell and
emits them as radiation. Observational efforts are being deployed with the
goal of direct detection [29, 30] of relativistic moving mirror radiation and
creation [31], and they complement the small but growing accumulation of
observations of the dynamical Casimir effect (see references in [32]).
Importantly, international efforts like the Analog Black Hole Evaporation via
Lasers (AnaBHEL) experiment [33] corroborate the accelerating relativistic
moving mirror as a viable avenue to probe the spectral physics of quantum
vacuum radiation. Surprisingly, the extreme accelerations experienced by an
electron during the process of radiative neutron beta decay [34, 35] also
provides a robust system to examine the thermalized DCE, i.e. the moving
mirror model [36].

This leads to the notion that analogs of black hole evaporation may also, in
fact, be found by examining radioactive decays. In the sense that a black hole
will emit radiation and lower its mass, an unstable atom also decays via the
same heuristic process. Although the mechanism of decay is different, there is
now evidence that nuclear decay, in particular beta decay, can be viewed as an
analog of black hole evaporation. In fact, neutron decay under acceleration
has also been a method proposed to probe the closely related Unruh effect
[37]. In the black hole scenario, and under the moving mirror formalism, the
mirror boundary acts as an event horizon and the mirror’s acceleration
parameter plays the role of the black hole surface gravity (for instance,
there is an exact radiative and thermal correspondence between the
Schwarzschild [38], Reissner-Nordström [39], and Kerr [40] spacetimes).
Curiously, radiative beta decay provides us with a dual scenario to
simultaneously study the analog system of an evaporating black hole, and thus
the thermalized DCE/moving mirror, all under the platform of a standard and
well-observed radiative process in particle physics. Thereby demonstrating a
novel system to study aspects of thermalized fluctuations of the quantum
vacuum produced under the influence of external conditions.

Here we report the observation of the DCE, or moving mirror, in neutron beta
decay. We examine the photon spectra produced by the radiative decay mode of
the free neutron using a novel Bogoliubov technique for infrared radiation
[36]. We find the temperature of the radiation emitted in agreement with the
local FDU temperature predicted by the Bogoliubov technique and an excellent
chi-squared statistic matching the theory to the experimental data. Here we
present promising evidence for the thermal DCE produced by an ensemble of
electrons accelerated during the beta decay process. Throughout, we use
natural units, $\hbar=c=k_{B}=1$.

##  II thermalized photon spectra

In consideration of a moving mirror which reflects, or emits, a massless
scalar field $\Phi(z)$, we define the mirror by the Dirichlet boundary
condition $\square\Phi(z)=0$ on the mirror surface. This scalar field can be
expanded in modes, $\phi$, which satisfy the lightcone wave equation,
$-\partial_{u}\partial_{v}\phi=0$. Here we have defined the null coordinates,
$u=t-z$ and $v=t+z$. Our positive and negative frequency solutions are then
given by,

$\displaystyle\phi^{\pm}_{u}$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{4\pi\omega^{\prime}}}e^{\mp i\omega^{\prime}u}$
$\displaystyle\phi^{\pm}_{v}$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{4\pi\omega^{\prime}}}e^{\mp i\omega^{\prime}v}.$
(1)

Given a specific mirror trajectory, we will require the above modes to vanish
identically at the mirror surface along the trajectory. At early times, or the
in state, we then select our positive frequency mode function to be,

$\phi^{in}_{\omega^{\prime}}=\frac{1}{\sqrt{4\pi\omega^{\prime}}}\left[e^{-i\omega^{\prime}v}-e^{-\omega^{\prime}p(u)}\right].$
(2)

Here, the function, $p(u)$, is how we enforce the boundary condition with
$p(u)=t_{m}(u)+z_{m}(t_{m}(u))$ characterizing the trajectory of the mirror.
Similarly, we can define the function, $f(v)=t_{m}(v)-z_{m}(t_{m}(v))$, for
late times, or the out state. We will then have the positive frequency modes,

$\phi^{out}_{\omega}=\frac{1}{\sqrt{4\pi\omega}}\left[e^{-i\omega
f(v)}-e^{-i\omega u}\right].$ (3)

Finally, we can then examine the number of out state particles which are
created in the in state vacuum, i.e. particles created the moving mirror. We
shall employ the trajectory which models the electron trajectory during the
process of inner bremsstrahlung [36]; Given a final velocity
$|\beta_{\textrm{f}}|=s$, we consider the trajectory determined by proper
acceleration, $a(t)=\kappa\beta\gamma^{3}(1-\beta/s)^{2}$. Here $\kappa$ is
the acceleration scale. For this accelerated mirror trajectory, the Bogoliubov
transformation between the early and late times of the mirror modes yields our
overall spectrum [36],

$|\beta_{\omega\omega^{\prime}}|^{2}=\frac{2s^{2}\omega\omega^{\prime}}{\pi\kappa(\omega+\omega^{\prime})}\frac{a^{-2}+b^{-2}}{e^{2\pi(\omega+\omega^{\prime})/\kappa}-1}.$
(4)

Here $a=\omega(1+s)+\omega^{\prime}(1-s)$, and
$b=\omega(1-s)+\omega^{\prime}(1+s)$. The first thing to note here is that we
do indeed have a thermal flux of radiation from our mirror which is indicated
by the explicit Bose-Einstein distribution in the above expression. This can
be made particularly explicit by considering high final speeds, $s\approx 1$,
then using the high frequency, $\omega^{\prime}\gg\omega$ approximation [41].
To leading order, one gets

$|\beta_{\omega\omega^{\prime}}|^{2}=\frac{1}{2\pi\kappa\omega}\frac{1}{e^{2\pi\omega^{\prime}/\kappa}-1}$
(5)

High final speeds and the low frequency approximation,
$\omega^{\prime}\ll\omega$ switches the primes, which gives the widely agreed
on result for thermal radiation [42]

$|\beta_{\omega\omega^{\prime}}|^{2}=\frac{1}{2\pi\kappa\omega^{\prime}}\frac{1}{e^{2\pi\omega/\kappa}-1}.$
(6)

We have found that the temperature is set by the asymptotic scale of the local
acceleration. To see this, we recall the relationship between the local
acceleration, $g$, and proper acceleration, $a$, is given by $g=2ae^{\eta}$
[43], here $\eta=\tanh^{-1}\beta$ is the rapidity of the electron. In the
ultra-relativistic limit and late times, the time-dependent local acceleration
$g$ becomes the constant local acceleration scale $\kappa$, i.e. $g=\kappa$.

As such, we find the temperature is set by the asymptotic local acceleration
scale, $\kappa$, i.e. the local FDU temperature. Finally, via the moving
mirror analysis of radiative beta decay [36], this local acceleration is set
by the emitted photon frequency spread, $\kappa=48\alpha\Delta\omega$; here
$\alpha\approx 1/137$ is the fine structure constant.

The particle distribution created by the moving mirror can be obtained by
integrating over $\omega^{\prime}$ which is the mode frequency with a left-
moving, incoming plane wave piece; the ‘in’ state frequency,

$\frac{d\mathcal{N}}{d\omega}=\int_{0}^{\infty}|\beta_{\omega\omega^{\prime}}|^{2}d\omega^{\prime}.$
(7)

In the limit of large $\kappa$, this integral can be done analytically. The
result is particle spectrum of photons to leading order in $\kappa$. This
spectrum is independent of $\kappa$,

$\frac{d\mathcal{N}}{d\omega}=\frac{1}{\pi^{2}}\left[\frac{\eta_{s}}{s}-1\right]\frac{1}{\omega},$
(8)

where $\eta_{s}=\tanh^{-1}s$ is the final rapidity of the electron.

The application of the above spectra to measurements of radiative beta decay
[34, 35] would confirm the presence of the DCE produced by the accelerated
electrons during the decay process. This would provide a novel system to
examine thermalized particle production. Moreover, under the moving mirror
framework, radiative beta decay could also be viewed as an analogue of black
hole evaporation.

For Hawking radiation emitted by black holes, the entirety of the dynamics are
fixed by the mass of the black hole. Similarly, the thermal properties of
radiative beta decay are fixed by the energy content of the decay process. The
total available energy, $\mathcal{E}$, in radiative beta decay is given by the
difference in neutron, proton, and electron rest masses,
$\mathcal{E}=m_{n}-m_{p}-m_{e}=782.3$ KeV. Note, this ignores the neutrino
mass. This reservoir of energy sets both the photon energy spread,
$\Delta\omega=\mathcal{E}$, as well as the final electron kinetic energy and
thus velocity, $s=\left[1-(\mathcal{E}/m_{e}+1)^{-2}\right]^{1/2}=0.918$.
Then, taking into account the Doppler shift produced by the final electron
velocity, [5], we will have the following blue shifted local acceleration
which is fixed by the conservation of energy from the decay,

$\kappa=48\alpha\Delta\omega\left[\frac{1+s}{1-s}\right]^{1/2}.$ (9)

Using the above, for radiative beta decay we have a local FDU temperature of
$T_{FDU}=211.8$ KeV. Note, this temperature is also fixed and should present
itself in the radiative beta decays of the free neutron as a fundamental
property of its kinematics. What is surprising is the total energy budget of
the decay, which typically sets the kinetic energy ranges of the daughter
products, also determines the acceleration, and thus the temperature.
Consequently, we find a local FDU temperature which is determined by the
conservation of energy for the system.

Figure 1: The radiative beta decay photon spectra for both the APD (top) and
BGO (bottom) detectors of the RDK II collaboration [34]. The theoretical
predictions for the DCE-moving mirror models are also included. The mirror
approximation is the high acceleration limit, Eq. (8), and the full mirror
expression is given by the integral of our Bogoliubov coefficient, Eq. (4).
The chi-squared statistics for each theory are presented in Table 1.

##  III Analysis

The data sets are comprised of the photon spectra measured by the RDK II
Collaboration [35]. There they employed two detectors comprised of bismuth
germanium oxide (BGO) scintillators and an avalanche photo diode (APD). The
low energy spectrum, 0.3 keV to 20 keV, was measured by the APD and the high
energy spectrum, 10 keV to 1000 keV, was measured by the BGO. We perform
single parameter best fits to each spectrum in order to establish an overall
scale which reflects the detector or experimental live time. The acceleration,
temperature, Doppler shift, final velocity $s$, and the peak photon frequency
are all determined by the conservation of energy of the system.

Having established an overall scale via the best fit, we then compute the
associated reduced chi-squared per degree of freedom,
$\chi^{2}/\nu=\sum_{bins}(y_{theory}-y_{exp})^{2}/\sigma^{2}/\nu$, to
establish the closeness of fit of the theory to the data. Here, $\nu$ is the
number of degrees of freedom and $\sigma$ is the error of each data point.
With no other free parameters, we find an excellent fit of our thermalized-DCE
moving mirror formalism to the data sets. In particular, the infrared portion
of the spectrum is found to reflect a strong presence of thermality. The
resulting chi-squared statistics are presented in Table 1 below.

$\chi^{2}/\nu$
---
Detector |  Approximate |  Full
APD |  1.21 |  1.08
BGO |  35.8 |  3.60
Table 1: The reduced chi-squared per degree of freedom of our theoretical
power spectra. The 1 standard deviation threshold requires
$\chi^{2}/\nu<1+\sqrt{\nu/2}$. For the APD and BGO, the threshold values are
1.35 and 1.32 respectively.

##  IV Discussion

One particular interesting aspect of black hole evaporation is the fact that
the temperature, $T=\frac{1}{8\pi M}$ [2], is characterized entirely by the
black hole mass, i.e. the energy budget of the black hole. When one considers
the analog setting of black hole evaporation via the accelerated moving
mirror, we find the temperature is given by the acceleration. However, in the
case of inner bremsstrahlung/radiative beta decay, this
temperature/acceleration is also characterized by the energy budget of the
system, i.e. the mass difference $\Delta M=m_{n}-m_{p}-m_{e}$ between the
neutron and daughter products, $T=\frac{24\alpha}{\pi}\Delta M$ [36].
Moreover, just as the properties of black holes are completely determined by
their mass, the kinematics of inner bremsstrahlung are also determined by the
mass difference. These similarities serve to highlight the extent to which the
moving mirror interpretation of radiative beta decay is a true analog of black
hole evaporation.

The presence of the DCE also offers some intriguing insights into the nature
of the radiation emitted. As the DCE requires a moving conductor to yield
photon emission, we recall what actually defines a conductor, the presence of
free/mobile electrons. As such, should one take a conductor and continue to
cut it in half ad infinitum, one would soon be left with a single “unit” of
the conductor, i.e. an electron. Could we then interpret the electrons
produced, and subsequently accelerated, during radiative beta decay to be
“single particle mirrors”? Moreover, since ordinary bremsstrahlung has a
rigorous interpretation from the point of view of the Unruh effect [44, 45],
we must also ask if there is also a DCE interpretation for it as well? Perhaps
experiments like those of the RDK II collaboration have the potential to shed
more light on some of these aspects of the radiation emission.

With an ever expanding array of experimental systems which are capable of
exploring thermalized particle production, we have found an interesting
example in the form of radiative beta decay of the free neutron [35]. The
rather peculiar presence of the DCE in these particle decays also raises the
question of other systems which may provide arenas to explore the tenets of
quantum field theory in curved spacetime. As an example, there has been a
growing notion of entropic gravity [46], modified Newtonian dynamics [47], and
quantized inertia [48] as a candidate of dark matter arising from an Unruh-
type effect on cosmological scales. These, and other interesting systems,
including analogues [49], may find the presence of quantum vacuum fluctuations
as an underlying mechanism for unexplained or non-intuitive physics.

##  V conclusions

In this letter, we examined the photon spectrum emitted during the radiative
beta decay of the free neutron measured by the RDK II collaboration. Using the
moving mirror formalism, we found excellent agreement with the DCE thermalized
at the local FDU temperature sourced by the acceleration. A comparison of the
data and theory bears confirmation of thermalization, especially in the
infrared, backed by a chi-squared statistic to support the observation. We
found and confirm the rapid acceleration of the electron, produced by neutron
decay, provides a novel system for investigation into the character and
relationship between acceleration and temperature.

## Acknowledgments

Funding comes in part from the FY2021-SGP-1-STMM Faculty Development
Competitive Research Grant No. 021220FD3951 at Nazarbayev University.
Appreciation is given to the organizers, speakers, and participants of the
QFTCS Workshop: May 23-27, 2022. We thank Thomas R. Gentile and Jeffery S.
Nico at the Physics Laboratory at the National Institute of Standards and
Technology (Maryland, USA) for helpful advice concerning interpretation and
analysis of the RDK II Collaboration data.

## References

  * Parker [1968] L. Parker, Phys. Rev. Lett. 21, 562 (1968).
  * Hawking [1974] S. W. Hawking, Nature 248, 30 (1974).
  * Unruh [1976] W. G. Unruh, Phys. Rev. D 14, 870 (1976).
  * Parker and Toms [2009] L. E. Parker and D. Toms,  _Quantum Field Theory in Curved Spacetime: Quantized Field and Gravity_, Cambridge Monographs on Mathematical Physics (Cambridge University Press, 2009).
  * Birrell and Davies [1984] N. Birrell and P. Davies,  _Quantum Fields in Curved Space_, Cambridge Monographs on Mathematical Physics (Cambridge Univ. Press, Cambridge, UK, 1984).
  * Fabbri and Navarro-Salas [2005] A. Fabbri and J. Navarro-Salas,  _Modeling Black Hole Evaporation_ (Imperial College Press, 2005).
  * Unruh [1981] W. G. Unruh, Phys. Rev. Lett. 46, 1351 (1981).
  * Steinhauer  _et al._ [2022] J. Steinhauer, M. Abuzarli, T. Aladjidi, T. Bienaimé, C. Piekarski, W. Liu, E. Giacobino, A. Bramati, and Q. Glorieux, Nature Commun. 13, 2890 (2022), arXiv:2102.08279 [cond-mat.quant-gas] .
  * Weinfurtner  _et al._ [2011] S. Weinfurtner, E. W. Tedford, M. C. J. Penrice, W. G. Unruh, and G. A. Lawrence, Phys. Rev. Lett. 106, 021302 (2011), arXiv:1008.1911 [gr-qc] .
  * Hu  _et al._ [2019] J. Hu, L. Feng, Z. Zhang, and C. Chin, Nature Phys. 15, 785 (2019), arXiv:1807.07504 [physics.atom-ph] .
  * Vezzoli  _et al._ [2019] S. Vezzoli _et al._ , Commun. Phys. 2, 84 (2019).
  * de Souza and Matsas [2022] L. K. de Souza and G. E. Matsas, ”Am. J. Phys” 90 (2022), 10.1119/5.0091957.
  * Fuentes  _et al._ [2022] I. Fuentes _et al._ , US Patent App. 17/596, 523 (2022).
  * Bekenstein [1973] J. D. Bekenstein, Phys. Rev. D 7, 2333 (1973).
  * Jacobson [1995] T. Jacobson, Phys. Rev. Lett. 75, 1260 (1995), arXiv:gr-qc/9504004 .
  * Wistisen  _et al._ [2018] T. N. Wistisen, A. Di Piazza, H. V. Knudsen, and U. I. Uggerhøj, Nature Commun. 9, 795 (2018), arXiv:1704.01080 [hep-ex] .
  * Lynch  _et al._ [2021] M. H. Lynch, E. Cohen, Y. Hadad, and I. Kaminer, Phys. Rev. D 104, 025015 (2021), arXiv:1903.00043 [gr-qc] .
  * Fulling [1973] S. A. Fulling, Phys. Rev. D 7, 2850 (1973).
  * Davies [1975] P. C. W. Davies, J. Phys. A 8, 609 (1975).
  * Abedi  _et al._ [2021] J. Abedi, L. F. L. Micchi, and N. Afshordi, (2021), arXiv:2201.00047 [gr-qc] .
  * Abedi  _et al._ [2017] J. Abedi, H. Dykaar, and N. Afshordi, Phys. Rev. D 96, 082004 (2017), arXiv:1612.00266 [gr-qc] .
  * Parker [1976] L. Parker, Nature 261, 20 (1976).
  * Agullo and Parker [2011] I. Agullo and L. Parker, Phys. Rev. D 83, 063526 (2011), arXiv:1010.5766 [astro-ph.CO] .
  * Moore [1970] G. T. Moore, J. of Math. Phys. 11, 2679 (1970).
  * Wilson  _et al._ [2020] J. H. Wilson, F. Sorge, and S. A. Fulling, Phys. Rev. D 101, 065007 (2020), arXiv:1911.04492 [hep-th] .
  * Fulling and Davies [1976] S. A. Fulling and P. C. W. Davies, Proc. R. Soc. Lond. A 348, 393 (1976).
  * Davies and Fulling [1977] P. Davies and S. Fulling, Proc. R. Soc. Lond. A A356, 237 (1977).
  * Cozzella  _et al._ [2017] G. Cozzella, A. G. S. Landulfo, G. E. A. Matsas, and D. A. T. Vanzella, Phys. Rev. Lett. 118, 161102 (2017), arXiv:1701.03446 [gr-qc] .
  * Chen and Mourou [2017] P. Chen and G. Mourou, Phys. Rev. Lett. 118, 045001 (2017), arXiv:1512.04064 [gr-qc] .
  * Chen and Mourou [2020] P. Chen and G. Mourou, Phys. Plasmas 27, 123106 (2020), arXiv:2004.10615 [physics.plasm-ph] .
  * Brown and Louko [2015] E. Brown and J. Louko, J. High Energ. Phys. 61, 2015 (2015).
  * Dodonov [2020] V. Dodonov, MDPI Physics 2, 67 (2020).
  * Chen  _et al._ [2022] P. Chen _et al._ (AnaBHEL), (2022), arXiv:2205.12195 [gr-qc] .
  * Nico  _et al._ [2006] J. Nico _et al._ , Nature 444, 1059 (2006).
  * Bales  _et al._ [2016] M. J. Bales _et al._ (RDK II), Phys. Rev. Lett. 116, 242501 (2016), arXiv:1603.00243 [nucl-ex] .
  * Good and Davies [2022] M. R. R. Good and P. C. W. Davies, (2022), arXiv:2206.07291 [gr-qc] .
  * Matsas and Vanzella [1999] G. E. A. Matsas and D. A. T. Vanzella, Phys. Rev. D 59, 094004 (1999), arXiv:gr-qc/9901008 .
  * Good  _et al._ [2016] M. R. R. Good, P. R. Anderson, and C. R. Evans, Phys. Rev. D 94, 065010 (2016), arXiv:1605.06635 [gr-qc] .
  * Good and Ong [2020] M. R. Good and Y. C. Ong, Eur. Phys. J. C 80, 1169 (2020), arXiv:2004.03916 [gr-qc] .
  * Good  _et al._ [2021] M. R. R. Good, J. Foo, and E. V. Linder, Class. Quant. Grav. 38, 085011 (2021), arXiv:2006.01349 [gr-qc] .
  * Hawking [1975] S. Hawking, Commun. Math. Phys. 43, 199 (1975).
  * Fulling [2005] S. A. Fulling, J. of Mod. Optics 52, 2207 (2005).
  * Good  _et al._ [2022] M. R. Good, C. Singha, and V. Zarikas, Entropy 24, 1570 (2022).
  * Higuchi  _et al._ [1992] A. Higuchi, G. E. A. Matsas, and D. Sudarsky, Phys. Rev. D 46, 3450 (1992).
  * Landulfo  _et al._ [2019] A. G. Landulfo, S. A. Fulling, and G. E. Matsas, Phys. Rev. D 100, 045020 (2019), arXiv:1907.06665 [gr-qc] .
  * Verlinde [2011] E. Verlinde, Journal of High Energy Physics 2011, 1 (2011).
  * Milgrom [1983] M. Milgrom, Astrophys. J. 270, 365 (1983).
  * McCulloch [2007] M. McCulloch, Monthly Notices of the Royal Astronomical Society 376, 338 (2007).
  * Jacquet  _et al._ [2020] M. J. Jacquet, S. Weinfurtner, and F. König, “The next generation of analogue gravity experiments,”  (2020).