# Maat: Automatically Analyzing VirusTotal for Accurate Labeling and Effective
Malware Detection

Aleieldin Salem<EMAIL_ADDRESS>Technische Universität MünchenGarching bei
MünchenGermany ,  Sebastian Banescu<EMAIL_ADDRESS>Technische Universität
MünchenGarching bei MünchenGermany and  Alexander Pretschner
<EMAIL_ADDRESS>Technische Universität MünchenGarching bei MünchenGermany

###### Abstract.

The malware analysis and detection research community relies on the online
platform VirusTotal to label Android apps based on the scan results of around
60 antiviral scanners. Unfortunately, there are no standards on how to best
interpret the scan results acquired from VirusTotal, which leads to the
utilization of different threshold-based labeling strategies (e.g., if ten or
more scanners deem an app malicious, it is considered malicious). While some
of the utilized thresholds may be able to accurately approximate the ground
truths of apps, the fact that VirusTotal changes the set and versions of the
scanners it uses makes such thresholds unsustainable over time. We implemented
a method, Maat, that tackles these issues of standardization and
sustainability by automatically generating an Machine Learning (ML)-based
labeling scheme, which outperforms threshold-based labeling strategies. Using
the VirusTotal scan reports of 53K Android apps that span one year, we
evaluated the applicability of Maat's ML-based labeling strategies by
comparing their performance against threshold-based strategies. We found that
such ML-based strategies (a) can accurately and consistently label apps based
on their VirusTotal scan reports, and (b) contribute to training ML-based
detection methods that are more effective at classifying out-of-sample apps
than their threshold-based counterparts.

android security, malware detection, machine learning

††ccs: Security and privacy Malware and its mitigation††ccs: Security and
privacy Mobile and wireless security

##  1\. Introduction

A fundamental step in the process of implementing and evaluating novel malware
detection methods is the creation of ground truths for the malicious and
benign applications (hereafter apps) used to train those methods by accurately
labeling them as malicious and benign. Inaccurate labels might impact the
reliability of studies that inspect trends adopted by malicious apps, and,
more importantly, might impede the development of effective detection methods
(Hurier et al., 2017, 2016; Miller et al., 2016; Kantchelian et al., 2015).
Manual analysis and labeling of apps cannot cope with the frequent release of
malware. Moreover, some detection methods (e.g., machine-learning-based
methods), require large numbers of data for training, which renders manual
analysis and labeling infeasible. Consequently, researchers turn to online
platforms, such as VirusTotal (VirusTotal, 2019), that provide scan results
from different commercial antiviral software (Wang et al., 2018; Li et al.,
2017; Suarez-Tangil et al., 2017; Wei et al., 2017; Yang et al., 2017).

The platform, VirusTotal, does not label apps as malicious and benign. Given a
hash of an app or its executable, the platform returns the labels given by
around 60 antiviral scanners to the app along with information about its
content and runtime behavior. It is up to the platform's user to decide upon
strategies to interpret such information to label apps as malicious and
benign. Unfortunately, there are _no standard procedures_ for interpreting the
scan results acquired from VirusTotal to label apps. Researchers hence use
their intuitions and adopt ad hoc threshold-based strategies to label the apps
in the datasets used to train and evaluate their detection methods or release
to the research community as benchmarks. In essence, threshold-based labeling
strategies deem an app as malicious if the number of antiviral scanners
labeling the apps as malicious meets a certain threshold. For example, based
on VirusTotal's scan reports, Li et al. labeled the apps in their
_Piggybacking_ dataset as malicious if at least one scanner labeled them as
malicious (Li et al., 2017). Pendlebury et al. labeled an app as malicious if
four or more scanners did so, and based the evaluation of their tool on such
threshold (Feargus Pendlebury and Cavallaro, 2019). Wei et al. labeled apps in
the _AMD_ dataset as malicious if 50% or more of the total scanners labeled an
app as such (Wei et al., 2017). Finally, the authors of the _Drebin_ dataset
(Arp et al., 2014) labeled an app as malicious if at least two out of ten
scanners they manually selected courtesy of their reputation (e.g., AVG,
BitDefender, Kaspersky, McAfee) did so.

Some of the aforementioned labeling strategies may indeed accurately label
apps better than others. However, researchers have found VirusTotal to
frequently change (e.g., by manipulating the scanners it uses in its scan
reports) (Peng et al., 2019; Miller et al., 2016; Mohaisen and Alrawi, 2014).
Changing the (number of) scanners in the scan reports affects labeling
strategies that deem apps as malicious based on a fixed number of VirusTotal
scanners that label apps as such, namely threshold-based labeling strategies,
as follows. Threshold values that used to yield the most accurate labels might
change in the future as VirusTotal changes the scanners it includes in its
scan reports. In other words, VirusTotal's dynamicity renders fixed threshold-
based labeling strategies not sustainable. To cope with the dynamicity of
VirusTotal, researchers have to manually analyze VirusTotal scan reports to
identify the current optimal thresholds to use in labeling the apps they use
in their experiments. In addition to infeasibility, this analysis process is
subjective and not systematic as researchers are expected to adopt different
approaches to analyzing VirusTotal scan reports to find the current optimal
thresholds.

Using out-of-date or inaccurate thresholds alters the distribution of
malicious and benign apps in the same dataset, effectively yielding different
datasets and, in turn, different detection results as revealed by recent
results (Feargus Pendlebury and Cavallaro, 2019; Salem and Pretschner, 2018).
On the one hand, researchers might dismiss promising detection approaches,
because they underperform on a dataset that utilizes a labeling strategy that
does not reflect the true nature of the apps in the dataset. On the other
hand, developers of inadequate detection methods might get a false sense of
confidence in the detection capabilities of their detection methods because
they perform well, albeit using an inaccurate labeling strategy (Pendlebury et
al., 2018; Sanders and Saxe, 2017).

Until a more stable alternative to VirusTotal is introduced, the research
community will continue to use VirusTotal to label apps using subjective
thresholds. So, the overarching objective of this paper is to provide the
research community with actionable insights about VirusTotal, the aspects of
its dynamicity, its limitations, and how to optimally interpret its scan
reports to label Android apps accurately. As part of this objective, we
demonstrate how threshold-based labeling strategies can be used to label
Android apps. However, through our measurements and experiments, we strengthen
existing evidence that this breed of labeling strategies is subjective, not
sustainable, and requires regular analysis of VirusTotal in order to be
effectively used to label apps. To address the limitations of threshold-based
labeling strategies, we suggest an automated and systematic procedure to infer
the adequate labeling strategy using our framework, Maat111Maat refers to the
ancient Egyptian concepts of truth, balance, harmony, and justice. Our
framework builds ML-based labeling strategies that harmonize the labels given
by different VirusTotal scanners to provide accurate and reliable labels to
apps.. Maat automatically analyzes a set of VirusTotal scan reports of pre-
labeled apps to identify the set of (likely) correct and stable VirusTotal
scanners at a given point in time. Maat then uses ML to build labeling
strategies based on this and further information. Maat defines our technical
contribution. Our methodological contribution then is a direct consequence:
Whenever a new detection method is to be trained or evaluated, we suggest to
apply Maat to the _most recent_ VirusTotal scan reports, hence build the
_currently best_ labeling strategy, and use this strategy to (re-)label
existing collections of apps. However, unlike threshold-based labeling
strategies, our experiments show that Maat's ML-based labeling strategies can
withstand VirusTotal's dynamicity for longer periods of time and, hence, need
to be re-trained on a regular basis.

The contributions of this paper, therefore, are:

  * •

Implementing and publicly releasing the code of Maat (Section 4): a framework
that provides the research community with a systematic method to generate ML-
based labeling strategies on-demand based on the current scan results provided
by VirusTotal. The results of our experiments show that Maat's ML-based
labeling strategies are less sensitive to the dynamicity of VirusTotal, which
enabled them to (a) accurately label apps based on their VirusTotal scan
reports more consistently than their threshold-based counterparts and (b)
improve the detection capabilities of ML-based detection methods (Section 5).

  * •

Through the measurements and experiments conducted in this paper, we found
four main limitations of VirusTotal that sometimes undermine its reliability
and usefulness. Those limitations are (a) refraining from frequently
reanalyzing and re-scanning apps, (b) changing the set of scanners used to
scan the same apps on frequent basis, (c) using inadequate versions of
scanners that are designed to detect malicious apps for other platforms, and
(d) denying access to the history of scan reports (Section 6).

  * •

Unlike previous research efforts that are in pursuit of identifying a
universal set of VirusTotal scanners that are more correct that authors, in
Section 4.2, we demonstrate that there is no universal set of scanners that
can always accurately label apps in any dataset and at any time period. In
fact those two factors significantly alters the correct set of scanners based
on Mohaisen et al.'s correctness score (Mohaisen and Alrawi, 2014). However,
using Maat, researchers can automate the process of identifying the current
set of VirusTotal scanners that can accurately label apps within a given time
period and, in turn, help train reliable ML-based labeling strategies.

##  2\. Datasets

In this section, we briefly discuss the composition and the role of the
datasets we used in motivating the need for Maat and for conducting
experiments to evaluate it. Table 1 shows the datasets we use in this paper.

The largest dataset we use in this paper is a combination of 24,553 malicious
apps from the _AMD_ dataset (Wei et al., 2017) and 24,162 benign apps we
downloaded from _AndroZoo_ (Allix et al., 2016). The malicious apps of _AMD_
are meant to provide an overview of malicious behaviors that can be found in
Android malware, spanning different malware families (e.g., DroidKungFu(Zhou
and Jiang, 2012), Airpush(Dunham et al., 2014), Dowgin(Feizollah et al.,
2015), etc.) and different malware types (e.g., Adware, Ransomware(ware), and
Trojan). To build the dataset, the authors of _AMD_ only considered apps whose
VirusTotal scan reports indicate that at least 50% of the scanners deem them
as malicious, clustered them into 135 malware families, and manually analyzed
samples of each family to ensure their malignancy. After analysis, the
behavior of each family is represented as human-readable, graphical
representation222Example: Airpush family’s first variety
(http://tiny.cc/34d86y) of the behavior adopted by apps in each of the 135
malware families that can be found in the dataset. The involvement of human
operators in labeling the apps and the high number of scanners deeming them
malicious significantly decreases the likelihood of a benign app being
mistakenly labeled as malicious. So, we consider all apps in the _AMD_ dataset
as malicious.

As for the benign apps we acquired from _AndroZoo_ , we downloaded a total of
30,023 apps that were gathered from the Android official app store, _Google
Play_. Google Play employs various checks to ensure the sanity of an app upon
being uploaded (Oberheide and Miller, 2012), but sometimes malicious apps make
it to the marketplace (Wang et al., 2018; Luo and Yan, 2014). So, we only
considered apps whose VirusTotal scan reports indicate that no scanners deemed
them as malicious at any point in time. This criterion does not guarantee that
the apps' scan reports will continue to have a _positives_ attribute of zero
in the future. However, given that the apps were collected from Google Play,
which already employs various checks to ensure the sanity of an app upon being
uploaded to the marketplace (Oberheide and Miller, 2012), we presume that the
VirusTotal scan reports of such apps will not radically change in the future.
Consequently, we consider all apps in the _GPlay_ dataset that fit the
aforementioned criterion (i.e., 24,162 apps) as benign.

Table 1. A summary of the datasets we utilize in this paper, their
composition, their sources, and the experiments within which they are used.

Dataset Name | Total Apps | Source | Usage
---|---|---|---
_AMD+GPlay_ | 24,553 |  _AMD_ 's Website |  $\circ$ Training ML-based labeling strategies (Section 4)
24,162 |  _AndroZoo_ 's servers |  $\circ$ Calculating scanner correctness (Section 4.2)
_AndroZoo_ | 6,172 |  _AndroZoo_ 's servers |  $\circ$ Training ML-based detection methods (Section 5.2)
_Hand-Labeled_ | 100 |  _AndroZoo_ 's servers |  $\circ$ Demonstrating impact of VirusTotal's dynamicity on threshold-based labeling strategies (Section 3)
$\circ$ Testing accuracy of labeling strategies (Section 5.1)
$\circ$ Testing accuracy of ML-based detection methods (Section 5.2)
_Hand-Labeled 2019_ | 100 |  _AndroZoo_ 's servers |  $\circ$ Demonstrating impact of VirusTotal's dynamicity on threshold-based labeling strategies (Section 3)
$\circ$ Testing accuracy of labeling strategies (Section 5.1)
$\circ$ Testing accuracy of ML-based detection methods (Section 5.2)

We refer to the combination of apps in the two previous datasets as
_AMD+GPlay_ , whose VirusTotal scan reports are used to train Maat's ML-based
labeling strategies. On April 12th, 2019, we downloaded the existing scan
reports of those apps. The overwhelming majority of the apps in the _AMD_
dataset were last scanned in 2018, albeit spread across different months. For
simplicity, we refer to those multiple points in time as 2018 to mark the year
in which the apps were last scanned. Between April 12th, 2019, and November
8th, 2019, we reanalyzed all of the 53K apps and downloaded the latest
versions of their VirusTotal reports every two weeks in accordance with
Kantchelian et al.'s recommendations (Kantchelian et al., 2015). Aware of the
fact that some apps in the dataset are as old as eight years, we attempted to
acquire their older VirusTotal scan reports. Unfortunately, access to such
reports is not available under academic licenses.

The second dataset we use is a random collection of 6,172 apps developed in
between 2018 and 2019 and downloaded from _AndroZoo_. So, we refer to it as
_AndroZoo_ throughout this paper. Unlike apps in _AMD+GPlay_ , this dataset
does not focus on a particular marketplace or class (e.g., malicious). This
dataset is used Section 5 to assess the ability of different labeling
strategies to train more accurate detection methods. This dataset is meant to
simulate the process of a researcher acquiring new Android apps and labeling
them to train a ML-based detection method that detects novel Android malware,
that VirusTotal scanners are yet to assign labels to. So, we use apps in this
dataset to statically extract numerical features from their APK archives,
represent them as vectors, and use them for training different types of
machine learning classifiers. We use different threshold-based and ML-based
labeling strategies to label those feature vectors prior to training the
aforementioned classifiers.

The trained classifiers are then used to label apps in two small test datasets
and compare the predicted labels with the ground truth. We refer to those
small datasets as _Hand-Labeled_ 333http://tiny.cc/95bhaz and _Hand-Labeled
2019_ 444http://tiny.cc/a7bhaz. Both datasets comprise 100 Android apps that
were downloaded from _AndroZoo_ and _manually analyzed and labeled_ , to
acquire reliable ground truth. The exact process we adopted in analyzing and
labeling those apps can be found online. The primary difference between both
datasets is that apps in the latter were developed in 2019. We ensured that
apps in both datasets do not overlap with apps in the _AMD+GPlay_ and
_AndroZoo_ datasets. We manually analyzed such 200 apps to acquire reliable
ground truth that depicts the apps' true nature.

##  3\. Motivating Examples

During the evaluation of an Android malware detection method we implemented,
we came across a dubious scenario involving a test app called
TP.LoanCalculator that is part of the _Piggybacking_ (Li et al., 2017)
dataset. Despite being labeled by the dataset authors as malicious, our
detection method deemed this test
app5552b44135f245a2bd104c4b50dc9df889dbd8bc79b as benign because it had the
same metadata (e.g., package name and description), compiler, and even
codebase as one benign app666d8472cf8dcc98bc124bd5144bb2689785e245d83 that the
detection methods keeps in a repository of reference benign apps.

Apps in the _Piggybacking_ dataset were labeled with the aid of VirusTotal
scan reports (Li et al., 2017). After querying VirusTotal for the scan reports
of both apps, we found that the test app was labeled malicious by 14 out of 60
antiviral software scanners, whereas all scanners deemed the reference app as
benign, which coincides with the authors' labels. However, we noticed that the
scan reports acquired from VirusTotal indicated that the apps were last
analyzed in 2013. So, we submitted the apps' Android Package (APK) archives
for re-analysis in 2018 to see whether the number of scanners would differ.
After re-analysis, the malicious test app had three more scanners deemed it
malicious. More importantly, the number of scanners deeming the benign
reference app as malicious changed from zero to 17 after re-analysis. So, the
reference app initially labeled and released as part of the _Piggybacking_
dataset as a benign app is, in fact, another version of a malicious app of the
type Adware.

The authors of _Piggybacking_ did not intentionally mislabel apps. The most
likely scenario is that, at the time of releasing the dataset, the reference
app was still deemed as benign by the VirusTotal scanners. To further
demonstrate the impact of time on the scan reports of both apps, we reanalyzed
them six months later (i.e., in April 2019), to check whether the number of
VirusTotal scanners deeming them malicious changed. We found that the number
of scanners deeming the test and reference apps as malicious decreased from 17
to 11 and 10 scanners, respectively. Moreover, the total number of scanners
respectively changed from 60 scanners to 59 and 57.

Another interesting fact is that the percentages of scanners deeming both apps
as malicious are 18.64% and 19.3%, with some renowned scanners including AVG,
McAfee, Kaspersky, Microsoft, and TrendMicro continuing to deem both apps as
benign. According to the dataset authors' strategy to label an app as
malicious if at least one scanner deems it so (Li et al., 2017), both apps
would be labeled as malicious. The same would not hold for the authors of the
_AMD_ dataset who consider an app as malicious if at least 50% of the
VirusTotal scanners deem it malicious (Wei et al., 2017).

With this example, we wish to demonstrate the following issues with using
VirusTotal to label Android apps. First, the lack of a standard method to
interpret VirusTotal's scan results encourages researchers to use different
threshold-based labeling strategies to label apps, which might lead to
completely different verdicts on the labels of the same apps. Assuming that
the research community manages to standardize the thresholds used to label
apps based on their VirusTotal scan reports, the second issue is that such
scan reports are dynamic and continuously change over time due to either (a)
scanners updating their verdicts on some apps or (b) the platform
adding/removing scanners (Miller et al., 2016; Mohaisen and Alrawi, 2014).
This continuous change leads to scan report attributes, such as the number of
scanners deeming an app as malicious (i.e., _positives_), to fluctuate.
Consequently, thresholds that used to reflect the ground truth of an app
accurately are also expected to change.

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC
Scoresvt$\displaystyle\geq$1vt$\displaystyle\geq$2vt$\displaystyle\geq$3vt$\displaystyle\geq$4vt$\displaystyle\geq$5vt$\displaystyle\geq$6vt$\displaystyle\geq$7vt$\displaystyle\geq$8vt$\displaystyle\geq$9vt$\displaystyle\geq$10vt$\displaystyle\geq$25%vt$\displaystyle\geq$50%drebin

(a) _Hand-Labeled_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC
Scoresvt$\displaystyle\geq$1vt$\displaystyle\geq$2vt$\displaystyle\geq$3vt$\displaystyle\geq$4vt$\displaystyle\geq$5vt$\displaystyle\geq$6vt$\displaystyle\geq$7vt$\displaystyle\geq$8vt$\displaystyle\geq$9vt$\displaystyle\geq$10vt$\displaystyle\geq$25%vt$\displaystyle\geq$50%drebin

(b) _Hand-Labeled 2019_

Figure 1. The labeling accuracy of different threshold-based labeling
strategies against apps in _Hand-Labeled_ and _Hand-Labeled 2019_ datasets
based on their VirusTotal scan reports downloaded between July 5th, 2019 and
November 8th, 2019. Accuracy is calculated in terms of the MCC of each
labeling strategy.

To demonstrate the fluctuations of VirusTotal's scan reports and its
attributes and how that impacts the performance of threshold-based labeling
strategies, consider the plots in Figure 1. In this figure we plot the
performance of different threshold-based labeling strategies, including ones
previously-utilized within the literature, on the _Hand-Labeled_ and _Hand-
Labeled 2019_ datasets between July 5th, 2019 and November 8th, 2019 in terms
of MCC metric (scikit learn, 2019d). As seen in the figure, the labeling
accuracy noticeably differs from on threshold-based labeling strategy to
another. For example, researchers using vt$\geq$3 or vt$\geq$4 (Feargus
Pendlebury and Cavallaro, 2019; Miller et al., 2016) to label apps in the
_Hand-Labeled 2019_ dataset will get more accurate labels than those adopting
vt$\geq$1 (Li et al., 2017) or vt$\geq$50% (Wei et al., 2017). Moreover, the
accuracies of such labeling strategies appear to fluctuate over a mere period
of four months especially against recently-developed Android (malicious) apps
in the _Hand-Labeled 2019_ dataset. In conclusion, the dynamicity of
VirusTotal means that fixed thresholds cannot be relied on for prolonged
periods of time to label Android apps as malicious and benign. So, researchers
have to analyze VirusTotal scan reports in order to identify the currently
optimal threshold to use in the labeling process, which is indeed a time-
consuming process. This process is further prolonged by the fact that
VirusTotal does not automatically re-analyze the apps it possesses in its
repositories and relies on its users to manually initiate re-scans. Needless
to say, the larger the datasets a researcher possesses, the longer it takes to
re-scan all apps given the limitations on the number of Application
Programming Interface (API) requests allowed per day for academic licenses.

VirusTotal Limitation 1 VirusTotal does not rescan the apps it possesses on a
regular basis and delegates this task to manual requests issued by its users.
One direct consequence of this decision is prolonging the process of acquiring
up-to-date scan reports of apps especially under academic licenses that grant
users a limit of 30K API requests per day.

##  4\. Maat's ML-based Labeling Strategies

In the previous section, we demonstrated the subjectivity of threshold-based
labeling strategies and their sensitivity to VirusTotal's dynamicity and
evolution. This dynamicity forces researchers to identify the current optimal
thresholds, which entails a semi-automatic process that is infeasible to
perform on a regular basis. Our framework, Maat, is designed to address these
problems as follows. First, to avoid the subjectivity of threshold-based
labeling strategies, Maat provides the research community with a systematic
and standardized method to analyze VirusTotal scan reports to devise labeling
strategies. Second, Maat's processes of analyzing VirusTotal scan reports and
of devising ML-based labeling strategies are on-demand and fully automated.
Third, the resulting ML-based labeling strategies do not rely on a fixed
number of VirusTotal scanners to label apps as malicious and benign. Instead,
using two different types of features extracted from VirusTotal scan reports,
Maat relies on ML algorithms to identify the currently correct and stable
VirusTotal scanners, which makes them less susceptible to the dynamicity of
VirusTotal and the changes it introduces to apps' scan reports.

###  4.1. Overview

Prior to delving into the measurements and experiments we performed, we
briefly discuss how our method, Maat, mines VirusTotal scan reports to build
ML-based labeling strategies. As seen in Figure 2, Maat starts by analyzing
the VirusTotal scan reports of apps in the training dataset that we explicitly
reanalyzed via VirusTotal and downloaded at different points in time (i.e.,
$t_{0},t_{1},...,t_{m}$). The training dataset that Maat uses to train ML-
based labeling strategies comprises the scan reports of apps in the
_AMD+GPlay_ dataset gathered between 2018 and September 27th, 2019.

In phase (1) from Figure 2 we identify the VirusTotal scanners that achieve an
average overall correctness score777The correctness score is based on the
ground truths given by the authors of _AMD_ to apps using their hybrid
analysis process that combines relying on VirusTotal scan reports and manual
analysis as discussed in Section 2. of at least 90% between 2018 and September
27th, 2019 as the most correct scanners. Maat also finds the scanners that
changed their verdicts at most 10% of the time (i.e., were stable 90% of the
time), are considered in the next phase to extract features from the scan
reports. The output of this phase is an intersection of the most correct and
stable VirusTotal scanners. The exact processes we adopted to find the most
correct and stable scanners are detailed in Section 4.2 and Section 4.3,
respectively.

Figure 2. The process adopted by Maat to construct ML-based labeling
strategies by analyzing VirusTotal scan reports and training a random forest
to label apps as malicious and benign according to their VirusTotal scan
reports.

In phase (2), we extract features from the VirusTotal scan reports of apps in
the _AMD+GPlay_ dataset. There are two types of features we extract from the
reports, namely _engineered_ features and _naive_ features.

Engineered features attempt to leverage the insights we gained from the
previous sections (e.g., which scanners are correct). So, based on the output
from phase (1), we consider the verdicts given to apps in the training dataset
only by the set of most correct and stable scanners. To accommodate the impact
of time on the maturity of an app's scan report, we also include the age of a
scan report in years, the number of times an app has been submitted for
(re)analysis (i.e., _times_submitted_), the _positives_ attribute, and the
_total_ attribute in this feature set. Lastly, to capture any patterns that
Android (malicious) apps share in terms of functionalities and runtime
behaviors, we extract from the VirusTotal scan reports the permissions that
apps request in their AndroidManifest.xml files, and the tags given to them by
VirusTotal (e.g., _checks-gps_ , _contains-elf_ , _sends-sms_ , etc.).

Naive features do not consider the outputs of phase (1). With naive features,
we consider the verdicts given by _all_ VirusTotal scanners to the apps in the
training dataset. So, the feature vector extracted from a VirusTotal scan
report will be a sequence of integers depicting the label given by each
scanner to an app (i.e., -1 for not scanned, 0 for scanned and deemed benign,
and 1 for scanned and deemed malicious). For example, assume that the scan
report of an arbitrary app ($\alpha^{*}$) contained scan results of three
scanners, that respectively deemed ($\alpha^{*}$) as malicious, malicious, and
benign, the feature vector depicting this scan report will be
($\hat{x}_{\alpha^{*}}=(1,1,0)$). With naive features, we allow ML-based
labeling strategies to utilize the verdicts of all VirusTotal freely scanners
regardless of their correctness or stability.

Phase (3) is an optional phase that selects the most informative features
extracted from the training dataset's scan reports. To avoid having to choose
the number of features to select arbitrarily, we utilize the SelectFromModel
(scikit learn, 2019a) technique to select the most informative features
automatically. In essence, this technique selects features based on the
importance given to them by a model (e.g., logistic regression, support vector
machines, decision trees, etc.). For example, during training, decision trees
iteratively utilize a criterion (e.g., Gini index), to decide upon the next
feature to consider in splitting data points into two, or more, classes; in
our case, this feature could be a scanner's verdict regarding the label of an
app. Ultimately, the trained tree will compile a set of features that it used
during splitting and assign an importance value to each one of them. The
SelectFromModel feature selection technique uses such importance values and
returns the user those features with importance values more than a preset
threshold (i.e., $1\times 10^{-5}$ in the case of decision trees). For our
experiments, we rely on decision trees as the model used by the
SelectFromModel technique to extract the most informative features.

We envision the process of utilizing the features extracted from VirusTotal
scan reports to label apps as a series or combination of questions, such as
_how many scanners deem the app malicious?_ _how old is the app?_ _does a
renowned scanner (e.g., AVG) deem the app as malicious?_ The machine learning
classifier that mimics this model, we reckon, is a decision tree. In order not
to rely on the decisions made by a single tree, Maat trains ML-based labeling
strategy as a collection of trees or a _random forest_. To estimate the
hyperparameters (e.g., the maximum depth each tree is allowed to grow), that
train the most effective forests, we use the technique of _grid search_
(scikit learn, 2019b) to select from among a set of parameters listed in on
Maat's website In our experiments, we compare the performance of random
forests trained using both search techniques.

The output of phase (4) is a random forest that takes a vector of numerical
features extracted from an app's VirusTotal scan report and returns a label
depicting the class of the app (i.e., 1.0 for malicious and 0.0 for benign).
Effectively, this random forest is a labeling strategy. In phase (5), given
the VirusTotal scan report of an arbitrary Android app, the report is
represented as a feature vector that matches the features used by the random
forest (e.g., naive versus engineered features), and is used to predict the
app's class.

###  4.2. Correctness of VirusTotal Scanners

As part of extracting engineered features to train ML-based labeling
strategies, in this section, we describe the process adopted by Maat to find
the set most accurate VirusTotal scanners during a given period of time. Given
the scan reports of apps in the training dataset gathered over a period of
time, Maat relies on Mohaisen et al.'s definition of scanner _correctness_ to
identify the set of correctVirusTotal scanners over this period. In (Mohaisen
and Alrawi, 2014), Mohaisen et al. defined correctness as follows: For a given
dataset, the correctness of an antiviral scanner is the number of correct
detections normalized by the size of the dataset. So, for each VirusTotal
scanner, Maat calculates the number of apps in the _AMD+GPlay_ dataset that it
managed to correctly label as malicious, and divide that number by the size of
the dataset. The correctness scores of each scanner at different points in
time are then averaged. Scanners whose correctness scores are greater than or
equal to 0.90 are included in the set of correct VirusTotal scanners. We chose
the number 0.90 to tolerate any fluctuations in the correctness rate that may
occur, for example, due to changing policies on how to label ambiguous malware
types, such as Adware, or the temporary use of inadequate versions of
scanners. So, the definition of a correct scanner we adopt in this paper is:

Definition A correct VirusTotal scanner is one that is able to correctly label
apps in a given dataset of pre-labeled apps, such that the number of correctly
labeled apps over the total number of apps in the dataset is greater than or
equal to 0.90.

Using this criterion, we retrieved a list of 18 scanners, namely AhnLab-V3,
Avira, Babable, CAT-QuickHeal, Comodo, Cyren, DrWeb, ESET-NOD32, F-Secure,
Fortinet, Ikarus, K7GW, MAX, McAfee, NANO-Antivirus, Sophos,
SymantecMobileInsight, and Trustlook.

The number of scanners that Maat identified to be correct between 2018 and
September 27th, 2019 on the _AMD+GPlay_ dataset noticeably differs from the
ten scanner Arp et al. used in 2014 to label apps in their _Drebin_ dataset
(Arp et al., 2014); only five _Drebin_ out of ten are included in Maat's list
of correct scanners. Apart from deeming them as popular and trustworthy, the
authors did not mention why they chose those ten scanners. Intuitively, Arp et
al. used a different set of Android apps and older versions of scan reports,
which implies that the set of correct scanners might change from one set of
Android apps to another and from time to time. In other words, there are no
set of VirusTotal scanners that are universally correct on different datasets.
To verify this, we retrieved the set of scanners that maintained correctness
scores of 90% or higher over a period of time for the _AMD_ , _AMD+GPlay_ ,
_GPlay_ , _Hand-Labeled_ and _Hand-Labeled 2019_ datasets (i.e., the datasets
with known ground truths).

Since we downloaded the apps in the _Hand-Labeled_ and _Hand-Labeled 2019_
datasets after downloading apps in the _AMD+GPlay_ dataset, we started re-
scanning and downloading their latest VirusTotal scan reports at later points
in time, viz. starting from July 15th, 2019 instead of 2018. So, to perform an
objective measurement, we retrieved the list of correct VirusTotal scanners
for the _AMD+GPlay_ , _Hand-Labeled_ , _Hand-Labeled 2019_ datasets within the
same time period, namely between July 15th, 2019 and September 27th, 2019. As
seen in Table 2, the set of VirusTotal scanners that have correctness rates of
at least 90% differ from one dataset to another, despite being based on scan
reports downloaded within the same period. Only a set of five scanners
continued to be the most correct in labeling apps in all datasets, viz. ESET-
NOD32, Fortinet, Ikarus, McAfee, and SymantecMobileInsight. As for the _GPlay_
dataset, a total of 58 correct scanners were found to be correct, which we
could not fit in the table.

Table 2. The set of VirusTotal scanners that had correctness rates of at least
90% between July 15th, 2019 and November 8th, 2019. Emboldened scanners depict
the intersection of the sets of correct scanners of the four datasets.

Dataset | Scanner(s) | Total
---|---|---
_AMD+GPlay_ |  |  AhnLab-V3, Avira, CAT-QuickHeal, Comodo, Cyren,
---
DrWeb, ESET-NOD32, F-Secure, Fortinet,
Ikarus, K7GW, MAX, McAfee, NANO-Antivirus,
Sophos, SymantecMobileInsight, TheHacker, Trustlook
18
_AMD_ |  |  Avira, CAT-QuickHeal, DrWeb, ESET-NOD32, F-Secure,
---
Fortinet, Ikarus, MAX, McAfee, NANO-Antivirus,
Sophos, SymantecMobileInsight
12
_Hand-Labeled_ |  |  AhnLab-V3, CAT-QuickHeal, Cyren, ESET-NOD32, Fortinet, Ikarus,
---
K7GW, McAfee, Sophos, SymantecMobileInsight, Trustlook
11
_Hand-Labeled 2019_ |  |  Ad-Aware, AegisLab, AhnLab-V3, Alibaba, Arcabit,
---
Avast-Mobile, BitDefender, ClamAV, Cyren, DrWeb,
ESET-NOD32, Emsisoft, F-Secure, FireEye, Fortinet,
GData, Ikarus, Jiangmin, K7AntiVirus, K7GW,
Kaspersky, Kingsoft, MAX, MalwareBytes, McAfee,
McAfee-GW-Edition, MicroWorld-eScan, Microsoft, NANO-Antivirus, Qihoo-360,
SUPERAntiSpyware, SymantecMobileInsight, Trustlook, ViRobot, Yandex,
ZoneAlarm, Zoner
37

We made the following observations based on the results in the table. Firstly,
the number of correct scanners found by Maat using apps in the _AMD+GPlay_
dataset remained the same (i.e., 18 scanners). However, between July 15th,
2019 and September 27th, 2019, Maat found the TheHacker scanner to be more
accurate than the Babable. Secondly, we noticed that adding more benign apps
to a dataset increases the number of scanners found to be correct. In other
words, the more malicious apps in a dataset, the lower the number of correct
scanners. For example, adding the scan reports of apps in the _GPlay_ dataset
to their _AMD_ counterparts increases the number of scanners found to be
correct between July 15th, 2019 and September 27th, 2019 from 12 scanners to
18 scanners. On a smaller scale, one can notice the same pattern in the _Hand-
Labeled_ and _Hand-Labeled 2019_ datasets. Since the former dataset has less
benign apps and more malicious ones (i.e., 76 benign versus 24 malicious), the
number of scanners found to be correct is less than that for the latter
dataset that has 90 benign apps and only ten malicious ones. The reason behind
this, we argue, is that benign apps do not reveal the detection ability of an
antiviral scanner. That is, deeming a benign app as benign does not reveal
whether the app has been analyzed and consciously deemed as such, or whether
the scanner did not scan the app before. Malicious apps, on the other hand,
reveal the competence of a scanner; detecting a malicious app implies that the
app has been analyzed and deemed malicious. In summary, adding more malicious
apps seems to filter out incompetent scanners.

What we learned from this measurement is that the set of VirusTotal scanners
that are correct over time might change depending on (a) the dataset itself
and its composition in terms of benign and malicious apps, and (b) the time
period within which the VirusTotal scan reports were gathered. However, we
noticed that there is a subset of scanners that persist across different
datasets, viz. the five scanners ESET-NOD32, Fortinet, Ikarus, McAfee, and
SymantecMobileInsight. So, is there a universal set of VirusTotal scanners
that are correct across different points in time and different datasets? The
answer depends on the definition of a universal set. If researchers expect
that set of scanners to maintain the same scanners and cardinality, then the
answer is no. However, it is expected for a small set of VirusTotal scanners
to persist within the set of correct scanners longer than others.

As discussed earlier, the composition of a dataset might affect the set of
correct scanners because adding more benign apps to the dataset conceals the
mediocrity of some scanners. Now the question is why and how does time impact
the performance of an antiviral scanner on VirusTotal? One possible answer is
the scanners suffer technical difficulties, which is reflected in their
detection performance. A more concerning possbility is that VirusTotal changes
the set of scanners it includes in the scan reports of apps across time. Since
we calculate correctness based on the verdicts of scanners found in
VirusTotal's scan reports, excluding a scanner's verdict from such scan
reports will give the wrong impression that the scanner was unable to classify
the apps correctly and, in turn, decrease the scanner's correctness score.
Luckily, Maat supports tabulating and visualizing the correctness of
VirusTotal scanners to provide its users with further insights about the
performance of different scanners over time. For example, Figure 3 displays
the correctness scores of ten scanners utilized by Arp et al. in (Arp et al.,
2014) to label apps in the _Drebin_ dataset. These scanners are
AntiVir888AntiVir refers to the antiviral scanner developed by Avira, which is
the name that VirusTotal uses for this scanner now., AVG, BitDefender, ClamAV,
ESET, F-Secure, Kaspersky, McAfee, Panda, and Sophos. We refer to this group
of scanners as the _Drebin_ scanners.

20182019-04-122019-04-262019-05-102019-05-242019-06-072019-06-212019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates0.50.60.70.80.91.0CorrectnessAVGAviraBitDefenderClamAVESET-
NOD32F-SecureKasperskyMcAfeePandaSophos

Figure 3. The overall correctness rates of the _Drebin_ scanners on apps in
the _AMD+GPlay_ dataset between 2018 and November 8th, 2019.

The correctness scores in the figure imply that some scanners are more correct
and stable than others. However, the performance of F-Secure and BitDefender
raise the following concerns. The correctness of F-Secure appears to
noticeably oscillate on a frequent basis. For example, the overall correctness
of F-Secure steeply dropped from above 0.90 on June 7th, 2019 to around 0.70
on June 21st, 2019 only to increase to 0.90 two weeks later on July 15th,
2019. While it might be acceptable that the performance of an antiviral
scanner drops and remains low, such as the case of BitDefender, there is no
apparent reason for the frequent fluctuations in performance. In what appears
to be a random decision, we found that VirusTotal excludes the verdicts of
F-Secure from the scan reports of apps in the _AMD+GPlay_ dataset. For
example, on the dates April 26th, 2019, May 10th 2019, and May 24th, 2019, the
verdicts of F-Secure were included in the scan reports of _all_ , 98%, and 97%
of apps in the dataset, respectively. Consequently, due to the exclusion of
the scanner from the scan reports of some apps, the correctness scores of
F-Secure on those dates were respectively 0.92, 0.90, and 0.88. Moreover, we
found that the scanner alters its verdicts across different points in time.
For instance, despite being included in fewer scan reports on June 7th, 2019
(i.e., 96% instead of 97%), its correctness score increased from 0.88 to 0.94.
So, a combination of its own internal decisions and VirusTotal's exclusions
contributed to its fluctuating performance.

VirusTotal Limitation 2 VirusTotal changes the set of scanners it includes in
the scan reports of apps over time by including and excluding the verdicts of
scanners regardless of the quality of those verdicts.

Second, despite its mediocre performance as reported by VirusTotal,
BitDefender continues to be given good reviews by users on the Google Play
marketplace and, more importantly, on platforms that assess the effectiveness
of antiviral software such as _AV-Test_ (Institute, 2019). Given that
VirusTotal states that the versions of scanners it uses "_may differ from
commercial off-the-shelf products. The [antiviral software] company decides
the particular settings with which the engine should run in VirusTotal_ "
(VirusTotal, 2019), we compared the VirusTotal version of BitDefender against
the one that can be found on the Google Play app marketplace. We found that,
as of September 2019, the version used by VirusTotal's for BitDefender is 7.2,
whereas the versions available on Google Play have codes between 3.3 and 3.6.
The 7.2 version of BitDefender corresponds to a free edition version developed
for Windows-based malware that targets older versions of Windows such as
Windows XP (Magazin, 2008). The positive reputation that BitDefender has in
the market suggests that using its adequate version (i.e., the one that is
designed to detect Android malware), would yield a detection performance
better than the version on VirusTotal. To verify this hypothesis, we
downloaded and installed the latest version of the BitDefender scanner from
the Google Play marketplace, installed it on an Android Virtual Device (AVD),
and used it to scan ten apps randomly sampled from the _AMD_ dataset. Unlike
the results obtained from VirusTotal that the scanners are unable to detect
any of those apps, we found that BitDefender detects 70% of the sampled apps.

VirusTotal Limitation 3 VirusTotal may replace the versions of scanners with
inadequate ones that are not designed to detect Android malware presumably
based on the request of the scanner's vendor or managing firm.

So, do some antiviral software companies believe that offering older versions
on VirusTotal will encourage users to download their products instead of
relying on the online version available on VirusTotal? Do such companies
enforce a fee on VirusTotal in order to use their new products, which the
latter did not agree to? Answering those questions is not in the scope of this
paper and, in fact, impossible to answer on behalf of antiviral software
companies.

###  4.3. Stability of VirusTotal Scanners and Scan Reports

Whether due to the dynamicity of VirusTotal or internal decisions, the
examples of F-Secure and BitDefender in the previous section show that some
VirusTotal scanners have unstable, fluctuating performance. To train reliable
ML-based labeling strategies, Maat attempts to rely on VirusTotal scanners
that do not often change the labels they assign to Android apps (i.e., stable
scanners). Maat identifies stable VirusTotal scanners over a period of time by
considering those scanners whose _certainty_ score exceeds 0.90. This score is
calculated as follows. For each scanner, the labels given by the scanner to
each app in the training dataset (i.e., _AMD+GPlay_), over a period of time
are retrieved. Regardless of the correctness of the labels, the certainty
score is calculated by dividing the total number of the most common label
(i.e., malicious versus benign) by the total number of scanners. Effectively,
the score calculates the variance of the labels that are given to apps by
VirusTotal scanners approaching zero over a period of time. As an example, say
that ESET-NOD32 had the labels
$\\{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\texttt{malicious}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\texttt{benign}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\texttt{malicious}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\texttt{malicious}}\\}$
for an app ($\alpha$); the certainty score will be three (i.e., counts of
malicious), divided by a total of four labels yielding a percentage of 0.75.
To calculate the certainty score for an entire dataset, Maat averages the
scanners' certainty scores achieved on individual apps in this dataset.

Similar to the case with the correctness scores, Maat's default threshold of
the certainty score is 0.90 to allow for marginal fluctuations in the labels
given to apps by scanners, which might be a result of VirusTotal excluding
those scanners from the apps' scan reports or changing their versions to
inadequate ones. So, we define a stable VirusTotal scanner in this paper as
follows:

Definition A stable VirusTotal scanner is one that achieves an average
certainty score of at least 0.90 on apps in a given dataset over a period of
time. This score indicates that, on average and regardless of the correctness
of the assigned label, the scanner maintained the same label it assigns to an
app 90% of the time.

Using this threshold, Maat retrieved a set of 44 scanners (i.e., 74.5%) that
had certainty scores of at least 0.90 on apps in the _AMD+GPlay_ dataset
between 2018 and September 27th, 2019. Unfortunately, VirusTotal does not
grant access to the history of scan reports of apps under academic licenses,
which hinders our efforts to assess the stability of different scanners to
periods before 2018

VirusTotal Limitation 4 VirusTotal does not grant access to academic
researchers to the history of scan reports of apps previously added and
scanned on the platform, even if such apps were added by the academic
community itself.

However, it is worth mentioning that any definition of stability or method to
retrieve the most stable scanners, including this one, suffers from two
limitations. Firstly, there is no absolute guarantee that the performance of a
VirusTotal scanner will continue to be the same, either because of technical
difficulties suffered by an antiviral software company or because of
VirusTotal's utilization of different versions of a scanner, as seen in the
case of BitDefender. Secondly, any definition of stability does not guarantee
correctness. Over the past three years, we tracked the verdicts given by
VirusTotal scanners to a repackaged, malicious
version999aa0d0f82c0a84b8dfc4ecda89a83f171cf675a9a of the K9 Mail open source
app (k9m, 2019) that has been developed by one of our students during a
practical course. Despite being a malicious app of type Ransom, the scanners
continued to unanimously deem the app as benign since February 8th, 2017, even
after analyzing and re-scanning the app. Another example is an
app10101066c16d79db25dc9d602617dae0485fa5ae6e54b2: A calculator app grafted
with a logic-based trigger that deletes user contacts only if the result of
the performed arithmetic operation is 50. that we repackaged three years ago;
the app continued to be labeled as benign by all scanners until only K7GW
recognized the app's malignancy in July 2019 and labeled it as a Trojan. As
mentioned in the previous section, Maat is meant to help optimally utilize
VirusTotal and interpret its scan results, but it cannot control either its
dynamicity or correctness of its VirusTotal. So, since the set of stable
scanners may change in the future as VirusTotal changes, it is recommended to
re-train Maat's ML-based labeling strategies that rely on the set of stable
scanners whenever possible (e.g., after re-scanning the apps in the
framework's training dataset and downloading their up-to-date scan reports).
The more interesting insight, we argue, is that the majority of antiviral
software firms that allow VirusTotal to utilize their products do not seem to
rely on the platform as a source of apps to analyze and include in their
signatures database. So, it seems that those firms mainly focus on Android
malicious found in the wild (e.g., in an app marketplace), and pay little to
no attention to ones that are uploaded to platforms, such as VirusTotal, even
if they are malicious.

###  4.4. Features Extracted from Scan Reports

In this section, we recap on the type of features that Maat extracts from the
VirusTotal scan reports of its training dataset, and relate them to the
insights we gained from the measurements performed in the previous sections.
As mentioned in Section 4.1, there are two types of features that Maat can
extract from scan reports, namely engineered and naive.

##### Engineered Features

.

The engineered features Maat extracts from scan reports (listed online) can be
divided into three caegories. In the first category of features, Maat
considers the verdicts of VirusTotal scanners that had a correctness score and
a certainty score of at least 0.90. As seen in Section 4.2 and Section 4.3,
the performance of some correct scanners, such as F-Secure, fluctuates over
time (i.e., not stable), whereas the stability of scanners in general does not
guarantee correctness. To mitigate both issues, Maat relies on the verdicts
given by the intersection of correct and stable scanners. Taking the
intersection between the 18 correct scanners the 44 stable scanners yields a
set of 16 VirusTotal scanners that include all the correct scanners apart from
F-Secure, which already exhibited fluctuation in detection performance (see
Section 4.2), and Trustlook. The reason behind this is that the continuity of
correctly labeling apps leads to the stability of labels. In other words, if a
scanner is consistently accurate at giving the same, correct label to an app,
it is ipso facto a stable scanner. This relationship between correctness and
stability, as discussed before, does not go the other way around (i.e.,
stability does not imply correctness).

The second category of features is based on attributes found in VirusTotal
scan reports that imply the age and, perhaps, the maturity of the app and its
scan report. Maat extracts the age of the app's scan report as the difference
between the extraction date and that of _first_seen_ along with the
_times_submitted_ attribute, the _positives_ attribute, and the _total_
attribute. The first two features are meant to reflect the maturity of an
app's scan report. In our analysis, we noticed that older malicious apps have
higher ranges of _positives_ and _total_ values than newer ones. For example,
as of September 27th, 2019, the malicious apps in the _AMD_ dataset have an
average _positives_ value of 26.26$\pm$5.53 and and average _total_ value of
59.91$\pm$1.13. Regardless of how malicious they are, newly-developed
malicious apps–and indeed false-positive benign apps–can never reach those
ranges, perhaps for years. So, we thought that such values might assist Maat
to discern malicious, benign, and ambiguous malicious apps.

The third and last category of engineered features is meant to approximate the
structure and behavior of apps. Malicious apps tend to adopt similar
structures, re-use libraries, exploit similar vulnerabilities, or share the
same codebases. We assume that such trends can be reflected in the permissions
these apps request and the tags assigned to them by VirusTotal. So, as part of
the engineered features, we include the list of permissions (not) requested by
the app and the tags (not) assigned to them by VirusTotal.

In total, Maat extracts 372 features from the VirusTotal scan report of each
app. Having discussed the impact of the curse of dimensionality on the
performance of ML algorithms, we implemented Maat to select the most
informative features from this feature set. In Section 5, we discuss the types
of features selected from the full corpus of engineered features.

#####  Naive Features.

Naive features comprise the verdicts of all VirusTotal scanners, regardless of
their correctness or stability. With this set of features, as mentioned
earlier, we allow Maat's random forests to identify and choose the VirusTotal
scanners that train the most effective ML-based labeling strategies. We use
this type of feature for three reasons. First, naive features are fast and
easy to extract from VirusTotal scan reports. Second, we wish to investigate
whether allowing Maat's random forests to select scanners would yield a set of
scanners that overlap with the ones identified using the measurements in
Section 4.2 and Section 4.3. Third, we wish to assess whether the second and
third categories of engineered features help Maat train better ML-based
labeling strategies or hinder their performance. The dimensionality of naive
features is 60 scanners. However, we also allow Maat to select the most
informative features (i.e., scanner verdicts, in this case). As of September
27th, 2019, using the scan reports of apps in the _AMD+GPlay_ dataset, the set
of scanners selected from the full corpus of naive features comprised 16
scanners.

##  5\. Evaluating Maat

As discussed in Section 1, the accurate labeling of previously-analyzed and
detected apps is fundamental for the effectiveness of detection methods. So,
the quality of labeling strategies can be assessed in terms of (a) their
abilities to assign labels to previously-analyzed apps (e.g., based on their
VirusTotal scan reports) that accurately reflect their ground truth, and (b)
the extent to which such accurate labeling contributes to the effectiveness of
detection methods against out-of-sample apps. In this section, we evaluate the
performance of threshold-based labeling strategies versus Maat's ML-based
labeling strategies according to the aforementioned two criteria. In Section
5.1, we verify whether Maat's ML-based labeling strategies trained at one
point in time can maintain labeling accuracies in the future that rival those
achieved using threshold-based labeling strategies that adopt the current
optimal thresholds. The second set of experiments in Section 5.2 compares the
impact of threshold-based labeling strategies versus that of Maat's ML-based
labeling strategies on the effectiveness of ML-based detection methods against
out-of-sample apps over time.

###  5.1. Accurately Labeling Apps

The main objective in this set of experiments is to verify whether ML-based
labeling strategies trained at one point in time can sustain a similar
labeling accuracy using VirusTotal scan reports downloaded in the future. If
true, then unlike their threshold-based counterparts, Maat ML-based labeling
strategies do not need to be re-trained regularly to accommodate for
VirusTotal's dynamicity. So, we trained Maat ML-based labeling strategies
using VirusTotal scan reports of apps in the _AMD+GPlay_ dataset downloaded on
2018, and used the trained strategies to label apps in the test datasets
_Hand-Labeled_ and _Hand-Labeled 2019_ datasets between July 5th, 2019 and
November 8th, 2019. In the experiments, we used ML-based labeling strategies
that use the two types of features mentioned in Section 4.4 (i.e., engineered
versus naive) and trained using the technique of grid search. We also included
labeling strategies that used the SelectFromModel technique to select the most
informative features. For readability, we shorten the names of ML labeling
strategies in the following manner: engineered and naive features are referred
to as Eng and Naive, respectively, grid search is referred to as GS, and
strategies that use selected features are referred to using Sel. For example,
a labeling strategy that uses selected engineered features and the grid search
techniques will be referred to as Eng Sel GS.

Figure 4 depicts an example of the results we are discussing in this section.
In this figure, we compare the labeling performance of the ML-based labeling
strategies trained using scan reports from 2018 against two types of
threshold-based labeling strategies. The solid red line depicts the
performance of threshold-based strategies using the optimal threshold at each
scan date identified as follows. At each scan date, we calculated the labeling
accuracy of each threshold between one and 60 against apps in the _AMD+GPlay_
dataset. The threshold that yields the best MCC scores is considered as the
optimal threshold for this scan date. For example, using scan reports of the
_AMD+GPlay_ dataset re-scanned and downloaded on July 5th, 2019, the optimal
threshold was found to be 12 VirusTotal scanners. So, the labeling strategy
used to label apps in the _Hand-Labeled_ and _Hand-Labeled 2019_ datasets on
July 5th, 2019 is vt$\geq$12. The other type of threshold-based labeling
strategies we use (dashed orange line) depicts strategies that use the optimal
threshold that would achieve the best MCC scores on each of the test datasets.
These thresholds are found by trying different thresholds on the _Hand-
Labeled_ and _Hand-Labeled 2019_ datasets. They are meant to simulate a
scenario in which a researcher always manages to find the thresholds that
yield the best possible MCC scores on the test datasets. Effectively, the
performance of these thresholds depicts an upper bound on the performance of
threshold-based labeling strategies on the test datasets.

The performance of the brute-forced thresholds in Figure 4 and Figure 5
demonstrates the main limitation of the brute forcing approach to finding the
optimal threshold of VirusTotal scanners at a given point in time: in order
for the thresholds obtained by brute-forcing all possible values to generalize
to new Android malware, the dataset used to find these optimal thresholds
needs to be temporally diverse. Given that the _AMD+GPlay_ dataset does not
contain apps developed in 2019, the identified thresholds were high (i.e.,
between 10 and 13 scanners), and could not accurately label apps in the _Hand-
Labeled 2019_ dataset. As for Maat's ML-based labeling strategies, with a few
exceptions, we found that (a) their MCC scores mimicked that of the best
possible threshold at each scan date and (b) had stable MCC scores over the
period of time between July 5th, 2019 and November 8th, 2019. Recall that for
all scan dates, we used ML-based labeling strategies trained using VirusTotal
scan reports that date back to different months in 2018 and using a dataset of
older apps (i.e., _AMD+GPlay_). This means that using VirusTotal scan reports
of older apps, such labeling strategies can maintain decent, stable labeling
accuracies for at least a year even against newer apps.

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest ThresholdsEng GS 2018Eng GS 2019-04-12Eng GS
2019-04-26Eng GS 2019-05-10Eng GS 2019-05-24Eng GS 2019-06-07Eng GS 2019-06-21

(a) Eng GS on _Hand-Labeled_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest ThresholdsNaive GS 2018Naive GS 2019-04-12Naive GS
2019-04-26Naive GS 2019-05-10Naive GS 2019-05-24Naive GS 2019-06-07Naive GS
2019-06-21

(b) Naive GS on _Hand-Labeled_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest ThresholdsEng GS 2018Eng GS 2019-04-12Eng GS
2019-04-26Eng GS 2019-05-10Eng GS 2019-05-24Eng GS 2019-06-07Eng GS 2019-06-21

(c) Eng GS on _Hand-Labeled 2019_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest ThresholdsNaive GS 2018Naive GS 2019-04-12Naive GS
2019-04-26Naive GS 2019-05-10Naive GS 2019-05-24Naive GS 2019-06-07Naive GS
2019-06-21

(d) Naive GS on _Hand-Labeled 2019_

Figure 4. The labeling accuracies achieved by Maat's ML-based labeling
strategies Eng GS and Naive GS trained with scan reports downloaded in between
2018 and June 21st, 2019 against apps in the _Hand-Labeled_ and _Hand-Labeled
2019_ datasets over time.

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest ThresholdsEng Sel GS 2018Eng Sel GS 2019-04-12Eng Sel GS
2019-04-26Eng Sel GS 2019-05-10Eng Sel GS 2019-05-24Eng Sel GS 2019-06-07Eng
Sel GS 2019-06-21

(a) Eng Sel GS on _Hand-Labeled_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest ThresholdsNaive GS 2018Naive Sel GS 2019-04-12Naive Sel
GS 2019-04-26Naive Sel GS 2019-05-10Naive Sel GS 2019-05-24Naive Sel GS
2019-06-07Naive Sel GS 2019-06-21

(b) Naive Sel GS on _Hand-Labeled_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest ThresholdsEng Sel GS 2018Eng Sel GS 2019-04-12Eng Sel GS
2019-04-26Eng Sel GS 2019-05-10Eng Sel GS 2019-05-24Eng Sel GS 2019-06-07Eng
Sel GS 2019-06-21

(c) Eng Sel GS on _Hand-Labeled 2019_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest ThresholdsNaive GS 2018Naive Sel GS 2019-04-12Naive Sel
GS 2019-04-26Naive Sel GS 2019-05-10Naive Sel GS 2019-05-24Naive Sel GS
2019-06-07Naive Sel GS 2019-06-21

(d) Naive Sel GS on _Hand-Labeled 2019_

Figure 5. The labeling accuracies achieved by Maat's ML-based labeling
strategies Eng Sel GS and Naive Sel GS trained with scan reports downloaded in
between 2018 and June 21st, 2019 against apps in the _Hand-Labeled_ and _Hand-
Labeled 2019_ datasets over time.

We made the following observations based on the MCC scores in Figure 4 and
Figure 5. Firstly, with the exception of ML-based labeling strategies trained
using scan reports dating back to 2018, the performance of labeling strategies
using engineered features (i.e., Eng GS and Eng Sel GS), significantly
deteriorates on the _Hand-Labeled 2019_ dataset. However, labeling strategies
trained with Eng Sel GS seem to have higher scores than their counterparts
trained with Eng GS, which is more evident against the _Hand-Labeled 2019_
dataset, as seen in 4(c) versus 5(c). Secondly, in contrast, the MCC scores of
labeling strategies using the naive features appears to be more stable than
their counterparts using engineered features on both test datasets _Hand-
Labeled_ and _Hand-Labeled 2019_. Similar to the case with engineered
features, the selected naive features Naive Sel GS enable the ML-based
labeling strategies to achieve higher MCC scores and maintain almost the same
performance over time regardless of when the labeling strategy was trained.
Lastly, the performance of both types of features seems to fluctuate over
time, albeit not with the same rate or severity. Using selected features seems
to decrease or remove such fluctuations, especially upon using the naive
feature set. To understand the reasons that led to the manifestations of such
behaviors, we studied the structures of the random forests, which constitute
Maat's ML-based labeling strategies, including the features they learn, their
depth, and how they classify apps based on their VirusTotal scan reports.

####  5.1.1. Features Learned by ML-based Labeling Strategies

#####  Engineered Features.

In Figure 6, we show an example of the decision trees trained using engineered
features based on VirusTotal scan reports of the training dataset _AMD+GPlay_
downloaded in 2018. We found that on this date, the decision trees within the
train random forests had almost identical structures and features, whether
trained using the full or selected corpus of the engineered feature set (i.e.,
Eng GS or Eng Sel GS). As seen in the figure, the tree uses a mixture of the
feature categories mentioned in Section 4.4, namely attributes found in the
apps' scan reports (e.g., _positives_ and _total_), features that indicate the
age of an app, and the verdicts of some VirusTotal scan reports (e.g., Babable
and MAX). The tree's structure allows it to accurately classify Android apps
in the _Hand-Labeled_ and _Hand-Labeled 2019_ datasets as follows. First, the
left subtree classifies apps after checking the ratio of _positives_ to
_total_ attributes in the scan report, which allows it to cater to newly-
developed malicious apps. That is if the number of scanners is less than 2
(i.e., _positives_ $\leq$2.5) and the total number of scanners in the scan
reports is less than 24 (i.e., _total_ $\geq$24.5), the tree assumes that the
app is newly-developed yet around 9% of scanners deem it malicious, and in
turn labels the app as malicious. Otherwise, if the total number of scanners
is more 24, the tree considers the app's scan report to be mature (i.e., old
app), yet only a small subset of scanners deem it as malicious; in this case,
the tree considers the _positives_ to be false positives and deems the app as
benign. For our test datasets, we found that this subtree is effective at
classifying benign apps and minimizing false positives.

Figure 6. An example of a decision tree trained using grid search and all
engineered features Eng GS extracted from apps in the _AMD+GPlay_ dataset in
2018.

The tree's right subtree checks the _positives_ attribute again and labels
apps as malicious if more than 16 scanners deem an app malicious. This check
helps identify old malicious apps whose VirusTotal scan reports have matured
to include values of _positives_ higher than these thresholds. As per November
8th, 2019, 99.65% of the malicious apps in the _AMD+GPlay_ dataset and 67% of
those in the _Hand-Labeled_ test dataset had _positives_ values greater than
or equal to 16. However, this check does not help classify newly-developed
malicious apps, such as those in the _Hand-Labeled 2019_ dataset. In this
case, the tree checks the verdict of the Babable scanner. If the scanner deems
an app malicious, it checks the verdict of another scanner (i.e., TrendMicro's
MAX), and deems the app as malicious or benign according to this scanner's
verdict. Effectively, the tree deems the app as malicious if two scanners,
which we found to be among the correct scanners for the _AMD+GPlay_ and _Hand-
Labeled 2019_ datasets in Section 4.2, deem the app malicious. If Babable
finds the app benign, the tree makes a final check about the age of the app
being less than 7.3 years. However, this check can be ignored because,
regardless of the outcome, the tree assigns the app to the majority class in
this subtree, which is malicious.

For apps in our test datasets, we found that benign and malicious apps are
labeled according to the following decision paths. On the one hand, benign
apps are classified using the left subtree after satisfying the conditions
that _positives_ $\leq 2\ AND$ _total_ $\geq 25$. On the other hand, malicious
apps in the _Hand-Labeled_ dataset were labeled malicious after satisfying the
conditions _positives_ $\geq 3\ AND$ _positives_ $\geq 16\ AND$. However,
given their novelty, malicious apps in the _Hand-Labeled 2019_ datasets were
labeled malicious after satisfying the conditions _positives_ $\geq 3\ AND$
Babable $\leq 0.0$. Upon further investigation of the app's scan reports, we
found that they are correctly classified whenever VirusTotal excludes Babable
from their scan reports. Furthermore, we found that the MAX scanner never
detected any of those apps. So, effectively, the apps were labeled as
malicious if the check Babable $==-1$ is true. Although this check is
effective at labeling apps as malicious, it does not make sense to label apps
based on the absence of a scanner. This is an example of how VirusTotal's
first limitation of including and excluding scanners in the scan reports of
apps might impact the way Maat's ML-based trees are trained.

As for the performance of ML-based labeling strategies trained at different
points in time, we found that the main reason behind their underperformance is
their shallowness. We found that–whether they use all of the engineered
features or a selected subset of them–all decision trees in random forests
trained between April 12th, 2019 and June 7th, 2019 employ only one check
making them shallow trees of depth one. With this depth, the tree is unable to
represent apps in the training dataset, which is apparent in the impurity of
the leaf nodes. For example, the left leaf node of 7(a)'s second tree shows
that 55% of the remaining apps are benign, and 45% are malicious; apps will be
labeled as benign because the majority class, in this case, is benign. Later
in this section, we explain why trees of depth one can be trained using either
type of features, viz. engineered and naive.

(a) Using all engineered features

(b) Using selected engineered features

Figure 7. Four randomly selected decision trees in the ML-based labeling
strategies' random forests trained using grid search and engineered features
Eng GS extracted from VirusTotal scan reports of apps in the _AMD+GPlay_
dataset downloaded on April 12th, 2019.

Despite being all of the depth one, some of the decision trees in these random
forests are more effective than others. For example, the decision tree in
Figure 7 uses the verdict of ESET-NOD32 to classify apps, which we showed in
Section 4.2 to be a consistently correct scanner.

Unfortunately, other decision trees trained on April 12th, 2019 using all
engineered features are subjective and fail on their own to represent patterns
found in either the training dataset _AMD+GPlay_ or the test datasets. For
example, the second decision tree (from the left) relies on the _checks-gps_
tag to label apps. This VirusTotal tag does not help discern the malignancy of
an app, especially since using the Global Positioning System (GPS) module is
common among malicious and benign apps alike.

Using a selected subset of the engineered features yields decision trees that
are confident about their labels, as indicated by the Gini values in the leaf
nodes. Checks that did not yield confident labels, such as the VirusTotal tags
and the permissions requested by an app, are excluded. The decision trees rely
on either the verdicts of scanners that we found to be correct in Section 4.2
on the _AMD+GPlay_ dataset or scan report attributes such as _positives_ and
_total_. While selecting features helped increase the performance of ML-based
labeling strategies using engineered features, they still failed to perform
well on the _Hand-Labeled 2019_ dataset. The reason for this is two-fold.
Firstly, relying on the verdicts of only one scanner makes the decision tree
susceptible to VirusTotal's dynamicity and the resulting first limitation.
Regardless of the correctness and stability of the scanner, it risks being
excluded from the scan reports of apps, which forces a decision tree to label
all apps as benign. This explains the fluctuation of Naive GS and Naive Sel GS
on this dataset over time. Secondly, the scan report attributes of _positives_
and _total_ are also susceptible to VirusTotal's dynamicity. More importantly,
using threshold values such as five might suit the older apps in the
_AMD+GPlay_ dataset, where the malicious apps have _positives_ values usually
greater than ten and benign values of zero _positives_. However, such low
values may not generalize to newer apps where only a small subset of
VirusTotal scanners might already recognize their malignancies.

#####  Naive Features.

Using naive features and grid search Naive GS, we found that the random
forests that make up the ML-based labeling strategies had three different
structures and depths. The random forest trained using this type of features
on 2018 had depths of ten. For readability, in Figure 6, we show parts of an
example decision tree that belongs to this random forest. The decision tree in
the figure checks the verdicts of different VirusTotal scanners, some of which
we found to be correct, such as Fortinet and Ikarus, and others that we found
to be hesitant, such as Cyren. However, checking the verdicts of these
different scanners yield pure leaf nodes that are confident about the class of
an app (i.e., Gini index of zero). Although this random forest had decent and
stable MCC scores on both test datasets, one can notice the presence of checks
similar to the Babable example with engineered features, namely checks that
rely on the absence of a scanner's verdict. For example, the fourth node and
its two child nodes check whether the verdicts of the scanners Jiangmin,
VIPRE, and Kingsoft do not exist in the scan report being processed.

(a) Using all naive features

(b) Using selected naive features

Figure 8. Two decision trees trained using grid search and naive features
Naive GS and Naive Sel GS extracted from apps in the _AMD+GPlay_ dataset in
2018.

This type of checks does not exist in decision trees trained using selected
naive features. As seen in 8(b), decision trees in the Naive Sel GS random
forests had depths of four and their structures were different from the ones
trained with all naive features in two aspects. First, they relied on the
verdicts of correct scanners. That is we did not find any decision trees that
rely on the verdicts of VirusTotal scanners, such as GData, Alibaba, or
Kingsoft. Second, they relied on the verdicts of scanners that are included in
the scan reports and checked whether each scanner deemed an app as malicious
or benign rather than whether the scanner's verdicts were absent from the scan
report. The performance of Naive Sel GS ML-based labeling strategies mimicked
that of their Naive GS counterparts on the _Hand-Labeled_ dataset. However, on
the newer _Hand-Labeled 2019_ dataset, that performance decreases in terms of
MCC scores by about 0.1, albeit remaining stable.

The depth of four does not seem to hinder the performance of ML-based labeling
strategies using naive features. As seen in Figure 9, using an average depth
of four, the labeling strategies that used both all and selected naive
features had MCC scores that (a) were stable over time, and (b) matched and in
some cases outperformed those of threshold-based labeling strategies using the
current optimal threshold. The primary difference between random forests
trained using Naive GS and Naive Sel GS was the utilization of correct and
stable VirusTotal scanners. On the one hand, Naive GS decision trees relied on
the verdicts of a mixture of correct scanners (e.g., Sophos, ESET-NOD32, and
NANO-Antivirus) and other less reliable scanners to label apps. On the other
hand, Naive Sel GS exclusively relied on the verdicts of correct and stable
scanners, as identified in Section 4. In both cases, we noticed that the
deeper the decision trees grow, the lower the number of malicious apps that
need to be labeled. For example, in both decision trees in Figure 9, by the
time the last check is performed, less than 1% of the malicious apps remain to
be classified. So, it seems that the checks employed by the decision trees
attempt to identify malicious apps first, and the remaining apps are being
classified as benign.

(a) Using all naive features

(b) Using selected naive features

Figure 9. Two randomly selected decision trees in the ML-based labeling
strategies' random forests trained using grid search and naive features Naive
Sel GS extracted from VirusTotal scan reports of apps in the _AMD+GPlay_
dataset downloaded on April 12th, 2019

As for the performance of both types of ML-based labeling strategies on the
test datasets, we noticed that labeling strategies using Naive Sel GS had
fewer fluctuations in MCC scores over time on both datasets that its Naive GS
counterpart. Moreover, the mediocre performance of ML-based labeling
strategies trained using Naive GS on April 26th, 2019 increases upon using
selected naive features. The reason for that we found is that using selected
naive features (i.e., Naive Sel GS), the depth of the random forests increases
from one to four to resemble the structure of other Naive Sel GS random
forests. We visualize the structure of Naive GS and Naive Sel GS ML-based
labeling strategies trained on this date in Figure 10. On the one hand, random
forests trained using all naive features (seen in 10(a)) result into decision
trees that employ only one check of different VirusTotal scanners. The more
reliable and correct the scanner is (e.g., NANO-Antivirus or
SymantecMobileInsight), the more confident are the labeling decisions.
Decision trees that rely on the verdicts of scanners, such as Baidu, are
likely to yield unconfident labels. On the other hand, using selected naive
features yields random forests with decision trees of an average length of
four, which rely on the verdicts of four scanners, most of which we found to
be correct and stable.

(a) Using all naive features

(b) Using selected naive features

Figure 10. Three randomly selected decision trees in the ML-based labeling
strategies' random forests trained using grid search and naive features
extracted from VirusTotal scan reports of apps in the _AMD+GPlay_ dataset
downloaded on April 26th, 2019. The two trees on the left were trained using
all naive features Naive GS and the tree on the right was trained using
selected naive features Naive Sel GS.

So, we can sum up the results of this section as follows. First, we found that
naive features train ML-based labeling strategies have higher and more stable
MCC scores on both test datasets than their counterparts trained using
engineered features. Second, for both types of features, we found that
selecting the most informative features boosts the labeling accuracies of ML-
based labeling strategies and relatively smoothens the fluctuations of their
MCC scores over time. Third, in associating the labeling accuracies with the
structure of decision trees in the random forests constituting ML-based
labeling strategies, we found that decision trees that rely on the verdicts of
between three and four VirusTotal scanners, which we found to be correct and
stable on the training dataset _AMD+GPlay_ , yields the best and most stable
MCC scores. In the next section, we attempt to explain what controls the
structure of the trained random forests and the role that VirusTotal's
dynamicity plays in this.

####  5.1.2. Sensitivity to VirusTotal's Dynamicity

In Section 4.1, we mentioned that we use the technique of grid search to find
the hyperparameters that yield the most accurate random forests that
constitute Maat's ML-based labeling strategies. One of those hyperparameters
controls the maximum depth each decision tree in the random forest is allowed
to grow (i.e., _max_depth_). We varied the value of _max_depth_ to be one,
four, ten, and None, and allowed the technique of grid to identify the value
that yields the best validation accuracy ($\frac{TP+TN}{P+N}$) achieved using
the technique of cross-validation, which we set to be ten-fold. In our case,
this accuracy is calculated by splitting the scan reports of apps in the
_AMD+GPlay_ dataset into ten folds, train the random forests using nine of
those, and using the remaining one-tenth as validation dataset. The final
validation accuracy is an average of all ten accuracies achieved on the ten
validation datasets.

We found that the validation accuracies achieved by random forests of
different depths are very close to one another. For example, for Naive GS ML-
based labeling strategies trained on April 26th, 2019, we found that the
validation accuracies achieved by random forests with _max_depth_ values of
one, four, ten, and None were 1.0, 0.9999794703346335, 0.9999794703346335, and
0.9999589490951507, respectively. Based on those values and despite the
insignificant difference in validation accuracies, the grid search algorithm
suggested the random forests with _max_depth_ of one as the best random
forest. So, what causes this small difference? The difference between both
values is 0.00002053. This means that out of one tenth of the total number of
apps in the _AMD+GPlay_ datasets (i.e., $48,715\times 0.1=4,871$), an average
of 0.10 (i.e., $4,871\times 0.00002053=0.100011895$) apps gets misclassified
in the validation dataset. Since this value is an average, we can assume that
at some validation folds, a few apps out of 4,871 are misclassified.

In this case of Naive GS ML-based labeling strategies, since they rely on the
verdicts given by VirusTotal scanners, the frequent change in the verdicts of
scanners (i.e., VirusTotal's first limitation), is expected to have an impact
on the feature vectors used to train the labeling strategies' random forests.
In fact, we found that between April 12th, 2019, and April 26th, 2019 only 15%
of the apps in the _AMD+GPlay_ dataset had the exact same verdicts.
Furthermore, between these two dates, almost 85% of the apps had at least one
verdict change, 51.65% hat at least two verdicts change, and 23.4% had at
least three verdicts change. Examining these differences between all other
training dates (i.e., between 2018 and June 21st, 2019), yielded very similar
percentages. Nonetheless, these percentages do not reveal which verdicts have
been changed. So, the likely scenario on April 26th, 2019 is that the verdicts
in the VirusTotal scan reports of apps in the _AMD+GPlay_ dataset changed in a
manner that gave a slight edge to random forests with _max_depth_ of one
enough to make them be chosen as the best random forests. In that sense, the
dynamicity of VirusTotal can impact the performance of Maat's ML-based
labeling strategies' random forests yielding ones with shallow decision trees
that perform well on the training dataset, yet fail to generalize to test
datasets.

Another aspect of the ML-based labeling strategies' sensitivity to
VirusTotal's dynamicity is related to the platform's third limitation, namely
that it sometimes utilizes inadequate scanners and versions of scanners that
are not designed to cater to Android malware. In particular, this limitation
affected Naive Sel GS, which relies on the verdicts of scanners to label apps.
As seen in Section 4.2, some scanners, such as BitDefender, had better
labeling accuracy in 2018 before their versions were altered by VirusTotal.
Consequently, Naive Sel GS labeling strategies trained using 2018 versions of
apps in the _AMD+GPlay_ dataset might include scanners that were correct and
stable back then (e.g., Trustlook and Avira). These scanners ceased to be
correct and stable and, hence, were not able to help label apps accurately in
the test datasets according to their scan reports downloaded at different
points in time in 2019.

###  5.2. Enhancing Detection Methods

The second criterion we use to evaluate the applicability of Maat's ML-based
labeling strategies is their ability to contribute to training more effective
ML-based detection methods. In Section 1, we discussed that inaccurate
labeling might have a negative impact on the detection performance on
potentially effective detection methods. By addressing this issue of labeling
accuracy, we can contribute to helping the research community focus on
developing effective detection methods rather than being consumed by devising
labeling strategies that accurately label apps in their training datasets. Our
main hypothesis in this section is that accurate labeling should, in turn,
lead to more effective detection of out-of-sample apps. So, since Maat's ML-
based labeling strategies managed to accurately label apps in the test
datasets over a period of time, we hypothesize that they can accurately label
apps in the _AndroZoo_ dataset and lead to training ML-based detection methods
that accurately label apps in those test datasets, viz.  _Hand-Labeled_ , and
_Hand-Labeled 2019_ datasets.

We focus on ML-based detection methods given their popularity within the
academic community (Pendlebury et al., 2018; Suarez-Tangil et al., 2017; Tam
et al., 2017; Yang et al., 2017; Arshad et al., 2016). In essence, we train
different ML classifiers using static features extracted from the _AndroZoo_
dataset, which depict information about the apps' components (Sato et al.,
2013; Sanz et al., 2013), the permissions they request (Arp et al., 2014; Wu
et al., 2012), the API calls found in their codebases (Sachdeva et al., 2018;
Zhou et al., 2013), and the compilers used to compile them (Stazzere, 2016).
The feature vectors are then labeled using different threshold-based and ML-
based labeling strategies. We test the detection abilities of such classifiers
by assessing their ability to label apps in the _Hand-Labeled_ , and _Hand-
Labeled 2019_ datasets correctly. For readability, we use the shorter term
_classifier was labeled_ instead of _classifier whose feature vectors were
labeled_.

There is a plethora of approaches to using static features and ML algorithms
to detect Android malware. We utilize a detection method that is renowned in
the research community and has been used by different researchers as a
benchmark (Feargus Pendlebury and Cavallaro, 2019), namely _Drebin_ (Arp et
al., 2014). The _Drebin_ approach comprises three components: a linear support
vector machine _LinearSVC_ to classify apps, a thorough set of static features
extracted from Android apps that spans all app components, permissions, URL's,
etc., and the drebin labeling strategy. Using an implementation of _Drebin_ 's
feature extraction algorithm, we extracted a total of 71,260 features from
apps in the _AndroZoo_ , _Hand-Labeled_ , _Hand-Labeled 2019_ datasets. In
addition to _Drebin_ , we use the following classifiers: _K-Nearest Neighbors
(KNN)_ (Sanz et al., 2012), _Random Forest (RF)_ (Sanz et al., 2013), _Support
Vector Machine (SVM)_ (Arp et al., 2014), and _Gaussian Naive Bayes (GNB)_
111111Gaussian naive Bayes classifiers assume that features have a Gaussian
distribution. We argue that the static features we extract from the apps
indeed follow such distribution. For example, while a small number of apps
request either a very small or a considerable amount of requests, the majority
of apps request an adequate amount (e.g., in the lower tens). (Sanz et al.,
2013). Given that the KNN and RF classifiers can have different values for
their hyperparameters, we used the technique of grid search to identify the
classifiers that yielded the best validation accuracies and used them as
representatives of those classifiers. We also varied the values of the primary
hyperparameters of the KNN and RF classifiers as follows; we varied the number
of nearest neighbors to consider by the KNN classifier to
$K=\\{11,26,51,101\\}$ and the number of decisions trees in the random forest
to $\\{25,50,75,100\\}$. Consequently, we refer to these to classifiers as KNN
and RF. To train these classifiers, we statically extracted numerical features
from the APK archives of apps in the _AndroZoo_ , which depict information
about the apps' components (Sato et al., 2013; Sanz et al., 2013), the
permissions they request (Arp et al., 2014; Wu et al., 2012), the API calls
found in their codebases (Sachdeva et al., 2018; Zhou et al., 2013), and the
compilers used to compile them (Stazzere, 2016).

The labeling strategies we consider in this experiment are the conventional
threshold-based strategies of vt$\geq$1, vt$\geq$4, vt$\geq$10, vt$\geq$50%,
and drebin, threshold-based labeling strategies that use threshold brute-
forced using at each point in time, threshold-based labeling strategies that
use the best threshold at each point in time, and Maat's ML-based labeling
strategies that had the highest MCC scores in the previous experiment, namely
Naive GS and Naive Sel GS. In assessing the performance of different ML-based
detection methods, we are not concerned with absolute values that indicate the
quality of the classifier and the features it is trained with. Instead, we are
interested in their performance with respect to one another. Given that the
main difference between such classifiers is that their training feature
vectors were labeled using different labeling strategies, this experiment is
concerned with verifying the hypothesis of _more accurate labeling leads to
better detection_.

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest
Thresholdsvt$\displaystyle\geq$1vt$\displaystyle\geq$4vt$\displaystyle\geq$10vt$\displaystyle\geq$50%drebinNaive
GS 2018Naive GS 2019-04-12Naive Sel GS 2018Naive Sel GS 2019-04-12

(a) _Hand-Labeled_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest
Thresholdsvt$\displaystyle\geq$1vt$\displaystyle\geq$4vt$\displaystyle\geq$10vt$\displaystyle\geq$50%drebinNaive
GS 2018Naive GS 2019-04-12Naive Sel GS 2018Naive Sel GS 2019-04-12

(b) _Hand-Labeled 2019_

Figure 11. The MCC scores achieved by the _Drebin_ classifiers labeled using
different threshold-/ML-based labeling strategies against the _Hand-Labeled_
and _Hand-Labeled 2019_ dataset between July 5th, 2019 and November 8th, 2019.

In Figure 11 we plot the classification performance of different _Drebin_
classifiers against apps in the _Hand-Labeled_ and _Hand-Labeled 2019_
datasets in terms of MCC score. The feature vectors used to train each of
those classifiers have been labeled using a different labeling strategy. Given
that the performances of different classifiers are intertwined, we tabulate
their MCC scores in Table 3 for better readability. Studying these
performances, we made the following observations.

The first observation we made was that the performance of _Drebin_ classifiers
is better on the _Hand-Labeled 2019_ dataset than on the older _Hand-Labeled_
dataset. The reason behind this is that apps in the _AndroZoo_ dataset were
developed between 2018 and 2019. So, features extracted from these apps are
expected to be similar to apps in the _Hand-Labeled 2019_ dataset than to
those in the _Hand-Labeled_ dataset, especially since the _Drebin_ feature set
is designed to identify similarity in the components and features utilized by
the Android apps in the training dataset. This proximity in feature space
helps _Drebin_ 's Linear Support Vector Machine (SVC) classifier identify
patterns shared by malicious and benign apps better.

Table 3. Detailed view of the MCC scores achieved by the _Drebin_ classifiers
labeled using different threshold-/ML-based labeling strategies against the
_Hand-Labeled_ and _Hand-Labeled 2019_ dataset between July 5th, 2019 and
November 8th, 2019.

Labeling Strategy Scan Date |  2019-07-05 |  2019-07-19 |  2019-08-02 |  2019-08-16 |  2019-08-30 |  2019-09-13 |  2019-09-27 |  2019-10-11 |  2019-10-25 |  2019-11-08
---|---|---|---|---|---|---|---|---|---|---
[HTML]C0C0C0 _Hand-Labeled_
Brute-forced Thresholds | 0.00 | 0.00 | 0.00 | 0.18 | 0.24 | 0.18 | 0.00 | 0.00 | 0.00 | 0.18
Best Thresholds | 0.28 | 0.34 | 0.43 | 0.44 | 0.38 | 0.24 | 0.22 | 0.50 | 0.56 | 0.50
vt$\geq$1 | 0.21 | 0.24 | 0.15 | 0.28 | 0.31 | 0.27 | 0.35 | 0.25 | 0.31 | 0.31
vt$\geq$4 | 0.26 | 0.22 | 0.32 | 0.43 | 0.38 | 0.24 | 0.22 | 0.24 | 0.39 | 0.36
vt$\geq$10 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
vt$\geq$50% | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
drebin | 0.00 | 0.08 | 0.08 | 0.00 | 0.08 | 0.08 | 0.28 | 0.00 | 0.15 | 0.05
Naive GS 2018 | 0.59 | 0.41 | 0.53 | 0.55 | 0.61 | 0.48 | 0.51 | 0.36 | 0.55 | 0.55
Naive GS 2019-04-12 | 0.27 | 0.29 | 0.39 | 0.25 | 0.20 | 0.30 | 0.38 | 0.33 | 0.49 | 0.26
Naive Sel GS 2018 | 0.54 | 0.51 | 0.60 | 0.55 | 0.59 | 0.54 | 0.72 | 0.38 | 0.55 | 0.55
Naive Sel GS 2019-04-12 | 0.50 | 0.47 | 0.48 | 0.47 | 0.53 | 0.52 | 0.48 | 0.43 | 0.58 | 0.47
[HTML]C0C0C0 _Hand-Labeled 2019_
Brute-forced Thresholds | 0.30 | 0.00 | 0.30 | 0.19 | 0.33 | 0.33 | 0.30 | 0.27 | 0.53 | 0.43
Best Thresholds | 0.73 | 0.67 | 0.73 | 0.80 | 0.67 | 0.65 | 0.70 | 0.66 | 0.73 | 0.76
vt$\geq$1 | 0.18 | 0.18 | 0.19 | 0.18 | 0.24 | 0.26 | 0.34 | 0.25 | 0.40 | 0.37
vt$\geq$4 | 0.73 | 0.66 | 0.80 | 0.73 | 0.76 | 0.70 | 0.73 | 0.78 | 0.80 | 0.73
vt$\geq$10 | 0.53 | 0.23 | 0.43 | 0.38 | 0.33 | 0.33 | 0.43 | 0.27 | 0.48 | 0.48
vt$\geq$50% | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
drebin | 0.23 | 0.00 | 0.10 | 0.10 | 0.02 | 0.04 | 0.14 | 0.19 | 0.23 | 0.19
Naive GS 2018 | 0.84 | 0.76 | 0.73 | 0.63 | 0.80 | 0.76 | 0.70 | 0.67 | 0.80 | 0.78
Naive GS 2019-04-12 | 0.84 | 0.78 | 0.80 | 0.78 | 0.78 | 0.84 | 0.71 | 0.67 | 0.80 | 0.82
Naive Sel GS 2018 | 0.71 | 0.69 | 0.63 | 0.69 | 0.71 | 0.69 | 0.69 | 0.71 | 0.69 | 0.62
Naive Sel GS 2019-04-12 | 0.76 | 0.80 | 0.80 | 0.76 | 0.76 | 0.76 | 0.73 | 0.80 | 0.73 | 0.84

Secondly, we found that using best threshold at each point on time along with
Maat's ML-based labeling strategies usually helps the _Drebin_ classifier
achieve better MCC scores than threshold-based labeling strategies that use
fixed thresholds over time (e.g., vt$\geq$1, vt$\geq$10, and vt$\geq$50%).
Moreover, the average MCC scores of _Drebin_ classifiers labeled using Maat's
ML-based labeling strategies is higher than that of classifiers labeled using
the best thresholds at each scan date. This observation coincides with our
hypothesis that accurate labeling strategies, by and large, contribute to
training more effective ML-based detection methods. However, we noticed a
number of exceptions that contradict this hypothesis. For example, the Naive
GS ML-based labeling strategies trained in 2018 and the Naive GS strategies
trained on April 12th, 2019 had almost the same labeling accuracy against apps
in the _Hand-Labeled_ dataset. Despite this similarity, the _Drebin_
classifiers labeled using the latter strategy noticeably underperforms in
comparison to the former one. This behavior switches on the _Hand-Labeled
2019_ dataset: the performance of _Drebin_ classifiers labeled using Naive GS
2018 almost mimics that of those labeled using Naive GS trained on April 12th,
2019, despite the former being less able to label apps in this dataset
accurately.

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest
Thresholdsvt$\displaystyle\geq$1vt$\displaystyle\geq$4vt$\displaystyle\geq$10vt$\displaystyle\geq$50%drebinNaive
GS 2018Naive GS 2019-04-12Naive Sel GS 2018Naive Sel GS 2019-04-12

(a) KNN with _Hand-Labeled_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest
Thresholdsvt$\displaystyle\geq$1vt$\displaystyle\geq$4vt$\displaystyle\geq$10vt$\displaystyle\geq$50%drebinNaive
GS 2018Naive GS 2019-04-12Naive Sel GS 2018Naive Sel GS 2019-04-12

(b) KNN with _Hand-Labeled 2019_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest
Thresholdsvt$\displaystyle\geq$1vt$\displaystyle\geq$4vt$\displaystyle\geq$10vt$\displaystyle\geq$50%drebinNaive
GS 2018Naive GS 2019-04-12Naive Sel GS 2018Naive Sel GS 2019-04-12

(c) RF with _Hand-Labeled_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest
Thresholdsvt$\displaystyle\geq$1vt$\displaystyle\geq$4vt$\displaystyle\geq$10vt$\displaystyle\geq$50%drebinNaive
GS 2018Naive GS 2019-04-12Naive Sel GS 2018Naive Sel GS 2019-04-12

(d) RF with _Hand-Labeled 2019_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest
Thresholdsvt$\displaystyle\geq$1vt$\displaystyle\geq$4vt$\displaystyle\geq$10vt$\displaystyle\geq$50%drebinNaive
GS 2018Naive GS 2019-04-12Naive Sel GS 2018Naive Sel GS 2019-04-12

(e) GNB with _Hand-Labeled_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresBrute-
forced ThresholdsBest
Thresholdsvt$\displaystyle\geq$1vt$\displaystyle\geq$4vt$\displaystyle\geq$10vt$\displaystyle\geq$50%drebinNaive
GS 2018Naive GS 2019-04-12Naive Sel GS 2018Naive Sel GS 2019-04-12

(f) GNB with _Hand-Labeled 2019_

Figure 12. The MCC scores achieved by the KNN, RF, and GNB classifiers labeled
using different threshold-/ML-based labeling strategies against the _Hand-
Labeled_ and _Hand-Labeled 2019_ dataset between July 5th, 2019 and November
8th, 2019.

The third observation depicts another counter-example to the hypothesis that
related accurate labeling to effective detection. Although it falls within the
ranges of best thresholds identified at each scan date, using a threshold of
four VirusTotal scanners to label the _Drebin_ classifier (i.e., using
vt$\geq$4), yields scores that do not mimic the performance of the _Drebin_
classifiers using the current best thresholds. For example, while using any
threshold between three and six on July 5th, 2019 resulted in the same MCC
scores against apps in the _Hand-Labeled_ dataset, on the same scan date and
using the same test datasets, training a _Drebin_ classifier using a threshold
of three scanners yields a different MCC score than a classifier using a
threshold of four scanners, namely 0.28 and 0.26, respectively.

In general, we noticed that MCC score of each _Drebin_ classifier differs
depending on the utilized labeling strategy and the scan date of the
VirusTotal scan reports this strategy uses to label apps. In addition to these
two factors, we noticed that the utilized classifier also impacts these
scores. Figure 12 shows the MCC scores achieved by the KNN, RF, and GNB
classifiers against the same test datasets using the same labeling strategies.
Similar to the _Drebin_ classifier, the MCC scores achieved by these three
classifiers show that Maat's ML-based labeling strategies contribute to
training more effective detection methods than their threshold-based
counterparts, in spite of using a different set of features to represent the
apps in the training and test datasets. However, unlike the _Drebin_
classifier, the MCC scores of the KNN, RF, and GNB classifiers seem to be
stable across different scan dates, especially on the _Hand-Labeled 2019_
dataset. The values of these scores seem to differ from one classifier to
another. On the _Hand-Labeled 2019_ dataset, for instance, the average MCC
scores of the KNN, RF, and GNB classifiers were 0.64, 0.89, and 0.43,
respectively. So, it seems that the features used to represent apps in the
training dataset and the type of ML classifier also impact the performance of
a detection method.

To sum up, we found that accurately labeling apps based on their VirusTotal
scan reports does not guarantee training more effective detection methods.
Other factors, such as the utilized feature set and the date on which the
training and test apps were scanned by VirusTotal also impact the
effectiveness of such methods.

##  6\. Discussion

In this section, we discuss the insights we gained from our experiments, the
limitations of our work, and how we (plan to) address them.

#####  Optimally Using VirusTotal To Label Apps.

In Section 1, we mentioned that the main objective of this paper is to provide
the research community with insights about how to best interpret the raw
information given by VirusTotal to accurately label Android apps. So, we
studied the performance of two types of labeling strategies, viz. threshold-
based labeling strategies and a representative of scanner-based labeling
strategies that relies on ML algorithms trained by our framework Maat.
Threshold-based labeling strategies that rely on fixed thresholds suffer from
two problems. First, they are subjective as their thresholds reflect the
subjectivity of defining apps as malicious in terms of the number of scanners
deeming it as such. Second, they are susceptible to the dynamicity of
VirusTotal and its frequent manipulation of scan reports of apps, especially
newly-developed ones. So, the values of thresholds used by this type of
labeling strategies have to be frequently updated to cope with the online
platform's dynamicity, rather than being fixed and used for extended periods
of time, which can be an infeasible process. We implemented Maat to train ML-
based labeling strategies that are meant to mitigate the aforementioned two
problems facing their threshold-based counterparts. While they showed slight
sensitivity to VirusTotal's dynamicity, Maat's ML-based labeling strategies
exhibited steady labeling accuracies over time that mimic that of threshold-
based labeling strategies that use the best thresholds at every point in time.
Since they are trained at one point in time yet are able to maintain decent
labeling accuracies and contribute to more effective ML-based detection
methods, such ML-based labeling strategies have the potential to replace or
complement threshold-based labeling strategies. So, despite being fast and
intuitive, conventional threshold-based labeling strategies that rely on a
fixed threshold over time are not able to cope with VirusTotal's dynamicity.
Instead, the current threshold values that yield the best labeling accuracies
have to be re-calculated on a frequent basis. The easiest method to identify
those values is to measure the accuracy of thresholds between one and 60
against a diverse dataset of pre-labeled Android apps. While this process
seems to be theoretically feasible, it requires the satisfaction of different
conditions that might be cumbersome or even infeasible. The ability of Maat's
to automatically train ML-based labeling strategies that maintain labeling
accuracies that match those of the best possible threshold-based labeling
strategies over time suggests that they can use utilized to complement or
replace threshold-based strategies altogether.

#####  The Role of Accurate Labeling in Malware Detection.

We argued in Section 1 that accurate labeling of apps is fundamental to
training and evaluating effective malware detection methods. So, our main
hypothesis vis-à-vis labeling is that the more accurate the apps are being
labeled, the more effective the malware detection methods that use them as
training apps. In Section 5.2, we found that Maat's ML-based labeling
strategies that use the verdicts of VirusTotal scanners to label apps (i.e.,
naive features), and threshold-based labeling strategies that use the best
current thresholds contribute to training ML-based detection methods that are
more effective than those labeled using conventional threshold-based
strategies that rely on fixed thresholds (e.g., vt$\geq$1 or vt$\geq$50%).
Nevertheless, we did not find evidence that accurate labeling of feature
vectors is synonymous with better detection accuracy. That is, in some cases,
labeling strategies that performed worse at labeling apps based on their
VirusTotal scan reports contributed to training ML-based detection methods
that perform better than other labeling strategies that performed better at
labeling apps. We argue that such occasional discrepancies are due to the
performance of ML-based detection methods being subject to the chosen
classifier and the utilized features extracted from the training apps. In this
context, we found that the more accurate a labeling strategy is at labeling
apps based on their VirusTotal scan reports, the more likely it is going to
contribute to training more effective ML-based detection methods. The type of
features extracted from malicious and benign apps and the technique adopted by
the chosen ML classifier dictate how the classifier will segregate the feature
vectors of both types of apps. The position in the dimensional space of out-
of-sample feature vectors relative to the decision boundary learned by the ML
classifier also plays a vital role in the classifier's performance against
test datasets. These issues might have contributed to the few exceptions to
the aforementioned general rule.

#####  VirusTotal's Limitations.

Despite calls within the research community to replace VirusTotal with a more
reliable alternative, the online platform continues to be utilized by
researchers to label apps in the datasets they use to evaluate their malware
detection methods. These calls for replacement stem from the common knowledge
that platforms, such as VirusTotal, suffer from some drawbacks. However, these
drawbacks and their impact on the process of labeling apps were neither
thoroughly discussed nor demonstrated. Throughout this paper, we discussed
some of those limitations and their impacts on identifying the set of correct
scanners, estimating the time it takes scan reports to stabilize, and the
performance of different labeling strategies. Using the insights we gained
from our measurements and experiments, we identified the following limitations
of VirusTotal that jeopardizes its usefulness. Firstly, the platform does not
automatically re-scan apps and relies on manually re-scanning apps either via
its web-interface or via remote API requests. Secondly, the platform uses
scanners or versions of scanners that are not suitable to detect Android
malware. Thirdly, for reasons unknown to us, the platform changes the set of
scanners it uses to scan the same apps over time, which undermines the
sustainability of labeling strategies, such as threshold-based ones. Lastly,
the platform does not grant access to the history of scans, effectively
preventing researchers from studying the performance of scanners over extended
periods of time.

###  6.1. Limitations and Threats to Validity

##### Partial Reliance on VirusTotal's Ground Truths.

To label apps in the _AMD+GPlay_ dataset, we relied on the labels generated by
Wei et al. to label apps in the _AMD_ dataset, which combined filtration of
malicious using the vt$\geq$50% labeling strategy and manual analysis to
accurately label apps in the dataset as malicious. We also used the VirusTotal
scan reports of apps in the _GPlay_ dataset, which were downloaded from the
well-vetted Google Play store, to deem them as benign according to the
criterion _positives==0_ between 2018 and September 27th, 2019. The threat to
validity in this case is whether those measures we took to ensure the accuracy
of the labels in this dataset were not enough. Since we rely on the
_AMD+GPlay_ dataset to train Maat's ML-based labeling strategies, inaccuracies
in the labels of those apps threatens to undermine the credibility of our
findings. To ensure the credibility of our results, we plan on manually-
labeling as many Android apps as possible, use them to train Maat's ML-based
labeling strategies, and compare the results we achieve with the results in
this paper. Needless to say, manually analyzing thousands of apps is a lengthy
process.

##### Confinement to Used Datasets and Scan Reports.

The results we recorded and discussed in this paper are confined to the
datasets that we used. So, the threat to validity, in this case, is whether
the same conclusions we drew can be reached upon using other datasets (e.g.,
that Maat's ML-based labeling strategies are more consistent that their
threshold-based counterparts). In this paper, we attempted to assess the
reproducibility of our results. However, the experiments were conducted on the
same datasets, and the time period between the two sets of experiments was
less than two months. So, in the future, we plan to run our measurements and
experiments against different datasets to verify whether our results are
reproducible on other datasets.

##### Access to older scan reports.

The results we present in this paper, especially those vis-à-vis the
correctness and stability of VirusTotal scanners, were drawn from scan reports
gathered between 2018 and November 8th, 2019 because access to older scan
reports is only available under commercial licenses. So, we are not able to
capture the complete history of apps in the _AMD+GPlay_ and how they evolved.
In Section 4.2, we demonstrated that the time period during which the
VirusTotal scan reports were gathered alters the set of correct scanners. On
the one hand, this might affect the engineered features Maat extracts from the
scan reports, which we found to be less informative that their naive
counterparts. On the other hand, training ML-based labeling strategies using
naive features relies on the latest scan reports of apps in Maat's training
dataset (i.e., September 27th, 2019 and November 8th, 2019). Furthermore, apps
in the test datasets are also labeled according to their latest scan reports.
So, while lack of access to older scan report might slightly change the
results of our measurements in Section 4.2 and Section 4.3, it does not impact
our findings in Section 5.1 and Section 5.2, viz. that Maat ML-based labeling
strategies are more consistently accurate than their threshold-based
counterparts.

##### Generalization to other detection methods.

To further motivate the significance of our work, we showed that accurate
labeling generally enhances the performance of ML-based detection methods
trained using static features. This begs the question of whether we can
replicate the same results if we utilize different features and/or different
classifiers. The most appropriate method to verify this hypothesis is to
conduct the same experiments using different types of features and classifiers
in pursuit of counter-examples. However, there is a plethora of work in this
area that spans a multitude of approaches to malware analysis and detection
(Tam et al., 2017), which indeed cannot be comprehensively covered in this
work. So, we plan to conduct more experiments using different types of
features (e.g., dynamic features), and/or classifiers to further solidify our
findings.

##  7\. Related Work

#####  Studying VirusTotal.

Given the significant role it plays in malware analysis and detection process,
the research community has studied different aspects of VirusTotal and its
scanners. In (Mohaisen et al., 2013), Mohaisen et al. inspected the relative
performance of VirusTotal scanners on a small sample of manually-inspected and
labeled Windows executables. The authors introduced four criteria, called
correctness, completeness, coverage, and consistency, to assess the labeling
capabilities of VirusTotal scanners and demonstrated the danger of relying on
VirusTotal scanners that do not meet such criteria. The main objective of this
study is, therefore, to shed light on the inconsistencies among VirusTotal
scanners on a small dataset. In (Mohaisen and Alrawi, 2014), Mohaisen and
Alrawi built on their previous study and attempted to assess the detection
rate, the correctness of reported labels, and the consistency of detection of
VirusTotal scanners according to the aforementioned four criteria. They showed
that in order to obtain complete and correct (i.e., in comparison to ground
truth) labels from VirusTotal, one needs to utilize multiple independent
scanners instead of hinging on one or a few of them. Similarly, within the
domain of Android malware, Hurier et al. studied the scan reports of
VirusTotal scanners to identify the lack of consistency in labels assigned to
the same app by different scanners and proposed metrics to quantitatively
describe such inconsistencies (Hurier et al., 2016). More recently, Peng et
al. (Peng et al., 2019) showed that VirusTotal scanners exhibit similar
inconsistencies upon deeming Uniform Resource Locator (URL) as malicious and
benign. The authors also showed that some VirusTotal scanners are more correct
than others, which requires a strategy to label such URL s that does not treat
all scanners equally. In Section 4, we discussed how our method Maat analyzes
VirusTotal scan reports to identify correct and stable scanners as part of
extracting engineered features from such reports. This process indeed builds
on the insights in (Hurier et al., 2016; Mohaisen and Alrawi, 2014; Mohaisen
et al., 2013). In fact, we utilize the _correctness_ score introduced in
(Mohaisen et al., 2013) to assess the correctness of VirusTotal scanners over
time. However, our work is different from the aforementioned work in terms of
objectives. While the objective of (Peng et al., 2019; Hurier et al., 2016;
Mohaisen and Alrawi, 2014; Mohaisen et al., 2013) is to shed light on the
dynamicity of VirusTotal, our work attempts to take a step further by
providing the research community by explaining the potential reasons behind
VirusTotal's dynamicity and how they impact conventional threshold-based
labeling strategies.

#####  Label Unification.

The lack of universal standards to label and name malicious apps allows
different antiviral firms to give different labels to the same malicious app
(Maggi et al., 2011; Kelchner, 2010). For example, the same malware can have
the labels Worm:W32/Downadup.gen!A, Net-Worm.Win32.Kido.cp, W32/
Conficker.worm.gen.a, Worm:Win32/Conficker.gen!B, Worm-Win32/Conficker.gen!A,
which all share the case insensitive substring _worm_ (Kelchner, 2010). So,
considering the problem to be that of string manipulation, one of the main
objectives of academic research has been to devise methods to unify those
strings into one that still represents the malware type and family of a
malicious app. In (Perdisci et al., 2012), Perdisci et al. did not implement a
method to unify different antiviral labels, rather an automated approach,
_VAMO_ , that assesses methods that clusters similar labels together prior to
unifying them. This method can help eliminate noisy labels, which is a
prerequisite for accurate unification of labels. Wang et al. also studied the
malware naming discrepancies via analyzing the scan results in (Wang et al.,
2014), and identified two types of such discrepancies, viz. syntactic and
semantic. The authors implemented an approach, _Latin_ , that considers these
two types of discrepancies towards devising a consensus classification of
antiviral labels that can be used to look up information about malicious apps
in repositories such as _Anubis_. The work in (Perdisci et al., 2012) and
(Wang et al., 2014) did not present tools that output unified labels. In
(Sebastián et al., 2016), Sebastian et al. presented a fully-automatic, cross-
platform tool, _AVCLASS_ , that examines different labels given to the same
app by different scanners, and returns the most likely family name that such
app should assume. Similarly, Hurier et al. developed a tool, _Euphony_ , that
mines labels, analyzes the associations between them, and attempts to unify
them into common family groups (Hurier et al., 2017). In this paper, we
attempt to devise one label for malicious apps based on their VirusTotal scan
reports, which include multiple labels given by different antiviral scanners.
However, we do not attempt to give a label that indicates the malware family
or type of apps. Instead, we attempt to devise labels that indicate whether an
app is malicious or benign (i.e., a label that discerns the malignancy of the
app).

#####  Discerning Malignancy.

A more abstract form of labeling apps is to label them as malicious or benign.
Apart from threshold-based labeling strategies, researchers have devised more
sophisticated labeling strategies, primarily based on ML. In (Kantchelian et
al., 2015), Kantchelian et al. used the VirusTotal scan reports of around 280K
binaries to build two ML-based techniques to aggregate the results of multiple
scanners into a single ground-truth label for every binary. In the first
technique, Kantchelian et al. assume that the ground truth of an app (i.e.,
malicious or benign), is unknown or _hidden_ , making the problem of
estimating this ground truth is that of unsupervised learning. Furthermore,
they assumed that the verdicts of more consistent, less erratic scanners are
more likely to be correlated with the correct, hidden ground truth than more
erratic scanners. Thus, more consistent scanners should have larger weights
associated with their verdicts. To estimate those weights and, hence, devise
an unsupervised ML-based labeling strategy, the authors used an Expectation
Maximization (EM) algorithm based on a Bayesian model to estimate those
models. The second technique devised by Kantchelian et al. is a supervised one
based on regularized logistic regression. However, the authors did not
describe the nature of the features they use to train such an algorithm. So,
we assume that they relied on the verdicts of antiviral scanners in a manner
similar to the naive features we extracted from VirusTotal scan reports, as
discussed in Section 4. To devise an automated method to label apps based on
different verdicts given by antiviral scanners, Sachdeva et al. (Sachdeva et
al., 2018) performed measurements to determine the most correct VirusTotal
scanners using scan reports of a total of 5K malicious and benign apps. Using
this information, they assign a weight to each scanner that they use to
calculate a malignancy score for apps based on their VirusTotal scan reports.
Depending on manually-defined thresholds, the authors use this score to assign
a confidence level of Safe, Suspicious, or Highly Suspicious to test apps. The
main difference between our work and (Kantchelian et al., 2015) and (Sachdeva
et al., 2018) is actionability. That is to say, despite reporting decent
labeling accuracies, both efforts did not evaluate whether the newly-devised
labeling method can help build effective detection methods and left that for
future work: "Improved training labels, as obtained by the techniques
presented in this paper, should result in an improved malware detector."
(Kantchelian et al., 2015). In this paper, however, we attempted to evaluate
the applicability of our method by examining the classification accuracies
detection methods can achieve using labels predicted by the most accurate
threshold-/ML-based labeling strategies.

##  8\. Conclusion

The infeasibility of manually analyzing and labeling Android apps forces the
research community to use VirusTotal scan reports to label those apps, despite
proving to be a dynamic online platform. We argue that the solution to
VirusTotal's dynamicity is to replace it with a more stable platform. However,
until either VirusTotal addresses its limitations or an alternative platform
is developed and tested, the research community is expected to use VirusTotal
despite its known limitations. So, the main objective of this paper was to
provide insights to the research community about how optimally utilize
VirusTotal to scan apps based on their scan reports.

We started by discussing threshold-based labeling strategies that are widely-
adopted by researchers, and showed their sensitivity to VirusTotal's
dynamicity, which forces researchers to identify the optimal thresholds to use
every time they wish to label their apps. To tackle the limitations of
threshold-based labeling strategies and as a step towards standardizing the
utilization of VirusTotal scan reports, we developed a method, Maat, that
automates the process of analyzing VirusTotal scan reports gathered over a
period of time to devise ML-based labeling strategies that accurately label
apps and, in turn, contribute to more effective malware detection. To train
such ML-based strategies, we analyzed the VirusTotal scan reports of about 53K
Android apps gathered between 2018 and November 8th, 2019 in pursuit of
informative features to extract from such reports.

During this process, we addressed two issues of interest to the research
community, finding the sets of correct and stable VirusTotal scanners over a
period of time. In Section 4.2, we showed that there is no universal set of
VirusTotal scanners that will always accurately label apps as malicious and
benign regardless to the utilized dataset or scan reports. In contrast, we
found that the set of correct scanners changes from one dataset to another
depending on (a) its composition of malicious and benign apps, and (b) the
period during which the apps' scan reports were gathered. We also found that
VirusTotal, presumably upon request from the antiviral software companies
themselves, replaces adequate versions of some scanners with ones that may not
be suitable to detect Android malware, as with the case of BitDefender. As for
the stability of scanners, using what we defined as a _certainty_ score, we
found that the majority of VirusTotal scanners do not change the labels they
assign to apps over time. Nevertheless, the stability of such labels is not
immune to future change given the dynamicity of VirusTotal and its
manipulation of scanners. Moreover, stability of labels does not imply the
correctness of those labels, as discussed in Section 4.3.

In addition to those insights that helped us identify the main limitations of
VirusTotal, the actionability of Maat is two-fold. Firstly, we showed that ML-