*3.3. Feature Engineering*

Based on the previous works surveyed, a set of features that are commonly used for malicious domain classification [11,13,22,23,27,28,35,69,70] were extracted. Specifically, the following nine features were used as the baseline (note that the focus of this work is on the potential use of robust features and not on the specific features; thus, WLOG, we evaluated a set of nine commonly used features):

log2 *count*(*cij*) *length*(*Domain*(*i*)), where each *Domain*(*i*) consists of *ni* distinct characters

{*ci*1, *ci*2, ... , *cini*}. For example, for the domain "google.com", the entropy is −(5 · ( 110 · log2 110 ) + 2 · ( 210 · log2 210 ) + 3(· 310 · log2 310 )) = 1.25 The domain has 5 characters that appear once ("l", "e", ".", "c", and "m"), one character that appears twice ("g") and one character that appears three times ("o").

### *3.4. Robust Feature Selection*

Next, the robustness of the set of features described above was evaluated to filter those that could significantly harm the classification process due to the adversary's manipulations. Table 2 lists the common features along with the mean value and standard deviation (note that the std in some cases (e.g., mean TTL value) is higher due to fact that these features have a positive value by definition.) For malicious and benign URLs based on our dataset, note that some features have similar mean values for both benign and malicious instances while they are commonly used. Furthermore, whereas "Standard deviation of the TTL" has distinct values for benign and malicious domains, we later show that an intelligent adversary can easily manipulate this feature, leading to a benign classification of malicious domains.

In order to understand the malicious abilities of an adversary, the base features were manipulated over a wide range of possible values, one feature at a time. This analysis considers an intelligent adversary with black-box access to the model (i.e., a set of features or output for a given input). The robustness analysis is based on an ANN model that classifies the manipulated samples, where the train set is the empirically crawled data, and the test set includes the manipulated malicious samples. Figure 3 depicts the possible adversary manipulations over any of the features. We chose recall for the evaluation metric, representing the average detection rate after modifications.

**Table 2.** Classic features and statistical properties (\*—robust features).

**Figure 3.** Base feature manipulation graphs (\*—robust features).

The well-known features were divided into three groups: robust features, robust features that seemed non-robust (defined as semi-robust), and non-robust features. Next, it it is shown how an attacker can manipulate the classifier for each feature and define its robustness:

domains (with a low standard deviation). Therefore, as this feature's minimal value is 1, it is considered to be a *robust feature*.

An adversary can set the DNS TTL values to [0,120,000] (according to the RFC 2181 [71] the TTL value range is from 0 to 231 − 1). Figure 3 shows that even manipulating the value of this feature to 60,000 will deceive the model and cause a malicious domain to be wrongly classified as a benign URL. Therefore, the "Standard deviation of the TTL" is considered a *non-robust* feature.

Based on the analysis above, the *robust* features presented in Table 2 were selected, and the *non-robust* ones were dropped. Using this subset, the model was trained and achieved an accuracy of 95.71% with an F1-score of 88.78%, compared to an accuracy of 97.2% and an F1-score of 90.23% when using all the features (i.e., including the robust ones). Therefore, we extended our analysis and searched for new features that would meet the robustness requirements to build a robust model with a higher F1-score.

### *3.5. Novel Features*

We aim to validate that manipulating the features in order to result in the misclassification of malicious instances will require a disproportionate effort that will deter the attacker from doing so. The four novel features were designed according to this paradigm based on two communication information properties, passive DNS changes, and the expiration time of the SSL certificate. For each IP, we used *Urlscan* [68] to extract the geo-location, which in turn was appended to a communication country list. The communication Autonomous System Numbers (ASNs) is a list of ASNs, extracted using *Urlscan*, each IP address, and appended the ASNs list. Benign-malicious ratio tables for communication countries, and communication ASNs (Figures 4 and 5) were created using the URL dataset and the *Urlscan* service. The ratio tables were calculated for each element *E* (country—for the communication countries ratio table; ASN—for the communication ASNs ratio table). Each table represents the probability that a URL associated with a country (ASN) is malicious. In order to extract the probabilities, the number of malicious URLs associated with *E* was divided by the total URLs associated with *E*. Initially, due to the heterogeneity of the dataset (i.e., there exist some elements that appear only a few times), the ratio tables appeared to be biased. To overcome this challenge, an initial threshold was set as an insertion criterion which is later detailed in Algorithm 1.

**Figure 4.** Communication countries ratio.

**Figure 5.** Communication ASNs ratio.

The following is a detailed summary of the novel features:

this feature can be extracted from passive DNS records obtained from VirusTotal, which are scarce (in terms of record types).

• **Expiration time of SSL certificate**: When installing an SSL certificate, a Certificate Authority (CA) conducts a validation process. Depending on the type of certificate, the CA verifies the organization's identity before issuing the certificate. When analyzing our data, it was noted that most malicious domains do not use valid SSL certificates and those that only use one for a short period. Therefore, this feature was engineered in order to represent the time the SSL certificate remains valid. The "Expiration time of SSL certificate", in contrast to the binary feature version used by Ranganayakulu et al. [69], extends the scope and represents both the existence of an SSL certificate and the remaining time until the SSL certificate expires.

**Algorithm 1** Communication Rank **Input:** URL, Threshold, Type **Output:**Rank(CCRorCAR)

```
if Type = Countries then
  ItemsList = communication countries list of the URL
else
  ItemsList = ASNs list of the URL
end if
Rank = 0
for Item in ItemsList do
  Ratio = 0.75 {Init value}
  Total
       _
        norm = 1 {Init value}
  if TotalOccurrences(Item) >= Threshold then
    Total
          _
           norm = Normalize(Item)
    Ratio = BenignRatio(Item)
  end if
  Rank+=(log0.5(Ratio + )/Total_norm)
end for
```
**Table 3.** Novel features and statistical properties.

Algorithm 1 receives a URL as an input and returns its communication country rate or the ASN communication rate (based on the type of the input in the algorithm). For each item (i.e., country or ASN), first the algorithm initializes the value of the ratio variable to 0.75 (according to [65], 25% of all URLs in 2020 were malicious, suspicious, or moderately risky). It then normalizes an item's total occurrences (Total\_norm) to be 1. Next, in Step 9, if an item's total number of occurrences is ≥ to the threshold, the algorithm replaces the ratio. It normalizes occurrences to the correct values according to the ratio tables given in Figures 4 and 5. Finally, the algorithm sums the rank with a log base of 0.5 of the ratio ( is a very small value that was added for the special case where *Ratio* = 0) and divides this value by the normalized total occurrences.

Figure 6 depicts the detection rate as a function of the novel features' values for each feature in Table 3. This evaluation proves that manipulating our novel features does not affect the robust model (i.e., the detection rate remains steady). The negative correlation between "Expiration time of SSL certificate" feature and the detection rate may raise concern. Nevertheless, it is noteworthy that the average value for malicious domains is three times higher than the benign ones. While, theoretically, the adversary can lower this value, the implications of such an action mean acquiring (or attaining for free) an SSL certificate. Since there is a validation process involved in the acquisition of an SSL certificate, doing so will cause the adversary to lose its anonymity and disclose its identity.

**Figure 6.** Novel robust feature manipulation graphs.

### **4. Empirical Analysis and Evaluation**

This section describes the testbed used to evaluate models based on the types of features (both robust and not). General settings are provided for each of the models (e.g., the division of the data into training and test sets), as well as the parameters used to configure each of the models, and the efficiency of each model. (our code is publicly available at https://github.com/nitayhas/robust-malicious-url-detection; accessed on 20 March 2022).