# Uncovering IP Address Hosting Types Behind Malicious Websites

Nimesha Wickramasinghe∗, Mohamed Nabeel+,
Kenneth Thilakaratne∗, Chamath Keppitiyagama∗ , Kasun De Zoysa∗
∗University of Colombo School of Computing
+Qatar Computing Research Institute

###### Abstract

Hundreds of thousands of malicious domains are created by miscreants everyday.
These malicious domains are hosted on a wide variety of network
infrastructures. Traditionally, attackers utilize bullet proof hosting
services (e.g. MaxiDed, Cyber Bunker) to take advantage of relatively lenient
policies on what content they can host. However, these IP ranges are
increasingly being blocked or the services are taken down by law enforcement.
Hence, attackers are moving towards utilizing IPs from regular hosting
providers while staying under the radar of these hosting providers. There are
several practical advantages of accurately knowing the type of IP used to host
malicious domains. If the IP is a dedicated IP (i.e. it is leased to a single
entity), one may blacklist the IP to block domains hosted on those IPs as
welll as use as a way to identify other malicious domains hosted the same IP.
If the IP is a shared hosting IP, hosting providers may take measures to clean
up such domains and maintain a high reputation for their users.

_K_ eywords IP hosting types, malicious domains, measurement

##  1 Introduction

Malicious websites rely on hosting infrastructures to launch their attackers.
Traditionally, attackers utilized Bulletproof Hosting Services (BPHS) such as
MaxiDed and Cyber Bunker to take advantage of their relatively lenient
policies on what content they can host [1]. However, these IP address (IP)
ranges are increasingly being blocked or the services are taken down by law
enforcement [2]. Hence, attackers are now moving towards utilizing IPs from
regular hosting
providers while staying under the radar of these service providers [3].
Further, public hosting provides economies of scale as well as allows to
launch attacks at scale from multiple geographic locations. The lack of
considerable effort to collect information about the users who register with
the service providers nor having proper security mechanisms to detect or
prevent these attacks make the life of attackers easier [4]. It is, however,
not clear to what extent this transition from BPHS to regular hosting services
has been occurring.

Blocklists are a widely used network security mechanism to block undesired
network traffic based on IP reputation [5]. Blocklists can be applied at
various points in a security architecture, such as a host, web proxy, DNS
servers, email server, firewall, directory servers or application
authentication gateways [6]. While they are useful, such blocklists may result
in high collateral damage if the blocked IPs are co-hosting benign domains.
Current blocklists do not make a distinction between IP addresses solely used
for malicious activities and IP addresses that co-host benign domains.

Figure 1: Overall Pipeline of Identifying Hosting Types of Malicious Domains

Domain threat intelligence services rely on hosting IPs of malicious domains
to identify additional malicious domains controlled by attackers [7]. Such co-
hosting relationships yield favorable results only if those IP addresses host
domains belonging to the same entity. Identifying if an IP is associated with
domains belonging to one entity is a challenging task due to several reasons:
(1) one needs to identify _all_ the domains hosted on the IP address, (2)
ownership of a domain is often times not available or difficult to find, and
(3) the domains hosted IPs change over time.

While the hosting type of an IP address is quite useful in practice, it is not
readily available except for selected cases. For example, major cloud service
providers such as Amazon AWS [8], Microsoft Azure [9] and Google GCE [10],
have made their IP address ranges public, but there is a long tail of other
hosting providers whose IP ranges are not readily available. Further, third-
party services that provide such information are either severely throttled or
quite expensive [11].

In this work, we design machine learning models to solve the problem of
identifying the hosting type of an IP address. We first make the distinction
between _hosting_ and _non-hosting_ IP addresses. A hosting IP address belongs
to a hosting service and anyone may utilize, often with a usage fee, to host
their domains whereas a non-hosting IP, as the name suggests, are either not
leased to third-party users or residential IP addresses belonging to ISPs. We
further categorize hosting IP addresses as either _shared_ or _dedicated_. A
shared hosting IP (shared IP for short) hosts domains belonging to more than
one user or organization, whereas a dedicated hosting IP (dedicated IPs for
short) hosts domains belonging to a single user or organization.

Based on the trained classifiers, we study the properties of IP addresses
hosting malicious domains collected from VirusTotal URL feed [12]. Our study
identifies that a vast majority (95.2%) of malicious websites are hosted on
regular hosting IPs and 97.1% of these malicious websites are co-hosted with
other unrelated benign websites.

We make the following contributions in this work:

  * •

We build a machine learning model to identify hosting IPs from non-hosting IPs
with a precision of 98.03%, recall of 96.89% and false positive rate (FPR) of
1.95%.

  * •

We build another machine learning model to identify shared IP from dedicated
IPs in the hosting IP category with a precision of 94.22%, recall of 94.00%
and FPR of 5.77%.

  * •

We study the hosting IP types of IPs hosting malicious domains and show that
overwhelming majority of malicious domains utilize shared hosting
environments.

##  2 Definitions

In this section, we formally define the different hosting types. Recall that
we first categorize all IPs in the wild into hosting and non-hosting IPs.
Hosting IPs are further categorized into shared and dedicated IPs.

Hosting IP: An IP address that belongs to a hosting provider which allows
third parties to host on (E.g.: Amazon AWS [13])

Non-hosting IP: An IP address that is not publicly offered to third-parties to
host. These IPs typically include IP ranges of ISPs who usually lease them to
businesses or use them as resdiential IP blocks (E.g.: Verizon residential IP
addresses), and of large organizations/universities (E.g.: Facebook, MIT).

Dedicated IP: - A hosting IP that hosts domains belonging to one single
entity. An entity can be either a user or an organization.

Shared IP: A hosting IP that hosts domains belonging to two or more entities.

##  3 Study Design

In this section, we outline our study methodology and data collection.

###  3.1 Methodology

We first collect hosting and non-hosting IPs distributed across the world.
Then, our domain experts label a subset of hosting IPs as shared and
dedicated. Hosting classifier takes a ground truth set of IPs along with their
features, and builds a machine learning model to classify IPs as hosting and
non-hosting. Shared/Dedicated classifier takes a ground truth set of hosting
IPs to build a machine learning model to classify IPs as shared and dedicated.
Figure 1 shows the final pipeline of identifying the hosting types of
malicious domains based on the above mentioned two trained classifiers.

###  3.2 Data Collection and Labeling

In order to study and understand discriminative features, 400,000 IP addresses
are collected, including 200,000 non-hosting IPs and 200,000 hosting IPs. The
next section comprises the steps we follow to collect these IP addresses.

####  3.2.1 Non-hosting IP Collection

We first collect a tentative diverse set of non-hosting IPs and perform
additional checks to ascertain that they are in fact non-hosting.

Universities, government and military organizations are highly likely to own
huge subsets of IP addresses which are not used for web hosting [7]. We
extract IP blocks owned by these institution by keyword matching organization
from IP-ASN database from the MaxMind DB. We select a random subset of IPs
from this collection and gather domains hosted on these IP address utilizing
Farsight Passive DNS service. We extract those those IPs whose owner is
matching with the owner of the websites. In order to diversify the non-hosting
IPs, we also include 20,000 Residential IPs obtained from a prior work [14].
Overall, we collect 200,000 non-hosting IP addresses.

####  3.2.2 Hosting IP Collection

We collect hosting IP addresses from publicly advertised hosting IP ranges
from major hosting provides such as GCP, AWS and Azure, and from MaxMind IP-
ASN database for web hosting providers such as Dream Host, Host Monster,
GoDaddy, Bluehost, Host Dime, Host Gator, Liquid web and Host Europe. From
nearly 80 million publicly advertised hosting IP addresses, we selected a
random sample of 200,000 to conduct our study.

####  3.2.3 Dedicated and Shared IP Collection

From the collected hosting IPs, we identify a ground truth set of dedicated
and shared IPs. We first collect the domains hosted on each of these IP
addresses using the Passive DNS service. We then collect WHOIS records for
these domains. It should be noted, as observed in prior research [15], we are
only able to collect WHOIS record for a limited number of domains and the
registrant in most of these collected WHOIS records are either redacted or
privacy protected. This limits our ability to determine the domain ownership
solely using WHOIS records. Hence, we follow the following procedure to
identify dedicated and shared IP addresses.

  * •

If an IP is hosting only one domain, it is by definition a dedicated IP.

  * •

If WHOIS records are available and the registrants are not redacted or privacy
protected, we mark those hosting IPs having same registrant for all domains as
dedicated and otherwise shared.

  * •

If all websites hosted on a given IP redirects to the same parent website, we
mark it as dedicated as they are all control by the same entity.

  * •

For the remaining, we visually inspect the web page content to find similarity
in terms of information included, trademark and contact information. If all
websites hosted on an IP has similar content, we mark the IP as dedicated.

####  3.2.4 Data Sources

We utilize Farsight Passive DNS service [16] to extract historical hosting
information about domains and IPs. For the 400,000 IP addresses, we collected
nearly 45 million Passive DNS records in Aug. 2020 and in Dec. 2020. In order
to diversify our findings, we collected Passive DNS records across different
time intervals. VirusTotal URL Feed [12] on the other hand, is used to
identify malicious domains for this study. VirusTotal provides an aggregated
report for each URL containing results from 70+ scanners. Following best
practices [17], we select those domains with at least 5 scanners detecting
positive as our malicious domain corpus. Maxmind database [18] is utilized to
extract IP ASN information.

##  4 Classifying IP Addresses

###  4.1 Feature Engineering

According to our classification hierarchy, the first step is to identify
features with high discriminative power to identify if a given IP is a Hosting
IP or a Non-hosting IP. The second step is to identify features to classify
the hosting IPs as dedicated or shared. The identified features for the two
steps are describe in this section.

####  4.1.1 Features for the Hosting/Non-hosting IP Classifier

Table 1: Hosting/Non-hosting classification features

ID | Feature Name | Description | Type
---|---|---|---
1 |  number of TLD+2 |  Number of TLD+2 domains resolved to the given IP |  Numerical
2 |  number of TLD+3 |  Number of TLD+3 domains resolved to the given IP |  Numerical
3 |  number of domains |  Number of domains resolved to the given IP |  Numerical
4 |  percent DNS |  Percentage of IPs in /24 IP prefix with DNS records |  Numerical
5 |  mean TLD+3 |  Mean number of TLD+3 domains resolved to IPs in /24 IP prefix |  Numerical
6 |  max TLD+3 |  Maximum number of TLD+3 domains resolved to IPs in /24 IP prefix |  Numerical
7 |  mean TLD+2 |  Mean number of TLD+2 domains resolved to IPs in /24 IP prefix |  Numerical
8 |  max TLD+2 |  Maximum number of TLD+2 domains resolved to IPs in /24 IP prefix |  Numerical
9 |  number of owners |  Number of owners of the IP during the past 10 years |  Numerical
10 |  number of inetnums |  Number of unique historical inetnums (IP ranges) during the past 10 years |  Numerical
11 |  max inetnum size |  Maximum size of the inetnums during the past 10 years |  Numerical
12 |  min inetnum size |  Minimum size of the inetnums during the past 10 years |  Numerical
13 |  size of inetnum |  Size of the inetnums during the past 10 years |  Numerical
14 |  type of inetnum |  Assignment type of the current inetnum |  Categorical
15 |  most recent update |  Most recent WHOIS record update |  Numerical
16 |  number of whois |  Number of WHOIS records during the past 10 years |  Numerical

Table 5 shows the features we utilize to train the hosting/non-hosting
classifier. We provide a brief intuition and observations of these features
below. Number of TLD+2/TLD+3/all domains resolved to a given IP describe the
number of second level, third level and all domains hosted on a given IP.
Intuitively, as hosting providers need to host many domains in a limited range
of IP addresses compared to Non-hosting IP owners, it is highly likely for
Hosting IPs to host more domains per IP. The evaluation on the labeled set
shows that Non-hosting IPs host 1.06, 0.56, 2.07 domains on average for the
three categories whereas Hosting IPs host 452.6, 40.3, and 691.45 domains.

Features 4 to 8 extract statistics of domains resolved to IPs in /24 IP
prefix. Since IPs are allocated to different ISPs in subnets, it is highly
likely for /24 IP prefixes of Hosting IPs to have a higher number of domains
mapped compared to the number of /24 IP prefixes of Non-hosting IPs.

Features 9 to 14 are related to inetnums. An inetnum object describes a range
of IP addresses and all its attributes. It is known that hosting IPs are
resold and reallocated more often to different vendors [9] compared to non-
hosting IPs whose owners are fairly stable. This observation is utilized and
the historical ownership of IPs (Feature 9) to differentiate between the two
IP types. In the ground truth IPs, it is found that the average number of
historical owners for Hosting IPs is 6.8 whereas it is 1.2 for Non-hosting
IPs. All inetnums are created in a hierarchical manner and therefore it forms
an inetnum tree. Since Feature 9 shows a significant difference, it is likely
the allocated inetnums differ too and this behavior is captured with Feature
10. Features 11 to 13 represent maximum size/minimum size/size of the inetnums
respectively. Residential IPs under non-hosting IPs tend to have much more
objects in inetnums. Though there is no major difference between the maximum
and minimum number: the size difference is significant. For non-hosting IPs,
the average size of inetnum is 15,893,510.4 and 8,652,824.7 for Hosting IPs.
Feature 14 identifies the assignment type of the current direct inetnum. WHOIS
record of an IP address explains the NET TYPE of a given IP address. The
categories are: Direct Allocation (ISP), Direct Assignment (End user),
Reallocated (downstream ISP customer), and Reassigned (end user customer). It
is observed that hosting IPs are mostly in the Direct Allocation category as
they are allocated for other hosting providers by Regional Internet
Registries.

Features 15 and 16 are related to the timestamps of IP WHOIS records. As
Hosting IPs are changing hands often, intuitively they are likely to have more
recent last update dates. Feature 15 capture this attribute. The average last
update date for Non-hosting IPs is on average 7 years whereas the same for
hosting IPs is 2 years. Feature 16 measures the number of WHOIS records of the
last 10 years. The number of WHOIS records for Hosting IPs is likely to be
more than that for Non-hosting IPs as hosting IPs change frequently. We
observe from our data that the average number of WHOIS records associated with
a Hosting IP is around 3.67 whereas it is 1.26 for a Non-hosting IP.

####  4.1.2 Features for Dedicated/Shared IP Classifier

Table 2: Dedicated/Shared classification features

ID | Feature Name | Description | Type
---|---|---|---
1 |  number of TLD+2 |  Number of TLD+2 domains resolved to the given IP |  Numerical
2 |  number of TLD+3 |  Number of TLD+3 domains resolved to the given IP |  Numerical
3 |  number of domains |  Number of domains resolved to the given IP |  Numerical
4 |  number of owners |  Number of owners of the IP during the past 10 years |  Numerical
5 |  number of whois |  Number of WHOIS records in the past 10 years |  Numerical
6 |  avg daily churn |  The average daily churn of domains hosted on the IP |  Numerical
7 |  std daily churn |  The standard deviation of daily churn |  Numerical
8 |  avg duration |  The average duration of the domains hosted on the IP |  Numerical
9 |  std duration |  The standard deviation of the duration of the domains |  Numerical

In this section, we describe the features we utilize for dedicated/shared IP
classifier. We find that some of the features utilized with hosting/non-
hosting classifier are quite useful for this classifier. Specifically, as
shown in Table 2, features 1 to 5 are reused but their proportions are
different in the ground truth of dedicated and shared IPs.

Intuitively, Shared IPs need to host a higher number of domains belonging to
different entities compared to Dedicated IPs. The evaluation on the labeled
set shows that Dedicated IPs host, on average 796.08, 43.25, 539.828 second,
third and all level domains whereas Shared IPs host, on average, 2624.18,
609.88, 1650.06 for the same.

Features 6 and 7 represent Mean and Standard deviation of the daily churn of
domains hosted on a given IP address. Shared IPs are likely to have more churn
compared to the dedicated IPs due to its changing nature. We collect daily
hosted apexes on each IP for the last 60 days and measure the mean and the
standard deviation. The average daily churn for Dedicated IPs is 0.12, whereas
it is 1.18 for Shared IPs. The average standard deviation is 0.68 and 5.91 for
Dedicated and Shared IPs respectively.

Features 8 and 9 measures the Mean and Standard deviation of the duration of
domains hosted on a given IP address. Domains hosted on Shared IPs are likely
to be more ephemeral and likely to vary more across hosted domains compared to
those on Dedicated IPs. Our analysis shows that the average duration of all
the Dedicated IPs is 2.3 years and it is 1.2 years for Shared IPs. The
standard deviation for these two classes are 1.4 and 1.1 respectively.

####  4.1.3 Ground Truth and Feature Extraction

We utilize Passive DNS and historical IP WHOIS data to extract the features
described in Setion 4.1.1. Features 1-8 in hosting/non-hosting classifier and
Features 1-3 in dedicate/shared classifier are derived from passive DNS. The
rest of the features are derived from historical IP WHOIS data.

In Jan. 2021, we collect features for two ground truth data sets H-GT1 and
H-GT2 for Hosting/Non-Hosting IP classifier. IPs for these two data sets were
selected randomly from the 400,000 IPs we studied initially. Each set has 1000
IPs from each class along with the identified features. Similarly, we collect
features for two ground truth datasets D-GT1 and D-GT2 for Dedicated/Shared IP
classifier. Each set has 400 IPs from each class.

###  4.2 Training Machine Learning Models

We train 8 classifiers (Support Vector Classification (SV), Random Forest
(RF), Extra Tree (ET), Logistic Regression(LR), Decision Tree (DT), Gradient
Boosting (GB), AdaBoosting (AB) and K-Neighbors (KN) Classification) using the
features Section 4.1. We test all these classifiers using 5-fold cross
validation techniques and realized that RF outperform others in our task.
Hence, in the rest of the paper, we present results only from RF classifier.

(a) H-GT1

(b) H-GT2

Figure 2: ROC Curves for Hosing/Non-hosting Classifier for the Two Ground Truth Sets Table 3: Performance of Hosting/Non-Hosting Classifier for the Two GT Sets Dataset |  Precision |  Recall |  FPR
---|---|---|---
H-GT1 |  97.98 |  96.68 |  2.00
H-GT2 |  98.03 |  96.89 |  1.95
Figure 3: Feature Importance for Hosting/Non-Hosting Classifier

Figure 2 shows the ROC curves for the two ground truth datasets for
Hosting/Non-Hosting Classifier. We observe consistently high performance
across the two datasets. Table 3 shows the performance of the classifier. Our
classifier achieves a precision of 98.03%, a recall of 96.89% and FPR (false
positive rate) of 1.95%. Figure 3 shows the importance of each feature. The
most recent WHOIS update, the number of owners and domains are the most
discriminating features.

(a) D-GT1

(b) D-GT2

Figure 4: ROC Curves for Dedicated/Shared Classifier for the Two Ground Truth Sets Table 4: Performance of Dedicated/Shared Classifier for the Two GT Sets Dataset |  Precision |  Recall |  FPR
---|---|---|---
D-GT1 |  95.00 |  94.4 |  8.33
D-GT2 |  94.22 |  94.00 |  5.77
Figure 5: Feature Importance for Dedicated/Shared Classifier

Figure 4 shows the ROC curves for the two ground truth datasets for
Dedicated/Shared Classifier. We observe consistently high performance across
the two datasets. Table 4 shows the performance of the classifier. Our
classifier achieves a precision of 94.22%, a recall of 94.00% and FPR (false
positive rate) of 5.77%. Figure 5 shows the importance of each feature. The
number of owners, domains and TLD+2s are the most discriminating features.

Table 5: Top Hosting Providers with the Highest Number of Benign and Malicious
Domains

All Hosting | #AllDomains | Shared Hosting | #MalDomains | Dedicated Hosting | #MalDomains
---|---|---|---|---|---
Wix.com Ltd |  18,411,677 |  Cloudflare Net |  22,473 |  Amazon-02 |  2,126
Google |  14,679,240 |  Amazon-02 |  14,139 |  Cloudfare Net |  194
Cloudflare Net |  12,786,921 |  Namecheap |  654 |  DigitalOcean |  123
GoDaddy Inc. |  11,114,352 |  Unified Layer |  450 |  OVH |  68
Amazon-02 |  9,948,052 |  Hostinger |  249 |  Hostinger |  65

##  5 Hosting Types of Malicious Domains

Using the trained classifiers, we analyze the hosting types of 174,701
malicious domains collected from VirusTotal URL Feed in January 2021. We first
identify the hosting IP addresses for these websites. These domains are hosted
on 73,441 IP addresses. We are able to collect features for 47,141 IP
addresses. Hence, we study the hosting types of these latter set of IPs where
143,316 malicious domains are hosted.

We first use the trained Hosting/Non-hosting classifier to predict hosting and
non-hosting IPs in this set. With 95% confidence, we identify 44,890 (95.2%)
hosting IPs and 2,251 (4.8%) non-hosting IPs. Out of the hosting IPs, our
Dedicate/Shared classifier detects 43,605 (97.1%) shared IPs and 1,285 (2.9%)
dedicated IPs.

Figure 6: Number of Malicious and Overall Domains per Shared IP

Figure 6 shows the distribution of malicious and all domains on shared IPs. It
shows that while there are more than 200 domains hosted on 70% of shared IPs,
nearly 50% of these IPs host a handful of malicious domains. This shows that
attackers utilize large pool of unrelated domains making it difficult to
identify and block/take down. They also tend to choose shared hosting due to
to their low cost of renting and their poor security measures [19]. Hence, in
the rest of the paper, we present results only from RF classifier.

Table 5 shows the top 5 hosting providers in terms of total number of domains
hosted, total number of malicious domains hosted on shared IPs and the total
number of malicious domains hosted on dedicated IPs. It is concerning to
observe that Cloudflare and Amazon have the highest number of malicious
domains hosted under both shared and dedicated settings. Google as a hosting
service provider, on the other hand, has much less number of malicious domains
in their infrastructure. We believe this information is quite useful in
practice to make business decision to utilize either dedicated or shared
hosting and under which hosting provider.

##  6 Related Work

One may wonder if published IP address ranges could be used to identify
hosting IPs [20, 8, 9] from Non-hosting IPs. While they may help one identify
a subset of hosting IPs, they are not exhaustive due to several reasons: (1)
not all cloud services make the IP address ranges readily available to access
by the public, and (2) there is a long tail of hosting providers supporting
hosting IPs. In the prior work, IP address classification is mainly used to
efficiently allocate IPs to different organizations and to improve the
efficiency of routing IP packets between routers [21]. Xie et al. [22] propose
a technique to automatically classify IP addresses as dynamic, that is, DHCP
allocated IPs, and static using Hotmail server logs in order to identify
spams. Their observation is that most of the spam email servers are hosted at
dynamic IP addresses. However, this observation is not reflected on the
malicious domains we identified from VirusTotal URL feed and, thus, a
different categorization is required. Scott et al. [23] propose to capture the
IP footprint of CDN (content delivery network) deployments. While this work
identifies IPs of specific shared infrastructures of content delivery, these
IPs may not necessarily be public IPs the way we classify in our work in order
to study the malicious hosting types. Recently, Nabeel et al. [7] introduce
the concept of shared and dedicated IPs in their domain inference algorithms.
We observe a higher number of false positives than reported in their work and
we believe this is due to the fact they do not make a distinction between
hosting and non-hosting IPs. Recently, Xianghang et al. [14], design a machine
learning model to identify residential IP addresses. While their work is
useful in identifying potential features, our work is different from theirs as
we categorize IPs to identify hosting and non-hosting IPs, where residential
IPs are only a subset.

##  7 Conclusions

In this paper, we build first machine learning models to identify the hosting
types of IP addresses. Specifically, we build a classifier to categorize IP
addresses as hosting and non-hosting and a subsequent classifier to categorize
hosting IPs as shared or dedicated. From a corpus of 143,316 malicious
domains, we observe the concerning trend that attackers 97.1% of the time
utilize shared hosting IPs to host their malicious domains. Further, we show
that majority of these shared IPs host many potentially benign domains and
only a handful of malicious domains. This makes reputation systems that rely
on hosting infrastructure properties less effective. We believe more should be
done by hosting providers to safeguard their shared hosting infrastructures.

## References

  * [1] M. Goncharov.  Criminal hideouts for lease: Bulletproof hosting services.  Forward Looking Threat Research (FTR), 2015.
  * [2] Brian Krebs.  Cyber bunker taken down.  https://krebsonsecurity.com/2019/09/german-cops-raid-cyberbunker-2-0-arrest7-in-child-porn-dark-web-market-sting/, 2019\.  Accessed: 17-04-2021.
  * [3] Yin Pa, Katsunari Yoshioka, and Tsutomu Matsumoto.  Detecting malicious domains and authoritative name servers based on their distinct mappings to ip addresses.  Journal of Information Processing, 23:623–632, 09 2015.
  * [4] Davide Canali, Davide Balzarotti, and Aurélien Francillon.  The role of web hosting providers in detecting compromised websites.  05 2013.
  * [5] Sivaramakrishnan Ramanathan, Jelena Mirkovic, and Minlan Yu.  BLAG: Improving the accuracy of blacklists.  In NDSS, 2020.
  * [6] Timothy Shimeall and J.M. Spring.  Introduction to information security: A strategic-based approach.  Introduction to Information Security: A Strategic-Based Approach, pages 1–360, 11 2013.
  * [7] Mohamed Nabeel, Issa Khalil, Bei Guan, and Ting Yu.  Following passive dns traces to detect stealthy malicious domains via graph inference.  ACM Transactions on Privacy and Security, 23:1–36, 07 2020.
  * [8] Team AWS.  AWS Public IPs.  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html/, 2021\.  Accessed: 17-02-2021.
  * [9] Team Azure.  Microsoft Azure Public IP Ranges.  https://github.com/bcoe/which-cloud/blob/master/data/PublicIPs.xml, 2021\.  Accessed: 17-02-2021.
  * [10] Team Google.  Google Public IP API.  https://github.com/bcoe/gce-ips/blob/master/index.js, 2021.  Accessed: 17-02-2021.
  * [11] Team IpInfo.  IP Lookup Service, howpublished = https://ipinfo.io, note = Accessed: 17-05-2021, year = 2021.
  * [12] VirusTotal.  VirusTotal – Free Online Virus, Malware and URL Scanner.  https://www.virustotal.com/, 2021.  Accessed: 28-02-2021.
  * [13] Team AWS.  Amazon Web Services.  https://aws.amazon.com, 2020.  Accessed: 17-05-2021.
  * [14] Xianghang Mi, Xuan Feng, Xiaojing Liao, Baojun Liu, XiaoFeng Wang, Feng Qian, Zhou Li, Sumayah Alrwais, Limin Sun, and Ying Liu.  Resident evil: Understanding residential ip proxy as a dark service.  In 2019 IEEE Symposium on Security and Privacy (SP), pages 1185–1201, 2019.
  * [15] Tobias Lauinger, Kaan Onarlioglu, Abdelberi Chaabane, William Robertson, and Engin Kirda.  Whois lost in translation: (mis)understanding domain name expiration and re-registration.  In Proceedings of the 2016 Internet Measurement Conference, IMC ’16, page 247–253, New York, NY, USA, 2016. Association for Computing Machinery.
  * [16] Farsight Security, Inc.  DNS Database.  https://www.dnsdb.info/, 2021.  Accessed: 28-02-2021.
  * [17] Ravindu De Silva, Mohamed Nabeel, Charith Elvitigala, Issa Khalil, Ting Yu, and Chamath Keppitiyagama.  Compromised or attacker-owned: A large scale classification and study of hosting domains of malicious urls.  In 30th USENIX Security Symposium (USENIX Security 21). USENIX Association, August 2021.
  * [18] MaxMind, Inc.  IP ASN Database.  https://maxmind.com/, 2021.  Accessed: 28-02-2021.
  * [19] Amirreza Niakanlahiji, Mir Mehedi Pritom, Bei-Tseng Chu, and Ehab Al-Shaer.  Predicting zero-day malicious ip addresses.  New York, NY, USA, 2017. Association for Computing Machinery.
  * [20] Team Google.  Google Public IPs, howpublished = https://cloud.google.com/compute/docs/ip-addresses/ephemeraladdress, note = Accessed: 17-02-2021, year = 2021.
  * [21] M. A. Ruiz-Sanchez, E. W. Biersack, and W. Dabbous.  Survey and taxonomy of ip address lookup algorithms.  Magazine of Global Internetworking, 15(2):8–23, March 2001.
  * [22] Yinglian Xie, Fang Yu, Kannan Achan, Eliot Gillum, Moisés Goldszmidt, and Ted Wobber.  How dynamic are ip addresses?  In The Proceedings of the Special Interest Group on Data Communication (SIGCOMM), pages 301–312. ACM, 2007.
  * [23] Will Scott, Thomas Anderson, Tadayoshi Kohno, and Arvind Krishnamurthy.  Satellite: Joint analysis of cdns and network-level interference.  In Proceedings of the 2016 USENIX Conference on Usenix Annual Technical Conference, pages 195–208. USENIX Association, 2016.


# Does Explainable Machine Learning Uncover the Black Box in Vision
Applications?

Manish Narwaria∗ ∗Department of Electrical Engineering, Indian Institute of
Technology Jodhpur, NH 62, Surpura Bypass Rd, Karwar, Rajasthan 342037, India.
e-mail<EMAIL_ADDRESS>
###### Abstract

Machine learning (ML) in general and deep learning (DL) in particular has
become an extremely popular tool in several vision applications (like object
detection, super resolution, segmentation, object tracking etc.). Almost in
parallel, the issue of explainability in ML (i.e. the ability to
explain/elaborate the way a trained ML model arrived at its decision) in
vision has also received fairly significant attention from various quarters.
However, we argue that the current philosophy behind explainable ML suffers
from certain limitations, and the resulting explanations may not meaningfully
uncover black box ML models. To elaborate our assertion, we first raise a few
fundamental questions which have not been adequately discussed in the
corresponding literature. We also provide perspectives on how explainablity in
ML can benefit by relying on more rigorous principles in the related areas.

##  I The Need to Revisit Current Explainable ML Philosophy

The powerful modeling capabilities of ML/DL has fueled a transition from white
box to black box modeling, in research, industry and education [1].
Consequently, despite the success of ML/DL in several applications, the issue
of explainability (i.e. ability to understand how an ML model arrives at a
decision/prediction) remains one of the bottlenecks in adopting ML/DL models
to a wider canvas of applications. Accordingly, it is fairly well accepted
that ML/DL models should be amenable to a level of scrutiny that goes beyond
simple audit. Hence, explainable ML remains an active research area [2], [3]
in both industry and academia. However, there appears a lack of discussion on
whether the current philosophy behind explainable ML can help to achieve the
eventual goal of uncovering black box models. This is especially required in
vision applications as meaningful visualization/explanation can greatly
facilitate more transparent deployment of ML/DL models in practice. In that
context, we raise three important questions which in our opinion are
fundamental to a rigorous discussion on explainable ML, and have not been
adequately raised or addressed in the current literature. We also attempt to
provide perspectives and some insights into these questions. Hence, the goal
is not to criticize existing efforts on explainable ML. Rather it is to build
upon them, and in the process raise awareness about possible issues whose
mitigation may help to develop more understandable ML/DL models.

###  I-A Explainable ML: A Prehoc Necessity or A Posthoc Evil?

A commonly accepted argument for the need of explainability in ML is based on
how serious the consequences might be if the trained model makes an error. For
instance, consider medical image analysis where an ML/DL algorithm erroneously
classifies an image as “normal” when in fact there was say a tumor (or some
other serious medical abnormality) present. Obviously, such an error can have
serious medical implications. Likewise, errors made by an ML algorithm in the
context of autonomous vehicles might also undeniably lead to disastrous
consequences. On the other hand, errors made by ML/DL are considered less
serious, from a practical view point, in several other vision applications
such as object recognition in photos/videos on the web, visual content
retrieval, gaming, vision based recommender systems, audiovisual communication
etc. Such a dichotomous but seemingly logical philosophy has unfortunately
lead to a proliferation of black box models in the first place. As a result,
explainability has been possibly relegated to merely a _posthoc_ analysis of
the trained ML model rather than being be viewed as a fundamental and inherent
concept in ML/DL algorithm design. It is therefore reasonable to ask if we
should revisit the current philosophy by making explainability a pre-design
necessity and not merely a post design baggage.

###  I-B Do visual explanations generate new knowledge?

DL in particular has gained popularity since it achieves better prediction
performance than traditional non ML and ML approaches, especially in many
vision applications. This can be largely attributed to the fact that feature
extraction in DL is typically implicit (i.e. does not require handcrafted
features) and data dependent. In contrast, traditional approaches rely heavily
on feature engineering (explicit modeling) and possible use of apriori domain
knowledge. It is therefore natural to ask: what additional information was
extracted and exploited by the DL model that was missed in the more
traditional approach for solving the same problem? Stated differently, can the
explanations obtained from trained ML/DL models bridge the seeming gap in
terms of additional knowledge that DL supposedly exploits which lead to better
performances in vision applications? In the context of this question, we note
that several existing efforts attempt to “explain” how DL works in
applications like image classification/recognition etc. by analyzing important
regions (in the form of superpixels [4], salient regions [5] or similar ideas
like layer-wise relevance propagation LRP [6] etc.). Hence, the main idea is
to find regions in a test image (video) on which the trained DL network
focuses on (or has relevance) for the purpose of classification/recognition.
This information presumably explains how the ML/DL algorithm arrives at a
decision. To analyze this in more details, it is convenient to consider the
example in Figure 1 where we used two test images in which the main object of
interest is a zebra. We then employed two pre trained models namely Densenet
[7] and Inception-Resnet [8] to recognize the object in the test images. We
found that both the models (pre-trained on nearly a million images from
Imagnet [9]) correctly recognized the object namely ‘zebra’ (with a
probability above 0.9) in both the test images. However, from an explainable
ML view point, it will be obviously more interesting to understand visual
signal information that the two DL models used to make the correct decision.

Figure 1: Visual explanations generated for two test images. The explanations
were generated by LIME method [4] for pre-trained Densenet and Inception-
Resnet models. Only super pixels that correspond to top two features as
determined by LIME are shown for easier visualization.

To that end, we employed LIME [4] to get further insights into the two black
boxes. LIME identifies regions in the image (in the form of superpixels) which
were more important for the ML/DL algorithm to arrive at the decision. In
Figure 1, the second and third image in each row shows the two most important
regions (visualized as explanation maps) that the respective DL models focused
on to correctly classify the image as ‘zebra’. While the 4 explanation maps in
Figure 1 (b), (c), (e) and (f) correspond to some parts of the object (zebra),
two key questions remain unanswered. First, do these specific image parts
stand out (i.e. are different) in comparison to other image regions and if so
how? Second, are these parts amenable to interpretation in terms of simpler
signals or patterns similar to basis functions (or basic building blocks) for
the object under consideration (zebra)?

###  I-C Can visual explanations constitute a many-to-one mapping?

Another important issue in explainable ML is that of a many-to-one mapping. It
means that a unique object (class label) can be potentially mapped to more
than one explanations. For example, ‘zebra’ object in Figure 1 has 4 different
explanations despite the fact that both the DL models recognized the object
class correctly. In an analogous manner, we can conclude that a given DL model
might not always rely on certain unique signal information (features) to
correctly identify the same object. For instance, we observe from Figure 1 (b)
that Densenet uses “legs” part of the zebra for first test image while it
relies on middle body part (Figure 1 (e)) in case of the second test image.
Likewise, while Inception-Resnet employs “legs and face” (Figure 1 (c)) region
of the zebra for first test image but focuses on “ face and middle part of
body” (Figure 1 (f)) in the second test image. That is, the two DL models
under consideration correctly identify the same object but rely on very
different visual signal information in the two test images in Figure 1.
Moreover, the dimensionality of the mapped space (i.e. the number of
explanations for the same object of interest) can increase quickly. Suppose we
have $m$ images of an object of interest whose class (label) is being
predicted by say $n$ ML models, and we employ $p$ explanation methods.
Obviously, unless we have unique explanations, we are potentially looking at
$m\times n\times p$ explanations for the same object. This leads to the
following question: how logical is it that ML models correctly recognize the
same object present in more than one test image but characterize it
differently (i.e. use very different explanations for each test image to
arrive at the correct prediction)?

##  II Explainable ML: A Possible Way Forward

We now attempt to provide some perspectives in the light of the questions
raised. Before doing that, two points are however worth emphasizing. First, we
note that the questions raised in the previous section are not influenced by
our choice of specific test images in Figure 1 or the 2 DL models (Densenet
and Inception-Resnet) or even the explanation method (LIME). Rather, the
genesis of these questions lies primarily in the current philosophy behind
explainable ML. Second, the said questions are not necessarily independent of
each other. Thus, it is possible that focusing on inherent ML model
explainability right from the start may reveal useful insights about knowledge
generation and uniqueness of explanations.

###  II-A Explainability as a first principle

With regard to the question raised in section I-A, it should be reasonable to
conclude that explainability ought be an integral part of ML/DL algorithm
design process itself and not an after thought. Thus, irrespective of the use-
case scenario, an ML algorithm should be explainable/interpretable. Of course,
the type of interpretability (i.e. the dimension) required can vary depending
on the application (similar to how one views various tools in say signal
processing). This could for instance involve learning of data-dependent but
more intuitive and interpretable basis functions or representations. Likewise,
ideas and design philosophy from signal processing, information theory and
related disciplines can be borrowed. For example, fundamental ideas of
frequency have been exploited to improve the interpretability of DL [10].
Other examples include the use of classical but interpretable concepts (such
as filtering) in Digital Signal Processing (DSP) [11], in vision [12] or from
physics [13], [14] etc. Similarly, exploitation of apriori knowledge such as
logical rules and knowledge graphs [15] could lead to possibilities of
enhancing our understanding of black box ML systems. Thus, it could be argued
that irrespective of the approach eventually used, the design philosophy
should focus on ML models that are inherently explainable. A _posthoc_
analysis could of course always be employed as an add-on for further
refinement or seeking application specific insights.

###  II-B Possibility of Knowledge Generation and Transfer

Supervised ML/DL approach depends heavily on data to derive mapping from a
visual signal to the task of interest (eg. recognition of objects from
images). Because the training process potentially uses a large amount of
labeled data (millions of images for instance in Imagnet [9]), it may not be
unreasonable to expect that explanations of black box ML/DL models should
reveal some new insights. However, it is likely that the current methods for
explainablity, that tend to rely exclusively on posthoc analysis, are
inadequate for this purpose. For instance, it is unclear if further analysis
of the explanations (such as those in Figure 1) say in terms of explicit
shape, texture, orientation or color will reveal any new insights about the
object under consideration. Thus, the answers to the questions discussed in
section I-B are probably in the negative. But perhaps more pertinently, these
questions are meant to emphasize the importance of explanations that are
meaningfully quantifiable and uniquely interpretable at least in the context
of specific application (or a set of related applications). Such functionality
may eventually open the possibility of generating potentially new and explicit
knowledge which is open to scrutiny and amenable to knowledge transfer. A
relevant example in this regard would be that of vision science which includes
calibrated psycho visual experiments and computational modeling [16]. The
findings of such experiments are not only fundamental but have had tremendous
impact on conceptualization and meaningful advancement of applications. This
includes next generation video technologies like HFR [17], HDR [18], visual
saliency [19], modern video compression [20], to list a few. On similar lines,
a bottom-up and knowledge centric explainable ML design philosophy may be
practically more useful and scalable.

###  II-C Uniqueness of explanations

The keen reader will probably agree that the aspect of many-to-one mapping
highlighted in section I-C should in general be problematic in the context of
explainable ML. Specifically, it implies that:

  1. 1.

An ML/DL algorithm may not be using unique visual information to correctly
identify the same object present in more than one test image.

  2. 2.

The method used to explain/interpret black box models may not be capturing the
correct features (information) that is being used by the ML model to make
predictions.

  3. 3.

Two or more ML models with same prediction accuracies might be using very
different explanations.

The first point indicates lack of consistency and reliability of the ML/DL
model under consideration. The second refers to possible deficiencies in the
method itself that was used to explain the ML model. The third point is
interesting since it refers to non equivalence of two or more ML models having
the same prediction performance. That is, it would be more logical that a
_better_ ML model should tend to use similar description (explanation) of the
same object in more than one test image.

Thus, consistency of explanations in addition to prediction accuracy should be
a more reasonable performance benchmark metric. In an ideal case, one could
hope that explanations for same object are same/similar across more than one
test images. This may provide an opportunity to simply characterize those
explanations and generate _signatures_ for different objects. In such case,
deployment of ML models in practice would be greatly simplified since
prediction could be done using logical rules and/or simpler models which in
turn were derived from more complex DL/ML models trained on large sets of
labeled data. Hence, uniqueness of explanations would increase ML model
reliability and also open possibilities of new knowledge generation.

##  III Concluding Remarks

There is no denying that ML models ought to be more transparent from the view
point of signal information exploited to make predictions. We have, however,
reasoned that the current philosophy behind explainable ML may not
meaningfully uncover the black box. To that end, we highlighted three specific
but not entirely independent aspects that probably deserve more attention.
These include:

  1. i.

A generally accepted but probably unnecessary dichotomy wherein the need for
explainability depends on use-case scenario. As a result, explainability might
not be treated as an inherent concept but merely an add-on in the context of
ML algorithm design.

  2. ii.

An apparent lack of clarity on what the explanations imply and how they could
help to improve our existing knowledge about specific tasks (say object
recognition). The expectation about new knowledge generation should not be
unreasonable especially when a tremendous effort goes into large scale data
collection and labeling, and eventually training the model on modern computing
devices.

  3. iii.

The problem of many-to-one mapping where one class label (object) might be
mapped to more than one explanations. Analogously, the same ML algorithm might
use different features to identify the same object.

We also provided perspectives on the identified issues and possible ways of
mitigating them. In particular, we emphasized that explainability should be
accorded high priority in ML algorithm design process and not left merely as a
_posthoc_ exercise. This could, for instance, be facilitated by relying on
philosophy of fundamental principles in related domains. As already pointed
out, such approach has already been employed [10]-[15], and could prove to be
the gateway to meaningfully more transparent ML models.

## References

  * [1] M. Narwaria, “The transition from white box to black box: Challenges and opportunities in signal processing education,” _IEEE Signal Processing Magazine_ , vol. 38, no. 3, pp. 163–173, 2021.
  * [2] N. Xie, G. Ras, M. van Gerven, and D. Doran, “Explainable deep learning: A field guide for the uninitiated,” _arXiv_ , 2020.
  * [3] V. Buhrmester, D. Münch, and M. Arens, “Analysis of explainers of black box deep neural networks for computer vision: A survey,” _arXiv_ , 2019.
  * [4] M. T. Ribeiro, S. Singh, and C. Guestrin, “”why should i trust you?”: Explaining the predictions of any classifier,” in _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , ser. KDD ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 1135–1144. [Online]. Available: https://doi.org/10.1145/2939672.2939778
  * [5] V. Petsiuk, A. Das, and K. Saenko, “RISE: randomized input sampling for explanation of black-box models,” in _British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018_. BMVA Press, 2018, p. 151. [Online]. Available: http://bmvc2018.org/contents/papers/1064.pdf
  * [6] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek, “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation,” _PLoS ONE_ , vol. 10, no. 7, p. e0130140, 07 2015.
  * [7] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2261–2269.
  * [8] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , ser. AAAI’17. AAAI Press, 2017, p. 4278–4284.
  * [9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [10] A. M. Tseng, A. Shrikumar, and A. Kundaje, “Fourier-transform-based attribution priors improve the interpretability and stability of deep learning models for genomics,” in _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , ser. NIPS’20. Red Hook, NY, USA: Curran Associates Inc., 2020.
  * [11] J. Engel, L. Hantrakul, C. Gu, and A. Roberts, “DDSP: Differentiable digital signal processing,” _CoRR_ , 2020. [Online]. Available: http://arxiv.org/abs/1907.07374
  * [12] R. Chellappa, “The changing fortunes of pattern recognition and computer vision,” _Image and Vision Computing_ , vol. 55, pp. 3 – 5, 2016, recognizing future hot topics and hard problems in biometrics research. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S026288561630066X
  * [13] N. Thuerey, P. Holl, M. Mueller, P. Schnell, F. Trost, and K. Um, _Physics-based Deep Learning_. WWW, 2021. [Online]. Available: https://physicsbaseddeeplearning.org
  * [14] M. Kellman, “Physics-based learning for large-scale computational imaging,” Ph.D. dissertation, EECS Department, University of California, Berkeley, Aug 2020\. [Online]. Available: http://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-167.html
  * [15] K. Beckh, S. Müller, M. Jakobs, V. Toborek, H. Tan, R. Fischer, P. Welke, S. Houben, and L. von Rüden, “Explainable machine learning with prior knowledge: An overview,” _CoRR_ , vol. abs/2105.10172, 2021. [Online]. Available: https://arxiv.org/abs/2105.10172
  * [16] D. Marr, _Vision: A Computational Investigation into the Human Representation and Processing of Visual Information_. USA: Henry Holt and Co., Inc., 1982.
  * [17] A. B. Watson, “High frame rates and human vision: A view through the window of visibility,” _SMPTE Motion Imaging Journal_ , vol. 122, no. 2, pp. 18–32, 2013.
  * [18] A. Chalmers and K. Debattista, “Hdr video past, present and future: A perspective,” _Signal Processing: Image Communication_ , vol. 54, pp. 49–55, 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S092359651730019X
  * [19] M. Carrasco, “Visual attention: The past 25 years,” _Vision Research_ , vol. 51, no. 13, pp. 1484–1525, 2011, vision Research 50th Anniversary Issue: Part 2. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0042698911001544
  * [20] T. Guionnet, M. Raulet, and T. Burnichon, “Forward-looking content aware encoding for next generation uhd, hdr, wcg, and hfr,” _SMPTE Motion Imaging Journal_ , vol. 129, no. 7, pp. 26–32, 2020.


# Graph Representation Learning on Tissue-Specific Multi-Omics

Amine Amor  Pietro Lio  Vikash Singh  Ramon Viñas Torné  Helena Andres Terre

###### Abstract

Combining different modalities of data from human tissues has been critical in
advancing biomedical research and personalised medical care. In this study, we
leverage a graph embedding model (i.e VGAE) to perform link prediction on
tissue-specific Gene-Gene Interaction (GGI) networks. Through ablation
experiments, we prove that the combination of multiple biological modalities
(i.e multi-omics) leads to powerful embeddings and better link prediction
performances. Our evaluation shows that the integration of gene methylation
profiles and RNA-sequencing data significantly improves the link prediction
performance. Overall, the combination of RNA-sequencing and gene methylation
data leads to a link prediction accuracy of 71% on GGI networks. By harnessing
graph representation learning on multi-omics data, our work brings novel
insights to the current literature on multi-omics integration in
bioinformatics.

Machine Learning, Graph Representation Learning, Variational Graph Auto-
Encoders, Generative Models, Link Prediction, Multi-Omics, ICML

##  1 Introduction

Thorough understanding of human health and pathological conditions requires
the analysis of molecular data at different levels, such as genome, epigenome,
transcriptome, proteome, and metabolome. To account for the interactions
between these omics and study complex biological processes holistically, it is
fundamental to follow an integrative approach which combines multi-omics (i.e
multiple modalities of biological data) (Huang et al., 2017). Integrative
approaches help to evaluate the flow of information from one omic layer to
another, and therefore contribute to bridge the gap between genotype and
phenotype. In the era of precision medicine, high-throughput technologies can
generate very large amounts of multi-omics data, and contribute to improve
prognostics of disease phenotypes. While there has been a significant interest
in building integrative systems in bioinformatics (de Anda-Jáuregui &
Hernández-Lemus, 2020), multi-omics integration on tissue-specific data has
been underexplored. Motivated by the lack of research on tissues functional
diversity, we focus our study on tissue-specific biological data using 3
modalities (i.e omics): Gene-Gene Interaction networks (GGI), RNA sequencing
data and gene methylation profiles. Therefore, our input consists of a network
of interacting genes with their gene expression features (i.e RNA sequencing
and gene methylation).

Overall, the novelty of our work relies on the analysis of tissue-specific
data and the integration of multi-omics features using a graph embedding model
(VGAE).

##  2 Related Work

###  2.1 Tissue-specific research

The heterogeneity of cells across tissues is a major challenge for
understanding biological processes and developing therapeutic targets of
distinct tissues. Although tissue-specific mechanisms are rarely explored,
there have been research initiatives to identify tissue-specific molecular
profiles. Jambusaria et al. (Jambusaria et al., 2018) developed a predictive
model called “HeteroPath” which produces unique tissue-specific gene
regulatory networks. By identifying distinct cellular populations in tissue
transcriptomic datasets, “HeteroPath” contributes to improve the comprehension
of tissue-specific phenotypes. Whereas this study focuses on transcriptomics,
metabolomics have also been investigated in the context of tissue-specific
analysis. For instance, CORDA (Schultz & Qutub, 2016) (Cost Optimization
Reaction Dependency Assessment) is a genome scale model that detects important
metabolic reactions across various human tissues. Using CORDA algorithm, the
authors developed 76 healthy and 20 cancer tissue-specific reconstructions,
and identified metabolic pathways shared across tissues.

We notice that these papers explore metabolomics and transcriptomics
independently, to infer molecular signatures of tissues. Motivated by the
potential complementarity of omics features, our approach incorporates diverse
modalities of omics to provide a more global molecular perspective of distinct
human tissues.

Figure 1: A VGAE (adapted from (Kipf & Welling, 2016)) that performs an
intermediate integration of gene methylation and RNA sequencing features. Both
input networks are tissue-specific Gene-Gene Interaction (GGI) networks. While
they have the same adjacency matrix (from Tissue 1 GGI network), they are
enriched with different node features (i.e gene methylation or RNA sequencing
features).

###  2.2 Graph representation learning on tissue-specific expression data

Biological processes can be described in terms of molecular interactions that
occur across multiple omics layers. This type of data comes in the form of
interaction networks, which have been used to train several graph embedding
models on the prediction of gene-disease associations (Kircali Ata et al.,
2018) (Singh & Liò, 2019) and the identification of molecular signatures
(Kuru, 2019).

Regarding tissue-specific analysis, Ohmnet (Zitnik & Leskovec, 2017) is an
unsupervised node feature learning framework which predicts multicellular
function through multi-layer tissue protein-protein interaction (PPI)
networks. It represents one of the rare initiatives that uses graph embedding
techniques on tissue-specific molecular interactions.

Overall, substantial research was conducted on multi-omics integration
frameworks and graph representation learning. However, to the best of our
knowledge, multi-omics integration on tissue-specific graphs/networks is a
research area that is relatively poor. Therefore this study leverages graph
representation learning on tissue-specific multi-omics data.

##  3 Multi-omics integration with VGAE

###  3.1 Data collection

We collect tissue-specific GGI networks, RNA sequencing data and gene
methylation profiles from 3 public databases:

  * •

HumanBase (GIANT): It is a public database that provides human genomic data
such as gene expression, regulation and interaction networks. From HumanBase,
we collect 5 tissue-specific Gene-Gene Interaction (GGI) networks (Greene et
al., 2015), which were built using gene expression and gene function from a
large compendium of tissues and cell-types.

  * •

The Genotype-Tissue Expression (GTEx) project: Launched by the National
Institutes of Health (NIH) in September 2010, the Genotype-Tissue Expression
project (GTEx (Lonsdale et al., 2013)) is a public resource that gives access
to tissue-specific gene expression and regulation data. The samples were
collected from 54 healthy tissue sites across nearly 1000 participants. From
GTEx, we download 5 tissue-specific filtered and normalised gene expression
matrices (RNA sequencing data).

  * •

MethBank 3.0: MethBank (Li et al., 2017) is a public database that was
developed in 2017 by the Big Data Center of Beijing Institute of Genomics. The
database incorporates 34 consensus reference methylomes derived from 4,577
healthy human samples at different ages. From MethBank, we collect normalised
healthy human gene methylation profiles for 5 tissues.

###  3.2 Variational Graph Auto-Encoder (VGAE)

To perform link prediction on tissue-specific GGI networks, we employ an
unsupervised variational graph autoencoder (VGAE) (Kipf & Welling, 2016) that
integrates distinct latent representations derived from RNA sequencing data
and gene methylation profiles (i.e Z1 and Z2 in Figure 1). The combined
representation is fed into the decoder of the VGAE which aims to reconstruct
the adjacency matrix of the original network. The reconstruction of an
adjacency matrix is also known as the link prediction problem. In the
reconstruction output (in Figure 1), solid lines (positive edges) represent
the existence of a link between 2 nodes, whereas dotted lines (negative edges)
represent the absence of link. In our study, we train our integrative VGAE on
tissue-specific Gene-Gene Interaction networks (GGI) where nodes represent
genes and edges/links represent a functional interaction between genes. In
Figure 1, the boxes represent the feature vectors associated with the genes in
the adjacency matrix. “Meth” represent gene methylation features whereas “RNA”
represent RNA-sequencing features.

As shown on Figure 1, there is a significant gap of dimensionality between RNA
sequencing features (n=208) and gene methylation features (n=9). Indeed, for
each gene, there is a vector of 208 RNA sequencing features and a smaller
vector of 9 methylation features. In order to preserve the unique distribution
of each data type, the integrative VGAE combines the features representations
in the latent space, rather than the input space. The first step consists of
training two separate GCN (Graph Convolutional Neural Network) encoders on a
GGI network enriched with RNA sequencing data and a GGI network enriched with
gene methylation data, respectively. The GCNs encode the features into 2
separate embeddings Z1 and Z1, which have the same dimensions (n=32). Z1 and
Z2 are then concatenated and fed into the rest of the VGAE which performs link
prediction. Since Z1 and Z2 have the same shape, this approach gives the same
weight to methylation and RNA sequencing features, despite their initial
imbalance of dimensions. Additionally, unlike an early integration approach
which combines features at the input level, our model does not increase the
dimensionality of the input space. However, our intermediate integration
requires to train an additional GCN encoder and therefore increases the number
of parameters to learn.

##  4 Evaluation and Results

The experiments aim to evaluate how much each omics data contributes to the
performance of the models. To that end, we conduct an ablation study which
consists of combining multi-omics in three different ways: GGI+RNA, GGI+Meth,
GGI+RNA+Meth. The ablation study helps to assess the individual importance of
each data modality (RNA-sequencing or gene methylation) as well as their
complementarity in achieving link prediction. This provides biological
insights into the relevance of particular omics in learning tissue-specific
representations.

Additionally, we compare the performance of the VGAE to the non-generative
Graph Auto-Encoder (GAE) in order to understand the relevance of generative
models in multi-modal learning on graphs.

###  4.1 Multi-omics integration results

The table illustrates the average link prediction performance of the VGAE on 5
tissue-specific GGI networks. Here, “Bal Acc” refers to the balanced accuracy
metric defined as $(TPR+TNR)/2$.

Integration | Bal Acc | F1 score
---|---|---
GGI | $51(\pm 1.9)$ | $33(\pm 2.5)$
GGI+Meth | $56.7(\pm 3.8)$ | $48.0(\pm 4.5)$
GGI+RNA | $71.4(\pm 3.9)$ | $69.6(\pm 2.2)$
GGI+RNA+Meth | $\textbf{71.6}(\pm 6.1)$ | $\textbf{70.1}(\pm 6.7)$

###  4.2 Generative vs Non-Generative models results

The table shows the average link prediction performance of the VGAE and GAE on
5 tissue-specific GGI networks, using the intermediate integration approach
described in section 3.

Model | Bal Acc | F1 score
---|---|---
GAE | $70.2(\pm 3.0)$ | $68.1(\pm 4.2)$
VGAE | $\textbf{71.6}(\pm 4.1)$ | $\textbf{70.1}(\pm 4.7)$

##  5 Discussion

###  5.1 Multi-omics integration

We discuss the results obtained from different types of multi-omics
integration in order to understand the value of each omics in achieving link
prediction on tissue-specific networks. On non-enriched GGI (Gene-Gene-
Interaction) networks, the VGAE achieves a very poor performance, which
highlights the importance of node features to learn informative graph
embeddings. By adding gene methylation node features (GGI+Meth), we observe a
notable improvement of the overall performance. The balanced accuracy grows
from 50% to 57% and the F1 score grows from 33% to 48%. On the other hand,
augmenting the networks with RNA node features (GGI+RNA) brings a considerable
enhancement in the link prediction performance. The incorporation of RNA
features causes the balanced accuracy and F1 score to increase from 50% to
70-71%. These results suggest that RNA sequencing features are more valuable
than gene methylation features and lead to more accurate graph embeddings on
tissue-specific GGI networks. While both RNA and methylation features enhance
the prediction performance of the VGAE, their combination (GGI+RNA+Meth) is
not particularly complementary for link prediction. Indeed, the VGAE’s
performance on GGI+RNA+Meth is almost equal to its performance on GGI+RNA.

###  5.2 Generative vs Non-Generative models

On the other hand, we observe that the VGAE results in a higher link
prediction performance than the GAE. Indeed, the balanced accuracy and F1
score are roughly 1-2% higher in the case of the VGAE. The higher performance
of the VGAE shows the benefits of latent space regularisation. By enforcing
the latent distribution to be close to a gaussian distribution, the VGAE
regularises the latent space and enables a better generalisation performance.
Moreover, the VGAE provides flexibility in the learning process because we can
tune the KLD loss with a parameter $\alpha$ and the reconstruction loss with a
parameter $\beta$. Increasing $\alpha$ would augment the generative power of
the VGAE whereas increasing $\beta$ would further optimise the reconstruction
performance. Overall, these results highlight the relevance of generative
models in performing multi-modal learning on multi-omics networks.

##  6 Conclusion

In summary, our work explores multi-modal learning on tissue-specific gene-
gene interaction (GGI) networks. Our approach towards multi-omics integration
consists of enriching GGI networks with RNA-sequencing and gene methylation
features. Since omics modalities are collected separately across distinct
tissues, our data is tissue-specific. In order to learn powerful molecular
representations, we decide to leverage graph embedding models (i.e VGAE) which
have the benefit of being scalable to the incorporation of multiple omics
modalities. By evaluating our VGAE model on the addition and the removal of
omics features, we conduct an ablation study that provides insights into the
benefits of each omics data type (i.e RNA-sequencing and gene methylation).

We observe that the performance of the model increases significantly with the
integration of gene methylation profiles and RNA features. Additionally, we
discover that RNA features lead to a higher improvement than methylation
profiles, which suggests that RNA-sequencing data is more insightful for
learning tissue-specific molecular signatures.

On the other hand, the VGAE outperforms the non-generative GAE, which reveals
the potential of generative models in multi-modal learning on graphs. Overall,
our integrative VGAE achieves a link prediction accuracy of 71% on the multi-
omics networks (GGI+RNA+Meth), which proves its ability to compress high-
dimensional biological networks into informative low-dimensional embeddings.

Overall, this study highlights the benefits of multi-omics integration for
link prediction on biological networks. Our insights are based on a
variational graph auto-encoder (VGAE) which extracts low-dimensional
representations from healthy tissue-specific GGI networks. These
representations can serve to enrich existing biological datasets and
contribute to downstream supervised tasks such as the detection of bio-markers
and the identification of tissue-specific diseases.

##  7 Future Work

This study shows novel insights into the benefits of multi-omics integration
in bioinformatcs. For future work, our approach could be leveraged to target a
concrete application in prognostics, such as the detection of breast cancer.

Based on our approach, we could collect multi-omics data from breast tissues
and train our VGAE model to distinguish between healthy breast representations
and cancer breast representations. Since our models are scalable and flexible
to the integration of heterogeneous omics features, the prediction of breast
cancer would only require to change the multi-omics input data. The omics data
could be specific to 2 classes: “Healthy breast tissues” and “Diseased breast
tissues”. Additionally, the tissue-specific representations learnt on breast
cancer could be used by downstream machine learning classifiers to perform
more specialised predictions, such as identifying breast cancer molecular
subtypes (Singh et al., 2018).

More generally, our proposed VGAE is interdisciplinary and can be harnessed to
perform multi-modal learning on any task involving graph structures (e.g
social networks and graph recommendation systems).

## References

  * de Anda-Jáuregui & Hernández-Lemus (2020) de Anda-Jáuregui, G. and Hernández-Lemus, E.  Computational oncology in the multi-omics era: State of the art.  _Frontiers in Oncology_ , 10:423, 2020.  ISSN 2234-943X.  doi: 10.3389/fonc.2020.00423.  URL https://www.frontiersin.org/article/10.3389/fonc.2020.00423.
  * Greene et al. (2015) Greene, C., Krishnan, A., Wong, A., Ricciotti, E., Zelaya, R., Himmelstein, D., Zhang, R., Hartmann, B., Zaslavsky, E., Sealfon, S., Chasman, D., FitzGerald, G., Dolinski, K., Grosser, T., and Troyanskaya, O.  Understanding multicellular function and disease with human tissue-specific networks.  _Nature genetics_ , 47, 04 2015.  doi: 10.1038/ng.3259.
  * Huang et al. (2017) Huang, S., Chaudhary, K., and Garmire, L. X.  More is better: Recent progress in multi-omics data integration methods.  _Frontiers in Genetics_ , 8:84, 2017.  ISSN 1664-8021.  doi: 10.3389/fgene.2017.00084.  URL https://www.frontiersin.org/article/10.3389/fgene.2017.00084.
  * Jambusaria et al. (2018) Jambusaria, A., Klomp, J., Hong, Z., Rafii, S., Dai, Y., Malik, A., and Rehman, J.  Additional file 1: of A computational approach to identify cellular heterogeneity and tissue-specific gene regulatory networks.  6 2018.  doi: 10.6084/m9.figshare.6679493.v1.  URL https://springernature.figshare.com/articles/Additional_file_1_of_A_computational_approach_to_identify_cellular_heterogeneity_and_tissue-specific_gene_regulatory_networks/6679493.
  * Kipf & Welling (2016) Kipf, T. N. and Welling, M.  Variational graph auto-encoders, 2016.
  * Kircali Ata et al. (2018) Kircali Ata, S., Ou-Yang, L., Fang, Y., Kwoh, C.-K., Wu, M., and li, X.  Integrating node embeddings and biological annotations for genes to predict disease-gene associations.  _BMC Systems Biology_ , 12, 12 2018.  doi: 10.1186/s12918-018-0662-y.
  * Kuru (2019) Kuru, H. I.  Graph embeddings on protein interaction networks.  _Bilkent University Institutional Repository_ , 02 2019.  URL http://repository.bilkent.edu.tr/handle/11693/49202.
  * Li et al. (2017) Li, R., Liang, F., Li, M., Zou, D., Sun, S., Zhao, Y., Zhao, W., Bao, Y., Xiao, J., and Zhang, Z.  Methbank 3.0: A database of dna methylomes across a variety of species.  _Nucleic acids research_ , 46, 11 2017.  doi: 10.1093/nar/gkx1139.
  * Lonsdale et al. (2013) Lonsdale, J., Thomas, J., Salvatore, M., Phillips, R., Lo, E., Shad, S., Hasz, R., Walters, G., Garcia, F., Young, N., Foster, B., Moser, M., Karasik, E., Gillard, B., Ramsey, K., Sullivan, S., Bridge, J., Magazine, H., Syron, J., and Moore, H.  The genotype-tissue expression (gtex) project.  _Nature genetics_ , 45:580–585, 05 2013.  doi: 10.1038/ng.2653.
  * Schultz & Qutub (2016) Schultz, A. and Qutub, A. A.  Reconstruction of tissue-specific metabolic networks using corda.  _PLOS Computational Biology_ , 12:1–33, 03 2016.  doi: 10.1371/journal.pcbi.1004808.  URL https://doi.org/10.1371/journal.pcbi.1004808.
  * Singh et al. (2018) Singh, A., Shannon, C. P., Gautier, B., Rohart, F., Vacher, M., Tebbutt, S. J., and Lê Cao, K.-A.  Diablo: from multi-omics assays to biomarker discovery, an integrative approach.  _bioRxiv_ , 2018.  doi: 10.1101/067611.  URL https://www.biorxiv.org/content/early/2018/03/20/067611.
  * Singh & Liò (2019) Singh, V. and Liò, P.  Towards probabilistic generative models harnessing graph neural networks for disease-gene prediction.  _CoRR_ , abs/1907.05628, 2019.  URL http://arxiv.org/abs/1907.05628.
  * Zitnik & Leskovec (2017) Zitnik, M. and Leskovec, J.  Predicting multicellular function through multi-layer tissue networks.  _CoRR_ , abs/1707.04638, 2017.  URL http://arxiv.org/abs/1707.04638.