# DECIFE: Detecting Collusive Users Involved in Blackmarket Following Services
on Twitter

Hridoy Sankar Dutta<EMAIL_ADDRESS>IIIT-DelhiIndia ,  Kartik Aggarwal
<EMAIL_ADDRESS>IIIT-DelhiIndia and  Tanmoy Chakraborty
<EMAIL_ADDRESS>IIIT-DelhiIndia

(2021)

###### Abstract.

The popularity of Twitter has fostered the emergence of various fraudulent
user activities - one such activity is to artificially bolster the social
reputation of Twitter profiles by gaining a large number of followers within a
short time span. Many users want to gain followers to increase the visibility
and reach of their profiles to wide audiences. This has provoked several
blackmarket services to garner huge attention by providing artificial
followers via the network of agreeable and compromised accounts in a collusive
manner. Their activity is difficult to detect as the blackmarket services
shape their behavior in such a way that users who are part of these services
disguise themselves as genuine users.

In this paper, we propose DECIFE, a framework to detect collusive users
involved in producing ‘following’ activities through blackmarket services with
the intention to gain collusive followers in return. We first construct a
heterogeneous user-tweet-topic network to leverage the follower/followee
relationships and linguistic properties of a user. The heterogeneous network
is then decomposed to form four different subgraphs that capture the semantic
relations between the users. An attention-based subgraph aggregation network
is proposed to learn and combine the node representations from each subgraph.
The combined representation is finally passed on to a hypersphere learning
objective to detect collusive users. Comprehensive experiments on our curated
dataset are conducted to validate the effectiveness of DECIFE by comparing it
with other state-of-the-art approaches. To our knowledge, this is the first
attempt to detect collusive users involved in blackmarket ‘following services’
on Twitter.

Followers, collusion, blackmarket, Twitter, OSNs

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
32nd ACM Conference on Hypertext and Social Media; August 30-September 2,
2021; Virtual Event, Ireland††booktitle: Proceedings of the 32nd ACM
Conference on Hypertext and Social Media (HT ’21), August 30-September 2,
2021, Virtual Event, Ireland††price: 15.00††doi:
10.1145/3465336.3475108††isbn: 978-1-4503-8551-0/21/08††ccs: Information
systems Social networks††ccs: Security and privacy

##  1\. Introduction

Are you a Twitter user? Do you want to boost your Twitter profile (by
increasing the follower count) within a limited time without getting suspended
by Twitter? Several online services are ready to assist you. What you need to
do is simple – pay them, and they will provide you followers; most of these
followers would be legitimate Twitter users. If you can not afford to pay
money, then you can opt for another option – just become a part of these
services, start following their customers and earn credits; these credits can
be used further to gain your own followers. Do not worry about being flagged
by Twitter policy as these services are so smart in their following mechanism
that they can easily deceive Twitter into thinking that their activity is
legitimate. This is the philosophy behind many online blackmarket following
services.

Figure 1. A heterogeneous network for modeling collusive users and their
interactions. (a) Three types of nodes. (b) User-tweet-topic heterogeneous
network. (c) Three types of edges involved in the network. The detailed
construction of the heterogeneous network can be found in Section 4.1.

Twitter is arguably the most popular Online Social Network (OSN) for mass
communication. It is also one of the key platforms for digital campaigning,
social networking and opinion dissemination. The popularity of Twitter has
attracted with it new online markets that help its users to make their
profiles attractive by increasing followers, retweets, likes, replies, etc. In
this context, Stringhini et al. (Stringhini et al., 2013) coined the term
“Twitter followers market” to characterize those syndicates catering to people
willing to pay for a quick increase of their followers. Many people try to
rapidly gain fame by exploiting this mechanism – they buy followers from these
online markets. However, the coordinators of these services manage the
activities of users to maintain the integrity and avoid societal, privacy and
security problems. Controlling such type of artificial following activities
has thus become one of the major challenges. There exist several blackmarket
agencies which have created thriving and intelligent ecosystems of producing
illicit followers. Users can gain such followers by paying money (premium
services) or for free by following customers of those services (freemium
services) (Shah et al., 2017). In this paper, we focus our attention on the
latter case, and call this type of users “collusive users” – users who gain
followers from blackmarket services. Previous literature (Dutta et al., 2018;
Dutta and Chakraborty, 2019; Arora et al., 2020; Dutta et al., 2020a; Chetan
et al., 2019) on collusive entities in online media reported that collusive
users are not ‘fake’ – they are not bots, the handles of these Twitter
profiles are normal human beings, and their following activities are not
mechanically controlled by any predefined policy. Rather, collusive users
exhibit a hybrid following behavior – on one hand, similar to non-collusive
users, they organically follow other users due to similar topical interest; on
the other hand, similar to fake users, they inorganically follow other
blackmarket customers only to gain credits. Designing a system to identify
collusive users would be useful for Twitter managers and social network
analysts to figure out how and what extent the profile of a user has turned
out to be popular due to the support of such blackmarket services.

In this work, we propose DECIFE, a novel heterogeneous graph attention network
for detecting collusive users who are involved in producing fake followers. We
first create a heterogeneous network (c.f. Figure 1) based on the relationship
and linguistic properties of users. We then decompose the network into four
different subgraphs to capture semantic relations between users. Finally, we
exploit a hierarchical aggregation network to learn node representations that
are passed on to a hypersphere learning objective to detect collusive users.
To evaluate our method, we collected data of collusive users from a popular
blackmarket service by designing a customized web scraper. The entire data
collection was carried out after taking proper IRB (institutional review
board) approval from our institute. We show the effectiveness of our proposed
DECIFE model by comparing it with three baseline methods –
FakeFolss(Castellini et al., 2017), FakeFols(Castellini et al., 2017),
FolMarket (Aggarwal and Kumaraguru, 2015) and three individual components of
DECIFE considered in isolation (ablation study).

In summary, the major contributions of the paper are four-fold:

  1. (1)

We deal with a novel problem of detecting collusive users in Twitter where
normal users get involved in artificial following activities through
blackmarket services for boosting their online profiles. To the best of our
knowledge, no existing work has investigated the problem of ‘collusive user
detection involved in blackmarket following services’ on Twitter.

  2. (2)

We introduce DECIFE, a novel framework to detect collusive users on Twitter.
It uses a hierarchical subgraph aggregation framework to leverage the
follower/followee relationships and linguistic properties of users.

  3. (3)

We prepare a new dataset of collusive users who are involved in blackmarket
following services on Twitter. This, to our knowledge, is the first dataset of
this kind.

  4. (4)

We conduct extensive experiments on the curated dataset of collusive users to
show the superiority of our method over state-of-the-art approaches.

Reproducibility: To encourage reproducible research, we have made the codes
and the anonymized version of the dataset publicly available at
https://github.com/LCS2-IIITD/DECIFE.

##  2\. Related Work

We discuss the related literature by dividing the existing work into two parts
– (i) detection of fake followers in OSNs, and (ii) study of blackmarket
services in OSNs.

###  2.1. Detection of Fake Followers in OSNs

Most of the approaches to detect fake followers identify a set of features and
use machine learning techniques. Wiltshire, in her
blog111https://www.gshiftlabs.com/social-media-blog/the-fake-followers-
epidemic, discussed the population of fake followers and found that there is
an increase of 1-3% fake followers for a Twitter account every couple of
months. Cresci et al. (2015) mentioned that there are two reasons to buy fake
followers: increase visibility and push advertisements. A tool, called “Fake
Followers Check”222https://bit.ly/2KzMRdd was developed to detect fake
followers on Twitter based on the ratio of friends and followers, usage of
repeated spam phrases, count of retweets, etc. Cresci et al. (2015) proposed a
machine learning approach to detect fake followers using multiple features and
different machine learning classifiers. Mehrotra et al. (2016) used
centrality-based graph features to detect fake Twitter followers. Lee et al.
(2015) found retweeters who can be used to spread the message effectively
among different group of people. Shah et al. (2014) proposed fBox, an
algorithm to detect suspicious friend and follower links on a large who-
follows-whom Twitter dataset. Cresci et al. (2014) developed a classifier for
fake follower detection using profile, timeline and relationship based
features. Jiang et al. (2014) proposed CatchSync to detect suspicious nodes
(followers and botnets) exhibiting synchronized behavior on Twitter social
network. Kwak et al. (2011) studied the dynamics of unfollow behavior in
Twitter based on online relationships of Korean-speaking Twitter users.
Aggarwal et al. (2018) identified users with increased follower count using
unsupervised local neighborhood detection method. Shen and Liu (2015) proposed
supervised spammer detection method with social interaction to detect spammers
on twitter based on content and social interaction. Li et al. (2014) detected
campaign promoters by mapping the problem to relational classification and
solved it using typed Markov Random Fields. Ferrara et al. (2016) discussed
how social bots which interact with humans and get unnoticed have risen in the
present scenario. Shen et al. (2014) distinguished fake followers from the
legitimate users using several discriminative features. Zhang and Lu (2016)
used a network-based strategy to identify fake followers. Zhang et al. (2016)
focused on the detection of zombie followers in Sina Weibo.

###  2.2. Study of Blackmarket Services in OSNs

Blackmarket services have gained substantial attention recently because of the
techniques they use to provide services to the customers. Stringhini et al.
(2013) was the first to analyze the Twitter follower markets based on the
market size and market price. A detailed analysis of blackmarket services is
presented in (De Micheli and Stroppa, 2013; Song et al., 2015) with the impact
on multiple OSNs. Most of the prior studies showed how fake followers in
social media help in promoting different agenda (Stringhini et al., 2013,
2012; Aggarwal and Kumaraguru, 2015). Farooqi et al. (2017) showed how
collusion networks collect ‘OAuth’ access tokens from colluding members and
abuse them to provide fake likes or comments to their members. Zhu et al.
(2016) proposed an automated approach to detect collusive behavior in
question-answering systems. Weerasinghe et al. (2020) studied models for
detecting Instagram posts that gained interaction through collusive networks.
Aggarwal and Kumaraguru (2015) also discovered an oligopoly structure of
merchants involved in blackmarket services. Shah et al. (2017) studied
multiple types of blackmarket agencies and analyzed a honeypot fraudster
ecosystem to provide insights about multifaceted behaviour of different
fraudsters. Motoyama et al. (2011) analyzed structure of social networks
present on six different underground forums to understand the social dynamics
of e-crime markets. Thomas (2013) studied various blackmarkets and developed a
classifier to detect fraudulent accounts sold by these marketplaces. Singh et
al. (2016) studied the behavioral characteristics of Twitter follower market
merchants based on user and content based features. Liu et al. (2016) detected
‘volowers’ (followers who provide voluntary following services) who make
profit in the follower markets. Recently, there are some preliminary works of
collusive user detection on Twitter and YouTube. Dutta et al. (2018); Dutta
and Chakraborty (2019); Arora et al. (2020) proposed techniques to detect
collusive users involved in blackmarket-based retweeting service on Twitter.
Dutta et al. (2020b) proposed CollATe, an end-to-end framework to detect
collusive entities on YouTube fostered by various blackmarket services. We
encourage the readers to go through (Dutta and Chakraborty, 2020) for a
comprehensive survey on collusive activities in different online media
platforms.

Differences with Previous Studies: The fundamental differences between the
studies discussed above on fake follower detection and the collusive user
detection are two-fold: (i) unlike fake followers who are mostly bots or whose
activities are mechanically controlled by predefined policies, collusive users
are normal human beings, and they themselves control their accounts.
Therefore, unlike fake users who mostly show “synchronous behavior” (Jiang et
al., 2014), collusive users are asynchronous in nature (Dutta et al., 2018).
(ii) Unlike fake followers, collusive users exhibit a hybrid following
behavior – in one hand, being a normal user, they follow other users
organically due to similar topical interest; on the other hand, being a
collusive user, they randomly follow other blackmarket customers inorganically
to gain credits (Dutta and Chakraborty, 2019). To the best of our knowledge,
ours is the first work to detect collusive users who gain artificial followers
from blackmarkets.

##  3\. Background and Dataset

Approaching blackmarket services is one of the quickest ways to boost the
impact of users on social media. These blackmarket services offer promotional
services on multiple online media such as OSNs (e.g., likes on Facebook;
followers, retweets, likes on Twitter; followers on Instagram), subscription-
sharing platforms (e.g., views, subscribers, likes on Youtube), music-sharing
platforms (e.g., plays, followers, likes, reposts, comments on SoundCloud,
fans on ReverbNation), business and employment-oriented platforms (e.g.,
followers, connections, endorsements on LinkedIn), etc. We identified the
blackmarket services providing collusive follower appraisals for Twitter by
querying on search engines with keywords such as “buy free followers”, “get me
followers”, “get followers quickly”. To collect collusive users, we selected
YouLikeHits333https://www.youlikehits.com/twitter2.php, a popular credit-based
freemium blackmarket service444Though there exist several blackmarket services
to get collusive appraisals, we choosed only YouLikeHits for our study due to
its popularity and extensive amount of literature (Dutta et al., 2018; Dutta
and Chakraborty, 2019; Dutta et al., 2020b) investigating this service..
Customers on navigating to these services can see a dashboard (earning area)
of other customers who are also using that service. YouLikeHits provide the
customers with an initial credit of $50$ which can be utilized to use various
facilities offered by the service. After taking proper IRB approval from our
institute, we developed a web scraper that used
Selenium555https://www.seleniumhq.org/ to start a headless web browser
navigating to the URL of the blackmarket service. We used popular Python
packages such as Requests, BeautifulSoup etc. to parse the Twitter user IDs
who submitted their profile for collusive follower appraisals. We further used
the Tweepy library666https://github.com/tweepy/tweepy to collect the metadata
and timeline of the Twitter users. Note that we also collect a set of non-
collusive users who surely have not participated in any kind of blackmarket-
driven activities to gain artificial appraisals. This set of users is only
collected for the test phase of our experiment and is not the part of original
dataset.

Figure 2. Architecture of our proposed DECIFE model for collusive user
detection on Twitter.

##  4\. DECIFE: Our Proposed Model

The objective of collusive user detection task is to learn how to
automatically identify Twitter users who submitted their accounts to
blackmarket services to gain collusive followers. Formally, given a set of
collusive users $u_{i}=\\{u_{1},u_{2},\dots\\}$, connected via relationships
$\Delta_{m}=\\{\Delta_{1},\Delta_{2},\dots\\}$ in a heterogeneous network
$\mathcal{G}$, DECIFE learns a hypersphere boundary $\mathcal{S}(r,c)$ (with
radius $r$ and center $c$) around the collusive users to identify whether a
new user is a collusive user or not. In this section, we present the
architecture of DECIFE. It consists of four major components: network
construction, feature extraction, hierarchical subgraph aggregation and
collusive user detection. In this section, we introduce each of these
components in detail. Figure 2 shows the schematic architecture of DECIFE.

###  4.1. Network Construction

Here, we show the construction of the heterogeneous network and subsequent
subgraphs for modeling the interactions of Twitter users.

Heterogeneous Network:  To model the Twitter network, we construct a
heterogeneous network (Sun and Han, 2013) as a directed network
$\mathcal{G}=\mathcal{(V,E)}$, where each node $v\in\mathcal{V}$ and each edge
$e\in\mathcal{E}$ are related with their node-type mapping functions
$\rho(v):\mathcal{V}\rightarrow\mathcal{X}$ and edge-type mapping functions
$\varphi(e):\mathcal{E}\rightarrow\mathcal{Y}$, respectively. $\mathcal{X}=$
{user($U$), tweet($T$), topic($O$)} denote the node-types and $\mathcal{Y}=$
{follows($r$), posts($p$), contains($c$)} denote the edge-types such that
$|\mathcal{X}|+|\mathcal{Y}|>2$ (a heterogeneous network contains at least 2
node-types or edge-types). An illustration of this network is shown in Figure
1. Every Twitter user $u_{i}\in U$ is represented with a feature vector
$x_{i}$ in $\mathcal{G}$ and is connected with its corresponding tweets
$t_{ij}\in T$ and to users she follows or is followed by. Every tweet $t_{ij}$
of a user $u_{i}$ is connected to a topic node $o_{k}\in O$ via one-to-one
mapping, where $k$ is the set of topics.

Network Decomposition: The heterogeneous network is decomposed into subgraphs
based on multiple relationships. A Relationship $\Delta$ between a
labelled777We refer to a labelled user as a user whose ground-truth label is
known. In our case, it refers to a collusive user which we collected from the
blackmarket service. user-node $u_{i}$ and another labelled user-node $u_{j}$
is a sequence of connected relations
$\mathcal{M}_{1}\rightleftharpoons\mathcal{M}_{2}\dots\rightleftharpoons\mathcal{M}_{n}$
in $\mathcal{G}$. Each relation $\mathcal{M}$ from source node
$v_{1}\in\mathcal{V}$ to target node $v_{2}\in\mathcal{V}$ with edge
$e\in\mathcal{E}$ is denoted as
$\langle\mathcal{X}(v_{1}),\mathcal{Y}(e),\mathcal{X}(v_{2})\rangle$. Note
that $(v_{1},v_{2})$ can belong to any node-type connected via edge $e$.

To capture different aspects of a user’s behavior, we use four types of
relationships, $\Delta_{m},m=\\{1,2,3,4\\}$ which are detailed below:

  1. (1)

$\Delta_{1}$: To describe a connection between users in the ‘following’
blackmarket services, we formulate a relationship having a common follower.

user${}_{1}\xrightarrow[]{follows}$user${}_{x}\xleftarrow[]{follows}$user2

  2. (2)

$\Delta_{2}$: Users in a blackmarket service tend to participate in a barter
system where they follow other users to gain credits that can later be used to
obtain followers of their accounts. As an example, a transition user (userx)
signed up on the service, follows users (user1) to gain credits which can be
availed to gain followers (user2) in return. To exploit this characteristic,
we formulate a relationship between users in a 1-hop manner.

user${}_{1}\xleftarrow[]{follows}$user${}_{x}\xleftarrow[]{follows}$user2

  3. (3)

$\Delta_{3}$: To describe a direct connection between two users, we formulate
a relationship in a 0-hop manner.

user${}_{1}\xrightarrow[]{follows}$user2

  4. (4)

$\Delta_{4}$: Twitter users involved in the promotion of campaigns/ websites/
events try to gain popularity from a particular target audience by gaining
more retweets/followers to their tweets/accounts. To capture the connection
between users tweeting/retweeting on the same topic, we formulate a
relationship based on their common topic of interest. We detail this strategy
of connecting users based on topical interest in Section 5.2.

user${}_{1}\xrightarrow[]{posts}$tweet${}_{1}\xrightarrow[]{contains}$topic${}_{1}\xleftarrow[]{contains}$tweet${}_{2}\xleftarrow[]{posts}$user2

###  4.2. Features Extraction

In a Twitter user network, the behaviours and attributes of an account are
directly linked with the user’s characteristics. Taking this into account, we
consider 18 features which are directly extracted or calculated from the user
metadata. We describe each feature, its corresponding description and type in
Table 1. We then concatenate these features to form an 18-dimension user-
metadata feature vector that is used to train our DECIFE model. The user-
metadata feature vectors, along with subgraph information, are fused at
2-levels using Hierarchical-Subgraph Aggregation (HSA) network to obtain final
user representations.

Table 1. User metadata features used by DECIFE.

Feature |  Description | Type
---|---|---
Follower count |  Number of followers of a user | Real
Friend count |  Number of friends of a user | Real
Status count |  Number of tweets posted and retweeted by a user | Real
Favorite count |  Number of status liked by a user | Real
Description presence |  If the account description is present | Binary
Description length |  Number of characters present in the user description | Real
URL |  If a URL is present in the user description | Binary
Location |  If the location is present in the user info | Binary
Profile image |  If a profile image is present in the user account | Binary
Background image |  If a background image is present in the user account | Binary
Account age |  Number of days elapsed since the account was created | Real
Account entropy |  Skewness in user’s activity, $\sum{c_{i}\log(c_{i})}$, where $c_{i}$ is the year-wise count of tweets by the user | Real
Emojis |  Avg. number of emojis per tweet posted by a user | Real
Retweets |  Ratio of retweet count to tweet+retweet count | Real
URLs |  Avg. number of URLs per tweet by a user | Real
User mentions |  Avg. number of user mentions per tweet by a user | Real
Words |  Avg. number of words in a tweet by a user | Real
Hashtags |  Avg. number of hashtags in a tweet by a user | Real

###  4.3. Hierarchical Subgraph Aggregation (HSA)

We employ a two-level hierarchical subgraph aggregation mechanism to detect
whether a user is collusive or not. In the first step, a weighted subgraph
convolution network is used to obtain hidden representations of the labelled
users based on the connections of a user node $u_{i}$ with its associated
users $u_{j},j\in\mathcal{N}_{i}^{m}$ ($\mathcal{N}^{m}_{i}$ being the set of
neighbors of users $u_{i}$ connected via relationship $\Delta_{m}$). Next, the
subgraph attention network learns the importance of different edge-types to
aggregate information among all nodes in the network.

####  4.3.1. Subgraph Convolution Network:

Considering that a pair of users in a subgraph can be connected via multiple
edges (e.g., two users posting multiple different tweets on the same topic),
we propose a weighted subgraph convolution network with the following layer
propagation rule:

(1)
$h_{i}^{m}=\sigma(b^{m}+\sum_{j\in\mathcal{N}_{i}}\frac{e^{m}_{ij}}{c^{m}_{ij}}x_{j}w^{m})$

where $e^{m}_{ij}$ is the edge-weight coefficient for the edge $e_{ij}$
connected via relationship $\Delta_{m}$ (c.f. Table 3), and $\mathcal{N}_{i}$
is the set of neighbors of node $i$. Further,
$c^{m}_{ij}=\sqrt{|\mathcal{N}_{i}||\mathcal{N}_{j}|}$ is the normalization
term, and $|\mathcal{N}_{i}|$ is the degree of node $i$. Here, $x_{j}$ is the
input feature vector of node $j$, and $w^{m}$ is the learnable weight
parameter for the subgraph $m$. The weight coefficient in subgraph convolution
network makes it superior for learning the connectivity information between
the user-nodes than a conventional graph convolution network.

In generalised-matrix form, the equation can be written as:

(2) $H^{m}=\sigma((D^{m})^{-1}A^{m}K^{m}XW^{m})$

where $A^{m}$ is the adjacency matrix, $K^{m}$ is the edge-weight matrix,
$D^{m}_{ii}=\sum_{j\in\mathcal{N}_{i}}A^{m}_{ij}$ is the diagonal matrix, and
$W^{m}$ is the learnable layer-wise weight matrix of a subgraph having
relationship $\Delta_{m}$.

Therefore, we obtain 4 groups of subgraph-specific node-embeddings $H^{m}$ for
each given relationship $\Delta_{m},m=\\{1,2,3,4\\}$. These are then used to
compute importance among subgraphs using Subgraph Attention Network as
described below.

####  4.3.2. Subgraph Attention Network:

Subgraph convolution network learns a node representation by leveraging the
importance between users only from a single type of relationship. However,
attending to multiple relationships at once can reveal the importance of
different edge-types to a node pair and aggregate rich semantic information
from them. The subgraph-specific node-embeddings $H^{m}$ obtained from
subgraph convolution network are projected using a multilayer feed-forward
neural network to undergo a non-linear transformation.

(3) $\tilde{H}^{m}=W_{1}\cdot H^{m}_{i}+b_{1}$ (4)
$w^{m}=\frac{1}{n}\sum_{i}^{n}W_{2}(\text{tanh}(\tilde{H}^{m}))+b_{2}$

where $W_{1},W_{2}$ are the weight matrices and $b_{1},b_{2}$ are the bias
vectors.

The weight coefficient $\beta^{m}$ of each subgraph is calculated by
normalizing the importance $w^{m}$ using a softmax function.

(5) $\beta^{m}=\frac{\text{exp}(w^{m})}{\sum_{k=1}^{n}\text{exp}(w_{k})}$

The information from both the levels can be aggregated to obtain the final
embeddings as follows:

(6) $Z=\sum_{m=1}^{3}\beta^{m}\cdot H^{m}$

To improve the stability of the training process, we extend this to multiple
heads $d=\\{d_{1},d_{2},\dots,d_{l}\\}$ by passing $Z^{l-1}$ as an input
through the Hierarchical Subgraph Aggregation Network,

(7) $Z^{l}=HSA(Z^{l-1})$

The final embedding $Z^{l}$ of each user-node is used to learn a hypersphere
boundary around the collusive distribution.

###  4.4. Collusive User Detection

Given a training set $\\{Z_{n}=Z_{1},Z_{2},..,Z_{n}\\}\in\mathbf{R}^{n\times
d}$ with embeddings of dimension $d$ belonging to $n$ collusive users
$u_{i}\in U$ derived by hierarchical aggregation, our DECIFE model is trained
to learn a hypersphere boundary that encloses the network representations
around majority of the collusive data. The objective is to obtain a minimum
volume hypersphere $S(r,c)$ with radius $r>0$ and center $c\in\mathcal{F}_{k}$
using a mapping function $\phi_{k}:Z\rightarrow\mathcal{F}_{k}$ to map the
feature space $Z$ into Hilbert space $\mathcal{F}_{k}$. The loss $\mathcal{L}$
is similar to the DeepSVDD (Ruff et al., 2018) and is defined as follows:

(8) $\mathcal{L}=\enskip r^{2}+\frac{1}{\mu
n}\sum_{i=1}^{n}{\mathcal{\xi}_{i}}\\\ \hskip
11.38109pts.t.\quad{||\phi_{k}(x_{i})-c||}^{2}_{\mathcal{F}_{k}}\leq
r^{2}+\mathcal{\xi}_{i},\quad\mathcal{\xi}_{i}\geq 0\enskip\forall\enskip i$

The slack variables $\mu\in(0,1]$ regulates the trade-off between penalties
$\mathcal{\xi}_{i}$ and the volume of sphere $S(r,c)$, and $\mathcal{\xi}_{i}$
allows a smooth boundary of the sphere. The initial radius is set as 0, and
the center is obtained through the first forward pass to our network model.

(9) $c_{0}=\frac{1}{n}\sum\texttt{DECIFE}(\mathcal{G})$

During validation and testing, a point is marked as collusive user if it falls
inside the hypersphere $S$ such that
$\quad{||\phi_{k}(x_{i})-c||}^{2}_{\mathcal{F}_{k}}>r^{2}$.

##  5\. Experimental Setup

We conduct an extensive evaluation of DECIFE on our collected dataset
(mentioned in Section 3). In this section, we start by briefly describing the
baseline methods, followed by detailed evaluation.

###  5.1. Baseline Methods

We consider three state-of-the-art methods – first two focus on fake follower
detection and the third one focuses on Twitter blackmarket follower services.
However, since these methods are not focused towards a network-based detection
approach, we conduct an ablation study by considering one relationship at a
time in our model. We also add these methods for comparing with DECIFE, which
is a combination of all the relationships.

FakeFolss: Castellini et al. (Castellini et al., 2017) detected fake Twitter
followers in a semi-supervised fashion by running denoising autoencoder. Their
model is trained on the data of only one class (5% of collusive users).

FakeFols: Castellini et al. (Castellini et al., 2017) further introduced a
supervised approach to compare it with semi-supervised approach mentioned in
FakeFolss. They used a bunch of features related to user profile and user
activity, and showed Random Forest to be the best classifier.

FolMarket: Aggarwal et al. (Aggarwal and Kumaraguru, 2015) characterized
underground blackmarket services which provide Twitter followers. They
explored three units of blackmarket services – merchants, customers and phony
followers. They also built a supervised classifier (SVM with RBF kernel) to
classify non-collusive users and phony followers.

Variants of DECIFE: We also derive multiple variants of DECIFE to
comprehensively compare and analyze the performances of each component. These
variants are DECIFE${}_{\Delta_{1}}$, DECIFE${}_{\Delta_{2}}$,
DECIFE${}_{\Delta_{3}}$ and DECIFE${}_{\Delta_{4}}$. Note that $\Delta_{i}$
refers to relationships mentioned in Section 4.1.

###  5.2. Implementation Details

####  5.2.1. Sentence Embeddings:

Since Twitter is a global social-media platform and the intention of the
collusive users is to earn followers regardless of the community, language
identification of all tweets using Langid (Lui and Baldwin, 2012) revealed a
mix of languages in the user timelines. Therefore, to capture the semantics of
the tweets in different languages, we use Language Agnostic Sentence Embedding
representations (LASER) (Artetxe and Schwenk, 2019) to build cross-lingual
features of the users’ tweets. For each user with $\delta$ historic tweets, we
obtain a $\delta\times 1024$ dimensional embeddings. These multilingual
embeddings are finally passed through a cosine-similarity based K-means
clustering algorithm888https://github.com/facebookresearch/faiss for topic
identification of each tweet.

####  5.2.2. Parameter Settings:

We implement our model using Python 3.8.3, PyTorch 1.4.0 and Deep Graph
Library (Wang et al., 2019). The hyperparameters are tuned based on the
validation set and results are reported on the test set. The penalty parameter
$\mu$ is set to 0.2. The network parameters are randomly initialized with a
fixed seed for all experiments and optimized using Adam optimizer. We set the
learning rate to 0.6 and weight decay regularization with $\lambda=0.0005$.
For the subgraph convolution network, hidden size $H^{m}=32$ is used. Further,
the hidden dimension $\tilde{H}^{m}$ for subgraph attention network is set to
128, and 2 heads are used to train the HSA model.

####  5.2.3. Data Statistics:

We start by collecting a real-world Twitter user dataset for the detection of
collusive users. A total of 11,076 users involved in the “following” activity
from blackmarket service are identified and labelled as collusive users. Among
these, 10,863 users are randomly used for training DECIFE, and the remaining
213 users are used for testing. The entire data collection process from the
blackmarket service is mentioned in Section 3. Along with this, we take into
account 213 users which are labelled as non-collusive. We would again like to
emphasize on the fact that we chose only those Twitter users as non-collusive
who are guaranteed not to be a customer of any blackmarket service. The set of
non-collusive users is only used at test time to measure the performance of
DECIFE.

Table 2. Dataset Statistics. Statistic | Value
---|---
# of train users | 10,863
# of test users | 426
# of tweets | 6,627,277
# of topics | 1,000

Table 3. Subgraph Statistics. Relationship $\Delta$ |  # edges $(i,j)$ |  Edge-weight $(e_{ij})$
---|---|---
$\Delta_{1}$ | 6,408,903 | # common followers
$\Delta_{2}$ | 7,206,330 | # common transition users
$\Delta_{3}$ | 64,669 | 1
$\Delta_{4}$ | 88,069,870 | $\sum_{k\in K}\textit{min}((o_{k})^{i},(o_{k})^{j})$

Further, the followers, followees and tweets of all users are extracted using
the Twitter API, which are used for building the heterogeneous network. The
heterogeneous network contains around 7.9 million user nodes, 6.6 million
tweet nodes and 1000 topic nodes. This network is decomposed into four
subgraphs, each having $11,289$ users as nodes connected via different
relationships $\Delta$. The results are reported after 10-fold cross-
validation. The collusive users are randomly partitioned into 10 groups. Each
group with the same set of non-collusive users are used for each fold.
Detailed statistics of the dataset and the subgraphs are shown in Tables 3 and
3, respectively.

##  6\. Experimental Results and Analysis

###  6.1. Performance Comparison

In order to evaluate the performance of DECIFE against the baseline methods,
we use three popular evaluation metrics: AUC-ROC, AUC-PR and F1-score. Table
6.1 summarizes the comparative analysis of the competing methods. In general,
DECIFE outperforms all the baselines. Among the baselines, FakeFols turns out
to be the best in terms of AUC-ROC and F1-score. Among different variants of
DECIFE, we observe that DECIFE${}_{\Delta_{2}}$ outperforms all the other
variants and our proposed model in terms of F1-score. The reason behind this
is the correct identification of a higher number of non-collusive users (true
positives) in comparison to the collusive users (true negatives), which is not
the primary objective of our predictive modeling task99993.4% of the non-
collusive users and 60.5% collusive users are predicted correctly by
DECIFE${}_{\Delta_{2}}$.. Note that for each of the evaluation metrics, we
average the final values over 10 runs. As can be seen in Table 6.1, our
proposed DECIFE model consistently achieves the best performance over others
by a significant margin. We observe that the performances of the baseline
methods are poor, indicating that they fail to generalize for collusive user
detection as they are primarily targeted towards fake user detection. On the
contrary, DECIFE leverages the benefit of multiple relationships that prove
the effectiveness of heterogeneous network to integrate multi-type
information.

Table 4. Performance comparison of the competing models. Method | AUC-ROC | AUC-PR | F1-score
---|---|---|---
FakeFolss | 0.417 | 0.455 | 0.537
FakeFols | 0.617 | 0.568 | 0.683
FolMarket | 0.500 | 0.476 | 0.645
DECIFE${}_{\Delta_{1}}$ | 0.427 | 0.423 | 0.606
DECIFE${}_{\Delta_{2}}$ | 0.689 | 0.563 | 0.802
DECIFE${}_{\Delta_{3}}$ | 0.818 | 0.833 | 0.745
DECIFE${}_{\Delta_{4}}$ | 0.771 | 0.745 | 0.688
DECIFE | 0.895 | 0.854 | 0.786
Figure 3. TSNE visualized features of collusive (red) and non-collusive (blue)
users from the test dataset. The density estimate plots along the axis depict
the probability distribution of the users in both sets as (a) raw features,
and (b) fully-trained model.

The TSNE visualizations (c.f. Figure 3) of the test set embeddings for
collusive and non-collusive users indicate that raw features cannot be used
for detecting collusive users, whereas in the trained model, the learned
hypersphere clearly encloses and distinguishes the collusive users. Moreover,
the Kernel Density Estimate (KDE) plots in Figure 3 depict the probability
distribution of users in both sets. The probability density of collusive users
in the trained model is high at points where density of non-collusive users is
low, which is not the case in raw features. This suggests that the collusive
users can be easily partitioned in the embedding space using the final trained
embeddings by our converged DECIFE model.

###  6.2. Sensitivity of Parameters

We also investigate the sensitivity of parameters and report the results (AUC-
ROC) of our proposed model with various parameters in Figure 4. We consider
the following three parameters:

  * •

Convolution hidden size dimension ($H^{m}$): We first test the effect of the
dimension of hidden layer in the subgraph convolution network. The results are
shown in Figure 4(a). We can observe that our model achieves the best
performance when $H^{m}$ is set to 32. However, the performance is very
sensitive with the change in the dimension, suggesting that the convolution
hidden size $H^{m}$ plays a crucial role in comparison to other parameters.

  * •

Attention hidden size dimension ($\tilde{H}^{m}$): We explore our experimental
results by varying the size of the attention vector ($\tilde{H}^{m}$) during
projection. The result is shown in Figure 4(b). We can find that the
performance of DECIFE generally slightly varies upto the hidden dimension size
128. For higher dimensions, the model struggles to converge and overfits as
the 32-dimensional embedding is projected into a 256-dimensional embedding.

  * •

Number of heads ($d$): In order to check the impact of our proposed
aggregation framework, we measure the performance of DECIFE with the number of
heads $d$. The result is shown in Figure 4(c). Based on the results, we
observe that multi-head HSA model ($d>1$) performs better than single head
($d=1$). But as the complexity increases ($d=6$), the model starts to overfit,
and the performance is significantly impacted. Also, we find that the training
process becomes more stable with the growth in the number of model heads.

Figure 4. Parameter sensitivity of DECIFE w.r.t (a) convolution hidden size
dimension, (b) attention hidden size dimension, and (c) the number of heads.

###  6.3. Qualitative Analysis

We qualitatively examine all 25 false negatives produced by our DECIFE model
in the testing phase of our dataset. In a Twitter network of collusive users,
the priority is to minimize social damage; we, thereby, review the collusive
users which are falsely detected as non-collusive by our model. We find that
18 of these users (~72%) have less than 25 followers, thereby forming a sparse
connectivity in our proposed heterogeneous graph. This also makes them unable
to learn the connectivity information between the nodes and thereby limiting
the performance of DECIFE to identify these users. For the remaining 7 users,
we could not find enough conclusive evidence from our collected data to
interpret their incorrect detection by our proposed model. We would also like
to mention that after manual investigation, 4 out of these 7 users do not
exists currently on Twitter.

Figure 5. Wordcloud of the hashtags used by collusive users (true negatives)
Figure 6. Category distribution of tweets posted by collusive users

In order to understand the behaviour of the correctly identified collusive
users (true negatives101010Note that collusive class is the negative class in
our experiment.), we highlight interesting quantitative covariates among these
accounts.
Current user status: At the time of conducting this research, we found that
only 2% of detected collusive users which are collected from the blackmarket
services are suspended/deleted by Twitter. This clearly shows how these users
are successfully able to evade the existing fraudulent account detection
techniques deployed by Twitter.
Popular hashtags: We analyze a total of 119,486 tweets posted by the collusive
users detected using DECIFE covering their entire tweeting activity in our
dataset. Figure 5 shows the wordcloud of the hashtags used in tweets after
removing two-letter words and common stopwords. Here, the font size
corresponds to the frequency of the text. Unsurprisingly, we observe keywords
related to the blackmarket services (‘followme’), entertainment (‘music’,
‘soundcloud’, ‘edm’, ‘youtube’, ’spotify’), freelance platforms(‘fiverr’),
digital asset (‘cryptocurrency’, ‘bitcoin’, ‘blockchain’), etc. This shows how
collusive entities could have manipulated the social growth of these entities.
We believe these keywords also provide new directions for future collusive
entity detection. We list out a few example tweets with these keywords in
Table 5.
Tweet categories: It is often interesting to see which categories of tweets
are posted by fraudulent users. We follow a zero-shot learning approach
mentioned in (Wenpeng Yin and Roth, 2019) to identify the categories present
in the tweets posted by the detected collusive users. We frame it as a zero-
shot topic-classification problem where the task is to map a sentence (tweet
in our case) to a class (category in our case) that is not observed during
training. We map the tweets into one of the 14 Interactive Advertising Bureau
(IAB) categories111111https://developer.twitter.com/en/docs/twitter-ads-
api/campaign-management/api-reference/iab-categories which is used by the
supply partners within the Twitter Audience Platform (TAP). We show the
category distribution of tweets posted by the detected collusive users in
Figure 6. It can be clearly seen that almost 50% of the tweets are from two
categories: Digital creators and News. A possible reason for this is the
recent popularity of these categories on Twitter, allowing tweets from these
categories to connect directly with their audience, discover emerging content
and trends, and build communities using two monetization
programs121212https://media.twitter.com/en/articles/products/2018/media-
studio/monetization.html: Amplify Pre-roll and Amplify Sponsorships.

Table 5. Example of a few tweets posted by collusive users. Tweet |  Category
---|---
Boost Your Google Ranking Fast with 250 High Quality Backlinks #fiverr #backlinks #traffic #website |  Freelancing
OMG, I have 1,000 followers! Thank you. Think I will make it to 10,000? #followme |  Blackmarkets
This is the first spinner that brings free BTC. Spin & Earn. #btcspinnerio #btcspinner #bitcoin #btc |  Digital asset
This twitter site is all about #sports #gambling, sports #handicapping. He give my free picks on #Soccer, #NFL-Foot |  Sports
Get up to 50GB of free space for all your photos, videos, docs, and music |  Technology
Check it out! socialspider will do viral youtube promotion usa or worldwide for $5 on #Fiverr |  Freelancing
#CyberMiles is a new blockchain protocol being developed to revolutionize how digital commerce and online marketplace |  Digital asset
Yo Bro my Music is poping right now. Everysong i drop on soundcloud touches 1000 in a Month. Can you drop my songs on your Youtube. |  Entertainment
Grab them while they last, 100 free DAT Tokens here #datumnetwork #ico #cryptocurrency |  Digital asset
Get $1million $CPPG tokens. #Airdrop #bounty #Crypto #cryptocurrency |  Digital asset

##  7\. Conclusion

The prevalence of collusion over social media has become a serious problem. In
this paper, we addressed the problem of detecting collusive users involved in
blackmarket services to produce artificial followers for Twitter users. We
proposed DECIFE, a framework that considers the follower/followee
relationships and linguistic properties of tweets for detecting collusive
users in Twitter. Compared with previous methods, our approach focuses on
multiple relationships between the users and can leverage information more
efficiently. Using extensive experiments, we demonstrated the effectiveness
and superiority of DECIFE with an attention mechanism on our curated dataset
over the state-of-the-art methods. To the best of our knowledge, this work is
the first attempt using heterogeneous network to detect collusive followers on
Twitter. The dataset we collected is also the first dataset of its kind. For
future work, in addition to the metadata and topical properties, we plan to
capture the temporal properties of these users that can help us improve the
performance of the predictive modeling task. We also plan to investigate the
effectiveness of the proposed approach with closely related problems such as
bot detection, identifying content polluters etc.

## Acknowledgement

The authors would like to acknowledge the support of ECR/2017/001691 (SERB)
and ihub-Anubhuti-iiitd Foundation set up under the NM-ICPS scheme of the
Department of Science and Technology, India.

## References

  * (1)
  * Aggarwal et al. (2018) Anupama Aggarwal, Saravana Kumar, Kushagra Bhargava, and Ponnurangam Kumaraguru. 2018.  The follower count fallacy: detecting Twitter users with manipulated follower count. In _SAC_. ACM, 1748–1755.
  * Aggarwal and Kumaraguru (2015) Anupama Aggarwal and Ponnurangam Kumaraguru. 2015.  What they do in shadows: Twitter underground follower market. In _IEEE PST_. 93–100.
  * Arora et al. (2020) Udit Arora, Hridoy Sankar Dutta, Brihi Joshi, Aditya Chetan, and Tanmoy Chakraborty. 2020.  Analyzing and Detecting Collusive Users Involved in Blackmarket Retweeting Activities.  _ACM TIST_ 11, 3 (2020), 1–24.
  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  _Transactions of the Association for Computational Linguistics_ 7 (2019), 597–610.
  * Castellini et al. (2017) Jacopo Castellini, Valentina Poggioni, and Giulia Sorbi. 2017\.  Fake Twitter Followers Detection by Denoising Autoencoder. In _WI_. 195–202.
  * Chetan et al. (2019) Aditya Chetan, Brihi Joshi, Hridoy Sankar Dutta, and Tanmoy Chakraborty. 2019.  CoReRank: Ranking to Detect Users Involved in Blackmarket-Based Collusive Retweeting Activities. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_. ACM, 330–338.
  * Cresci et al. (2014) Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2014\.  A Fake Follower Story: improving fake accounts detection on Twitter.  _IIT-CNR, Tech. Rep. TR-03_ (2014).
  * Cresci et al. (2015) Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2015\.  Fame for sale: Efficient detection of fake Twitter followers.  _Decision Support Systems_ (2015).
  * De Micheli and Stroppa (2013) Carlo De Micheli and Andrea Stroppa. 2013.  Twitter and the underground market. In _11th Nexa Lunch Seminar_ , Vol. 22. 1–6.
  * Dutta and Chakraborty (2019) Hridoy Sankar Dutta and Tanmoy Chakraborty. 2019.  Blackmarket-Driven Collusion Among Retweeters–Analysis, Detection, and Characterization.  _IEEE Transactions on Information Forensics and Security_ 15 (2019), 1935–1944.
  * Dutta and Chakraborty (2020) Hridoy Sankar Dutta and Tanmoy Chakraborty. 2020.  Blackmarket-driven collusion on online media: a survey.  _arXiv preprint arXiv:2008.13102_ (2020).
  * Dutta et al. (2018) Hridoy Sankar Dutta, Aditya Chetan, Brihi Joshi, and Tanmoy Chakraborty. 2018.  Retweet us, we will retweet you: Spotting collusive retweeters involved in blackmarket services. In _2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)_. IEEE, 242–249.
  * Dutta et al. (2020a) Hridoy Sankar Dutta, Vishal Raj Dutta, Aditya Adhikary, and Tanmoy Chakraborty. 2020a.  HawkesEye: Detecting fake retweeters using Hawkes process and topic modeling.  _IEEE Transactions on Information Forensics and Security_ 15 (2020), 2667–2678.
  * Dutta et al. (2020b) Hridoy Sankar Dutta, Mayank Jobanputra, Himani Negi, and Tanmoy Chakraborty. 2020b.  Detecting and analyzing collusive entities on YouTube.  _arxiv:2005.06243_ (2020).
  * Farooqi et al. (2017) Shehroze Farooqi, Fareed Zaffar, Nektarios Leontiadis, and Zubair Shafiq. 2017.  Measuring and mitigating oauth access token abuse by collusion networks. In _IMC_. 355–368.
  * Ferrara et al. (2016) Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016\.  The Rise of Social Bots.  _Commun. ACM_ (2016), 96–104.
  * Jiang et al. (2014) Meng Jiang, Peng Cui, Alex Beutel, Christos Faloutsos, and Shiqiang Yang. 2014.  Catchsync: catching synchronized behavior in large directed graphs. In _ACM SIGKDD_.
  * Kwak et al. (2011) Haewoon Kwak, Hyunwoo Chun, and Sue Moon. 2011.  Fragile online relationship: a first look at unfollow dynamics in twitter. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 1091–1100.
  * Lee et al. (2015) Lee, Mahmud, Chen, Zhou, and Nichols. 2015.  Who will retweet this? detecting strangers from twitter to retweet information.  _ACM TIST_ (2015).
  * Li et al. (2014) Huayi Li, Arjun Mukherjee, Bing Liu, Rachel Kornfield, and Sherry Emery. 2014.  Detecting Campaign Promoters on Twitter Using Markov Random Fields _(ICDM ’14)_. 290–299.
  * Liu et al. (2016) Yuli Liu, Yiqun Liu, Min Zhang, and Shaoping Ma. 2016\.  Pay Me and I’ll Follow You: Detection of Crowdturfing Following Activities in Microblog Environment.. In _IJCAI_.
  * Lui and Baldwin (2012) Marco Lui and Timothy Baldwin. 2012.  langid. py: An off-the-shelf language identification tool. In _Proceedings of the ACL 2012 system demonstrations_. 25–30.
  * Mehrotra et al. (2016) Ashish Mehrotra, Mallidi Sarreddy, and Sanjay Singh. 2016\.  Detection of fake Twitter followers using graph centrality measures. In _IEEE IC3I_. 499–504.
  * Motoyama et al. (2011) Marti Motoyama, Damon McCoy, Kirill Levchenko, Stefan Savage, and Geoffrey M Voelker. 2011.  An analysis of underground forums. In _ACM SIGCOMM_. ACM, 71–80.
  * Ruff et al. (2018) Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. 2018.  Deep one-class classification. In _International conference on machine learning_. 4393–4402.
  * Shah et al. (2014) Neil Shah, Alex Beutel, Brian Gallagher, and Christos Faloutsos. 2014.  Spotting suspicious link behavior with fbox: An adversarial perspective. In _IEEE ICDM_. 959–964.
  * Shah et al. (2017) Neil Shah, Hemank Lamba, Alex Beutel, and Christos Faloutsos. 2017\.  The Many Faces of Link Fraud. In _IEEE ICDM_. 1069–1074.
  * Shen and Liu (2015) H. Shen and X. Liu. 2015\.  Detecting Spammers on Twitter Based on Content and Social Interaction. In _2015 International Conference on Network and Information Systems for Computers_. 413–417.
  * Shen et al. (2014) Yi Shen, Jianjun Yu, Kejun Dong, and Kai Nan. 2014\.  Automatic fake followers detection in chinese micro-blogging system. In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_. Springer, 596–607.
  * Singh et al. (2016) Monika Singh, Divya Bansal, and Sanjeev Sofat. 2016\.  Followers or fradulents? An analysis and classification of Twitter followers market merchants.  _Cybernetics and Systems_ 47, 8 (2016), 674–689.
  * Song et al. (2015) Jonghyuk Song, Sangho Lee, and Jong Kim. 2015.  Crowdtarget: Target-based detection of crowdturfing in online social networks. In _ACM SIGSAC_. 793–804.
  * Stringhini et al. (2012) Gianluca Stringhini, Manuel Egele, Christopher Kruegel, and Giovanni Vigna. 2012.  Poultry markets: on the underground economy of twitter followers.  _ACM SIGCOMM Computer Communication Review_ 42, 4 (2012), 527–532.
  * Stringhini et al. (2013) Gianluca Stringhini, Gang Wang, Manuel Egele, Christopher Kruegel, Giovanni Vigna, Haitao Zheng, and Ben Y Zhao. 2013.  Follow the green: growth and dynamics in twitter follower markets. In _Proceedings of the 2013 conference on Internet measurement conference_. 163–176.
  * Sun and Han (2013) Yizhou Sun and Jiawei Han. 2013.  Mining heterogeneous information networks: a structural analysis approach.  _Acm Sigkdd Explorations Newsletter_ 14, 2 (2013), 20–28.
  * Thomas (2013) Kurt et all Thomas. 2013\.  Trafficking Fraudulent Accounts: The Role of the Underground Market in Twitter Spam and Abuse.. In _USENIX Symposium_. 195–210.
  * Wang et al. (2019) Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, et al. 2019\.  Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs.  (2019).
  * Weerasinghe et al. (2020) Janith Weerasinghe, Bailey Flanigan, Aviel Stein, Damon McCoy, and Rachel Greenstadt. 2020.  The Pod People: Understanding Manipulation of Social Media Popularity via Reciprocity Abuse. In _The WebConf_. 1874–1884.
  * Wenpeng Yin and Roth (2019) Jamaal Hay Wenpeng Yin and Dan Roth. 2019.  Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach. In _EMNLP_.  https://arxiv.org/abs/1909.00161
  * Zhang and Lu (2016) Yi Zhang and Jianguo Lu. 2016.  Discover millions of fake followers in Weibo.  _Social Network Analysis and Mining_ 6, 1 (2016), 16\.
  * Zhang et al. (2016) Zhedi Zhang, Futai Zou, Li Pan, Bei Pei, and Jianhua Li. 2016.  Detection of zombie followers in SINA Weibo. In _IEEE ICCC_. 2476–2480.
  * Zhu et al. (2016) Zhen-hui Zhu, Yang Zhi, and Ya-fei Dai. 2016.  A New Approach to Detect User Collusion Behavior in Online QA System. In _CNCT_. Atlantis Press.

≥


11institutetext: LIST, Key Laboratory of Computer Network and Information
Integration (Southeast University), Ministry of Education, Nanjing, China
11email<EMAIL_ADDRESS>Univ Rennes, Inserm, LTSI -
UMR1099, Rennes, F-35000, France 33institutetext: Centre de Recherche en
Information Biomédicale Sino-Français (CRIBs) 44institutetext: Dept. of
Medical Biophysics, University of Western Ontario, London, ON, Canada

# EnMcGAN: Adversarial Ensemble Learning for 3D Complete Renal Structures
Segmentation

Yuting He 11 Rongjun Ge 11 Xiaoming Qi 11 Guanyu Yang(🖂) 1133 Yang Chen 1133
Youyong Kong 1133 Huazhong Shu 1133 Jean-Louis Coatrieux 22 Shuo Li 44

###### Abstract

3D complete renal structures(CRS) segmentation targets on segmenting the
kidneys, tumors, renal arteries and veins in one inference. Once successful,
it will provide preoperative plans and intraoperative guidance for
laparoscopic partial nephrectomy(LPN), playing a key role in the renal cancer
treatment. However, no success has been reported in 3D CRS segmentation due to
the complex shapes of renal structures, low contrast and large anatomical
variation. In this study, we utilize the adversarial ensemble learning and
propose _Ensemble Multi-condition GAN_(EnMcGAN) for 3D CRS segmentation for
the first time. Its contribution is three-fold. 1)Inspired by windowing[4], we
propose the multi-windowing committee which divides CTA image into multiple
narrow windows with different window centers and widths enhancing the contrast
for salient boundaries and soft tissues. And then, it builds an ensemble
segmentation model on these narrow windows to fuse the segmentation
superiorities and improve whole segmentation quality. 2)We propose the multi-
condition GAN which equips the segmentation model with multiple discriminators
to encourage the segmented structures meeting their real shape conditions,
thus improving the shape feature extraction ability. 3)We propose the
adversarial weighted ensemble module which uses the trained discriminators to
evaluate the quality of segmented structures, and normalizes these evaluation
scores for the ensemble weights directed at the input image, thus enhancing
the ensemble results. 122 patients are enrolled in this study and the mean
Dice coefficient of the renal structures achieves 84.6%. Extensive experiments
with promising results on renal structures reveal powerful segmentation
accuracy and great clinical significance in renal cancer treatment.

##  1 Introduction

Figure 1: The challenges of the 3D CRS segmentation. a) The renal structures
has complex shapes making difficult feature extraction. b) The original CTA
images has low contrast bringing low segmentation quality on boundaries. c)
The renal structures has large anatomical variation in different cases causing
weak generalization ability.

3D complete renal structures(CRS) segmentation on CTA image targets on
segmenting the kidneys, tumors, renal arteries and veins in one inference.
Once successful, it will provide preoperative plans and intraoperative
guidance for laparoscopic partial nephrectomy (LPN) [20, 21], playing a key
role in renal cancer treatment [19]. _Preoperatively_ , the renal artery
reaching the interlobar arteries will guide the estimation of perfusion
regions to select the tumor-feeding branches and locate the arterial clamping
position [20]. The kidney and tumor will show the lesions’ location to pre-
plan the tumor resection surface [25]. _Intraoperatively_ , the veins outside
the hilum will help the clinicians exclude the unconcerned vessels thus
clamping the accurate arterial positions quickly. Besides, the 3D CRS visual
model will also be fused with the laparoscopic videos bringing the augmented
reality [15], so that the invisible regions will be supplemented to guide the
operation smoothly. With the assistance of 3D CRS segmentation, the safety of
renal surgery is improved, the pain of patients is relieved and the cost of
the treatment is reduced.

However, there is no effective solution for 3D CRS segmentation. Taha et al
[23] utilized a Kid-Net achieving the renal vessels segmentation. He et al [6]
proposed a semi-supervised framework and achieved the fine renal artery
segmentation. Li et al [9] designed a Residual U-Net for renal structures
segmentation. The related works are limited in our task: 1)On one hand, some
works [23, 6] only focused on partial renal structures lacking the
countermeasures to the complete structures in one-inference segmentation. The
one-by-one segmentation also will make overlap of different structures
especially on low-contrast boundaries. 2)On the other, some works [9, 23] lack
the fine detail of the segmented structures limiting the clinical downstream
tasks. For example, their arteries only reaches the segmental arteries losing
the ability of perfusion regions estimation.

Formidable challenges of 3D CRS segmentation are limiting its further
applications: 1) Complex shapes of renal structures. These structures have
complex shapes, for example, the arteries have tree-like shape while the
tumors have ball-like shape (Fig. 1 (a)). The model has to represent their
features simultaneously resulting in a difficult feature extraction process
and limiting its generalization ability. 2) Coarse-grained pattern and low
contrast. CT image has a large gray range (4096), and the structures of
interests in our task are only in a narrow range. This makes the contrast of
soft tissues and the boundary regions are low in original images (Fig. 1(b))
such as the tumors and veins, making coarse-grained pattern. Therefore, the
segmentation network will difficult to perceive the fine-grained pattern in
such a narrow distribution limiting the integrity of the structures and the
quality of boundaries. 3) Large anatomical variation. Renal vessels have
uncertain branch numbers and growth topology [6, 17], and the location of the
tumors and their damage to the kidneys are uncertain(Fig. 1(c)). These
anatomical variation makes it difficult to cover all variation, limiting the
model’s generalization.

The windowing [4] removes irrelevant gray ranges and expands the interested
distribution making the network focus on the wider distribution to perceive
fine-grained pattern. Inspired by the radiologist delineating the renal
structures in different window widths and centers (sub-windows) via windowing,
we propose the _multi-windowing committee (MWC)_. We select the sub-windows
which of superiority in renal structures on CTA images, and divide the CTA
image into these sub-windows thus expanding our task-interested distribution
and making fine-grained pattern such as boundaries (Fig. 2(a)). We train
multiple learners on these sub-window images for the fine-grained
representations on their covered distributions making segmentation
superiorities. Finally, these superiorities will be fused [18] improving the
integrated segmentation quality.

The shape regularisation utilizes the shape knowledge learned by an additional
model to encourage the CNN extracting higher-order shape features [13, 16,
10]. We propose the _Multi-condition GAN (McGAN)_ which equips the
segmentation model with multiple discriminators for global shape constraints
(Fig. 2(b)). During adversarial training, the discriminators learn to evaluate
the similarity between the segmented structures’ shapes and real shapes, thus
learning the higher-order shape knowledge of the renal structures via the min-
max game of the adversarial training [5]. Then, these knowledge are made as
the conditions that the segmented renal structures have to meet when
optimizing the segmentation models, improving the shape feature extraction
ability.

Fusing the segmentation superiorities of the learners will bridge their
representation preferences caused by large anatomical variations [18, 3].
These preferences will make the learners have better segmentation quality in
their advantageous regions and poor quality in vulnerable regions, if being
fused, the whole accuracy will be improved. Therefore, we propose the
_adversarial weighted Ensemble (AWE)_ (Fig. 2(c)) which utilizes the trained
discriminators to evaluate the pixel-wise segmentation quality of the results,
thus giving higher weights to better quality regions and making the ensemble
direct at the segmentation quality dynamically. So, the fusion of the
advantageous regions in the results will balance the representation
preferences and bring personalized ensemble process.

We propose an adversarial ensemble learning framework, EnMcGAN, for 3D CRS
segmentation for the first time. Our detailed contributions are as follow:

  * •

We propose the adversarial ensemble learning which equips the ensemble
segmentation model with adversarial learning for the first time, and propose
the _EnMcGAN_ for 3D CRS segmentation which will play an important role in
accurate preoperative planning and intraoperative guidance of LPN. Our
complete experiments demonstrate its excellent performance.

  * •

We propose the _multi-windowing committee (MWC)_ which divides CTA image into
the sub-windows which of superiority in renal structures, making fine-grained
pattern. Then the learners trained on these sub-windows are constructed as an
ensemble framework to fuse the fine-grained representations on the covered
distributions of these sub-windows, thus improving integrated segmentation
quality.

  * •

We present the _multi-condition GAN (McGAN)_ which embeds the shape knowledge
in segmentation model via multiple discriminators, thus encouraging the
segmented results being consistent with their shape prior knowledge and
improving the shape features extraction ability.

  * •

We propose the _adversarial weighted Ensemble (AWE)_ which utilizes
segmentation quality evaluation ability of the trained discriminators to fuse
the advantageous regions in the segmented results, thus bringing personalized
fine ensemble process and balancing the representation preferences.

##  2 Methodology

Figure 2: The illustration of our EnMcGAN: a) MWC enhances the contrast on
windowed images bringing salient boundaries and soft tissues (Sec. 2.1); b)
McGAN utilizes multiple discriminators for shape constraints (Sec. 2.2) and c)
AWE module utilizes the trained discriminators for dynamic ensemble weights
(Sec. 2.3).

As shown in Fig. 2, our proposed EnMcGAN takes adversarial ensemble learning
for high quality 3D CRS segmentation. It has three cooperative elements: 1)
Multi-windowing committee (MWC, Sec. 2.1) thresholds and normalizes the CTA
image enhancing the contrast and making the fine-grained pattern, and fuses
the segmentation superiorities of multiple segmentation learners trained in
different covered distributions improving the integrated segmentation quality.
2) Multi-condition GAN (McGAN, Sec. 2.2) utilizes shape constraints from the
discriminator to encourage the segmented renal structures meeting their real
shape conditions, thus improving the shape features extraction ability. 3)
Adversarial weighted Ensemble (AWE, Sec. 2.3) utilizes the trained
discriminator for dynamic ensemble weights, thus and balancing the
representation preferences and providing the personalized ensemble results.

###  2.1 Multi-windowing committee for multiple fine-grained representation

Our MWC (Fig. 2(a)) divides CTA image to multiple narrow windows [4] and
expends their distributions making the significant region and fine-grained
pattern. And then it fuse the segmentation superiorities from the
distributions covered by multiple sub-windows, thus improving integrated
segmentation quality.

Windowing for fine-grained pattern. The CTA image $x$ is divided to multiple
narrow windows with different window centers and widths. Firstly, nine
different windows are selected. We segment CTA images via k-means clustering
[14] for $c$ categories, in our experiment $c=5$. Then, the mean CT values of
these categories in our dataset are calculated and selected for three window
centers $c$, in our experiment,
$\\{c_{0}=1032.7834,c_{1}=1150.0825,c_{2}=1332.1959\\}$. We also define three
default window widths $\\{w_{0}=256,w_{1}=512,w_{2}=768\\}$, thus combining
for nine sub-windows. Then, the CTA image $x$ is thresholded to remove
irrelevant gray range and normalized to expand the interested distribution via
windowing, thus making nine windowed images $\\{x_{0},x_{1},...,x_{8}\\}$:
$x_{i}=\frac{max(min(x,c_{j}-\frac{w_{k}}{2}),c_{j}+\frac{w_{k}}{2})-(c_{j}-\frac{w_{k}}{2})}{w_{k}}$.
Finally, these images are used to train nine segmentation learners $S(\dot{)}$
which will learn the fine-grained representations on the distributions covered
by these sub-windows.

Summary of the advantages. 1) The CTA image is thresholded with different
window centers and widths for small gray ranges and normalized to $[0,1]$,
thus the irrelevant gray range will be removed and the interested distribution
will be expended. Therefore, these sub-windows will have significant regions
and fine-grained patterns improving the segmentation quality. 2) The image in
different narrow windows will have different significant regions, so our
multiple learners trained on these sub-window images will learn the fine-
grained representations on their covered distributions making segmentation
superiorities. These superiorities are fused improving the final segmentation
quality.

###  2.2 Multi-condition GAN for shape constraints

Our McGAN (Fig. 2(b)) utilizes multiple discriminators to provide the
segmentation model shape constraints, thus encouraging the segmented
structures meeting their real shape conditions, improving the shape feature
extraction ability.

McGAN for adversarial segmentation. In training stage, our McGAN inputs the
real or segmented renal structures together with the CTA image into five
discriminators resulting in a conditional GAN [12] for shape constraints of
renal structures. The segmentation model $S$ takes the DenseBiasNet [6] which
fuses the multi-receptive field features for the multi-scale feature
representation ability. The discriminators $\\{D_{0},...,D_{4}\\}$ follow the
3D version of the VGG-A [22] and activated by sigmoid function. As shown in
Fig. 2 (b), it takes the original image $x$ and the segmented $S(x_{i})_{n}$
or ground truth $y_{n}$ mask of each structure are input to discriminator to
learn the shape constraint of each renal structure and the evaluation ability
of segmentation quality. The binary cross-entropy loss $\mathcal{L}_{bce}$ is
calculated as the adversarial loss $\mathcal{L}_{adv_{n}}$ of each structure:

$\mathcal{L}_{adv_{n}}(\theta_{D_{n}})=\mathcal{L}_{bce}(D_{n}(x,y_{n}),1)+\mathcal{L}_{bce}(D_{n}(x,S(x_{i})_{n}),0).$
(1)

The segmentation model takes the windowed images $x_{i}$ as input and is
optimized by the multi-class cross-entropy loss $\mathcal{L}_{mce}$ from the
segmented results $S(x)$ and labels $y$, together with the adversarial losses
corresponding to renal structures from the discriminator. Therefore, the
hybrid segmentation loss is:

$\mathcal{L}_{seg}(\theta_{S})=\mathcal{L}_{mce}(S(x_{i}),y)+\lambda\sum^{N}_{n=0}\mathcal{L}_{bce}(D_{n}(x,S(x_{i})_{n}),1),$
(2)

where the $\lambda$ is the weight to balance loss functions in this hybrid
loss. In our experiment, it is 0.01. The $\theta_{S}$ and $\theta_{D_{n}}$ is
the parameters of the segmentation model and discriminators. As an ensemble
model, narrow-window images are used to train nine McGANs which share their
discriminators iteratively.

Summary of the advantages. During adversarial training, the discriminators
learn to evaluate the similarity between the segmented structures’ shapes and
real shapes. When optimizing the segmentation model, the real shapes will
become the conditions encoded by the discriminators, so the adversarial loss
will encourage the segmented structures to meet them improving the shape
feature extraction ability of the segmentation model.

###  2.3 Adversarial weighted ensemble for personalized fine fusion

Our AWE module (Fig. 2(c)) utilizes the trained discriminators from our McGAN
to evaluate the segmentation quality of renal structures and generate the
dynamic ensemble weights directed at the input image, thus bringing
personalized fine ensemble process and balancing the representation
preferences.

Ensemble process of AWE. In testing stage, our AWE module fuses the results
based on their segmentation quality, improving the ensemble results. Fig. 2
(c) illustrates the ensemble process of this module taking the kidney as an
example. The segmented kidneys $\\{kidney_{0},...,kidney_{8}\\}$ from the
segmentation learners together with the original image $x$ are putted into the
trained discriminators for evaluation scores $\\{s_{0},..,s_{8}\\}$. Then,
these scores are normalized via $w_{i}=\frac{s_{i}}{\sum^{8}_{n=0}s_{n}}$ for
the ensemble weights directed at the input image. Finally, these weights are
used to weight the average ensemble enhancing the results:
$\hat{y_{kidney}}=\frac{1}{9}\sum_{i=0}^{9}kidney_{i}*w_{i}$.

Summary of the advantages. The trained discriminators evaluate the
segmentation qualities of renal structures, so we utilize them for dynamic
ensemble weights directed at input images. Therefore, the incorrect weights
are avoided, and the ensemble results are effectively enhanced.

##  3 Materials and Configurations.

Dataset. 122 patients with renal cancer treated with LPN are enrolled in this
study. The kidney region of interests (ROIs) with tumors on their CTA images
were extracted as the dataset in our experiments. Their pixel sizes are
between $0.47mm$ and $0.74mm$, the slice thickness is $0.5mm$ and the image
size is $150\times 150\times 200$. Five kidney tumor subtypes including clear
renal cell carcinomas, papillary, chromophobe, angiomyolipoma and eosinophilic
adenoma are included in this dataset resulting in large heterogeneity and
anatomical variation, and the tumor volume is vary up to 300 times. The
kidney, tumor, vein and artery on these images are fine labeled.

Comparison settings. To evaluate the superiority of our framework, we perform
extensive experiments. The V-Net [11], 3D U-Net [2], Res-U-Net [9], Kid-Net
[23], DenseBiasNet [6] and the ensemble model, VFN [24], are trained as the
comparison models. For fair comparison, VFN also takes DenseBiasNet as the
segmentation learners. The Dice coefficient (DSC) $[\%]$, mean surface voxel
distance (MSD) are used to evaluate the coincidence of the region and surface.
Besides, the mean centerline voxel distance (MCD) is used to evaluate the
coincidence of the artery topology following [7].

Implementation. During training, $150\times 150\times 128$ patches are cropped
in the z-axis so that the dataset is enlarged and the GPU memory is saved.
Each model is optimized by Adam [8] with the batch size of 1, learning rate of
$1\times 10^{-4}$ and iterations of 40000. The 5-fold cross-validation is
performed for comprehensive evaluation. All methods are implemented with
PyTorch and trained on NVIDIA TITAN Xp GPUs.

##  4 Results and analysis

Our EnMcGAN brings the fine-grained pattern and significant regions, embeds
shape priori knowledge into segmentation model and bridge the representation
preferences caused by large anatomical variations thus achieving the excellent
3D complete renal structures segmentation. In this part, we will thoroughly
evaluate and analyze the effectiveness of our proposed EnMcGAN: 1) The
quantitative evaluation, qualitative evaluation and the ROC and PR curves in
comparative study (Sec. 4.1) will show the superiorities of our framework
compared with other models. 2) The ablation study (Sec. 2) will demonstrate
the contribution of each our innovation in our framework. 3) The the
performances of each segmentation learner and the number of fused learners
will be analysed in framework analysis (Sec. 4.3).

###  4.1 Comparative study shows superiority

Table 1: Evaluation on renal structures reveal powerful performance of our
framework. The ’En’ means the method is the ensemble learning model.

Method | Kidney | Tumor | Vein | Artery | Mean DSC
---|---|---|---|---|---
DSC | MSD | DSC | MSD | DSC | MSD | DSC | MCD | MSD |  $\pm$std
V-Net [11] | 94.1$\pm$2.3 | 1.17$\pm$0.72 | 67.5$\pm$26.8 | 7.01$\pm$6.36 | 66.9$\pm$14.8 | 4.57$\pm$6.35 | 83.0$\pm$8.2 | 3.21$\pm$2.66 | 2.03$\pm$2.81 | 77.9$\pm$7.8
3D U-Net [2] | 91.9$\pm$10.9 | 1.11$\pm$0.60 | 72.1$\pm$26.3 | 5.22$\pm$6.12 | 65.4$\pm$20.8 | 2.41$\pm$1.60 | 80.5$\pm$9.9 | 1.93$\pm$1.06$\pm$ | 1.30$\pm$1.28 | 77.5$\pm$12.2
Res-U-Net [9] | 92.5$\pm$3.8 | 1.63$\pm$0.95 | 51.2$\pm$29.7 | 13.02$\pm$16.73 | 63.4$\pm$17.7 | 3.00$\pm$1.73 | 81.9$\pm$6.1 | 2.95$\pm$2.00 | 1.91$\pm$2.00 | 72.2$\pm$7.9
Kid-Net [23] | 91.0$\pm$11.5 | 1.49$\pm$1.14 | 69.4$\pm$23.2 | 7.63$\pm$7.07 | 57.2$\pm$21.8 | 3.70$\pm$2.71 | 73.9$\pm$12.9 | 3.76$\pm$2.35 | 1.91$\pm$1.94 | 72.9$\pm$11.1
DenseBiasNet [6] | 94.1$\pm$2.4 | 1.31$\pm$0.87 | 67.0$\pm$26.9 | 8.51$\pm$8.70 | 71.8$\pm$14.8 | 2.13$\pm$1.81 | 87.1$\pm$6.5 | 2.00$\pm$1.18 | 0.97$\pm$0.85 | 80.0$\pm$7.1
VFN (En) [24] | 94.3$\pm$2.2 | 1.47$\pm$0.76 | 66.2$\pm$27.0 | 8.50$\pm$12.16 | 70.3$\pm$16.3 | 3.07$\pm$2.39 | 88.5$\pm$5.6 | 1.36$\pm$0.64 | 0.53$\pm$0.46 | 80.0$\pm$8.5
Ours (En) | 95.2$\pm$1.9 | 0.94$\pm$0.55 | 76.5$\pm$22.9 | 5.18$\pm$7.11 | 77.7$\pm$12.1 | 1.38$\pm$0.95 | 89.0$\pm$6.8 | 1.66$\pm$0.89 | 0.69$\pm$0.66 | 84.6$\pm$6.7

Quantitative evaluation. As shown in Tab. 1, our proposed framework achieves
the state-of-the-art performance. In our EnMcGAN, the artery achieves 89.0%
DSC, 1.66 MCD and 0.69 MSD which will strongly support the perfusion regions
estimation and the arterial clamping position selection. The kidney gets 95.2%
DSC and 0.94 MSD, the tumor gets 76.5% DSC and 5.18 MSD, and the vein gets
77.7% DSC and 1.38 MSD which will provide operation guidance. The ensemble
model, VFN, fuses the 2D information in different perspectives, achieving good
performance on artery. However, similar to the V-Net, 3D U-Net, Res-U-Net,
DenseBiasNet, it has poor performance on veins and tumors owing to the low
contrast and the complex shapes.

Figure 3: Our proposed framework has powerful visual superiority which will
provide visual guidance for surgery. Case 1 is a left kidney with
angiomyolipoma and the case 2 is a right kidney with clear renal cell
carcinomas.

Qualitative evaluation. As demonstrated in Fig. 3, our proposed framework has
great visual superiority which will provide visual guidance for surgery.
Compared with the ground truth, our EnMcGAN enhances the contrast and utilizes
the shape constraints from discriminator improving the integrity of tumors and
the segmentation quality of veins. The Kid-Net and 3D U-Net have blur
segmentation on small artery branches, and the DenseBiasNet, Res-U-Net, V-Net
and VFN have serious under-segmentation because of the low contrast, complex
shape and large anatomical variation in our CRS segmentation task.

Figure 4: The ROCs show that our EnMcGAN has higher segmentation accuracy than
other comparison methods in our renal structures.

ROC curves As shown in Fig. 4, the ROC curve of our EnMcGAN covers the other
comparison methods which means our proposed method has more powerful
performance in each segmented renal structure. Due to the class imbalance in
our task, the true positive rate will rise rapidly when plotting the ROC
curve. For better demonstration, we show the ROC curves with a true positive
rate between 0.8 and 1 in artery and kidney, and between 0.5 and 1 in vein and
tumor.

###  4.2 Ablation study shows improvements of the innovations

Table 2: The ablation study analyses the contributions of our innovations. McGAN | MWC | AWE | DSC(%)$\pm$std
---|---|---|---
Kidney | Tumor | Vein | Artery | AVG
|  |  | 94.1$\pm$2.4 | 67.0$\pm$26.9 | 71.8$\pm$14.8 | 87.1$\pm$6.5 | 80.0$\pm$7.1
✓ |  |  | 94.7$\pm$2.4 | 75.0$\pm$22.7 | 72.8$\pm$12.8 | 87.6$\pm$8.1 | 82.5$\pm$7.1
✓ | ✓ |  | 94.8$\pm$2.2 | 75.8$\pm$21.1 | 75.3$\pm$12.9 | 89.3$\pm$4.7 | 83.8$\pm$6.7
✓ | ✓ | ✓ | 95.2$\pm$1.9 | 76.5$\pm$22.9 | 77.7$\pm$12.1 | 89.0$\pm$6.8 | 84.6$\pm$6.7

The innovations in our framework brings significant enhancements. Our McGAN
takes the shape constraints and improves 8.0% DSC on tumor compared with the
basic segmentation network (DenseBiasNet). When taking our MWC with the
majority voting [1], the vein and artery achieve 2.5% and 1.7% additional DSC
improvement owing the fine-grained pattern in narrow windows. When adding our
AWE module, the DSC of the tumor and vein are increased by 0.7% and 2.4% due
to its dynamic ensemble weights directed at the input image. Totally, compared
with the basic network, our EnMcGAN enhances the 1.1%, 9.5%, 5.9% and 1.9% DSC
in kidney, tumor, vein and artery.

###  4.3 Framework analysis

Segmentation learners analysis. As illustrated in Fig. 5, two conclusions will
be summarized: 1) The learners have segmentation superiorities in different
narrow-window images which have different salient regions. For example, when
$c_{1}w_{2}$, the learner achieves significant performance on artery (88.0%),
but it has poor performance on tumor (72.3%). The $c_{0}w_{2}$ has ordinary
performance (85.4%) on artery, but it achieves the highest DSC (77.4%) on
tumor. 2) Fusing the segmentation superiorities of the learners will improve
the whole segmentation quality which higher than the quality of each learner.
Our EnMcGAN achieves 95.2%, 76.5%, 77.7% and 89.0% DSC on kidney, tumor, vein
and artery which are higher than the best learners of renal structures (94.9%,
77.4%, 74.9% and 88.0%).

Figure 5: The learners have segmentation superiorities in different narrow-
window images. The heatmaps show the DSC of the learners on the renal
structures.

Amount of fused learners analysis. As shown in Fig. 6, our with the amount of
the fused learners trained in different sub-windows increasing, the ensemble
accuracy will increase. We rank the trained segmentation learners in different
sub-windows, and fuse them start from the best via our AWE strategy. It
illustrate the characteristic in two aspect: 1) Overall, with the amount of
the fused learners increasing, the average ensemble performance is increasing
because more segmentation preferences are integrated into the model. 2) The
renal structures has different sensitivity to the increasing of the learners.
The performance of the kidney is almost unchanged because it has relatively
large volume and wide gray range. The performance of tumor is increasing and
decreasing, because it is sensitive to variation of window width and center.
When the learner trained in the bad sub-window of tumor fused into the
ensemble model, it will become the noise which will weaken the performance.

Figure 6: With the amount of the fused learners increasing, the ensemble DSC
will increase, and the structures have different sensitivity to the increasing
of the learners.

##  5 Conclusion

In this paper, we equips adversarial learning with ensemble segmentation
models, and propose the EnMcGAN, the first 3D CRS segmentation model, for
technical support of LPN. 1)Our multi-window committee divides CTA image into
narrow windows with different window centers and widths enhancing the contrast
and making fine-grained pattern, and constructs the ensemble model based on
these narrow windows fusing the segmentation superiorities on different
covered distributions. 2)Our multi-condition GAN utilizes the shape
constraints of adversarial losses to encourage the segmented renal structures
being consistent with their real shape, thus segmentation will tend to extract
the features that matches the priori shape. 3)Our adversarial weighted
ensemble module uses the trained discriminator to score the quality of each
structure from each learner for dynamic ensemble weights, enhancing the
ensemble results. Extensive experiments with promising results reveal powerful
3D CRS segmentation performance and significance in renal cancer treatment.

####  5.0.1 Acknowledgements.

This research was supported by the National Natural Science Foundation under
grants (31571001, 61828101, 31800825), Southeast University-Nanjing Medical
University Cooperative Research Project (2242019K3DN08) and Excellence Project
Funds of Southeast University. We thank the Big Data Computing Center of
Southeast University for providing the facility support on the numerical
calculations in this paper.

## References

  * [1] Breiman, L.: Bagging predictors. Machine learning 24(2), 123–140 (1996)
  * [2] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 424–432. Springer (2016)
  * [3] Dietterich, T.G., et al.: Ensemble learning. The handbook of brain theory and neural networks 2, 110–125 (2002)
  * [4] Goldman, L.W.: Principles of ct and ct technology. Journal of nuclear medicine technology 35(3), 115–128 (2007)
  * [5] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)
  * [6] He, Y., Yang, G., Chen, Y., Kong, Y., Wu, J., Tang, L., Zhu, X., Dillenseger, J.L., Shao, P., Zhang, S., et al.: Dpa-densebiasnet: Semi-supervised 3d fine renal artery segmentation with dense biased network and deep priori anatomy. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 139–147. Springer (2019)
  * [7] He, Y., Yang, G., Yang, J., Chen, Y., Kong, Y., Wu, J., Tang, L., Zhu, X., Dillenseger, J.L., Shao, P., Zhang, S., Shu, H., Coatrieux, J.L., Li, S.: Dense biased networks with deep priori anatomy and hard region adaptation: Semi-supervised learning for fine renal artery segmentation. Medical Image Analysis 63, 101722 (2020)
  * [8] Kingma, D.P., Ba, J.L.: Adam: A method for stochastic optimization. In: ICLR 2015 : International Conference on Learning Representations 2015 (2015)
  * [9] Li, J., Lo, P., Taha, A., Wu, H., Zhao, T.: Segmentation of renal structures for image-guided surgery. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 454–462. Springer (2018)
  * [10] Luc, P., Couprie, C., Chintala, S., Verbeek, J.: Semantic segmentation using adversarial networks. In: NIPS Workshop on Adversarial Training (2016)
  * [11] Milletari, F., Navab, N., Ahmadi, S.A., aaaaa: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV). pp. 565–571 (2016)
  * [12] Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  * [13] Mosinska, A., Marquez-Neila, P., Koziński, M., Fua, P.: Beyond the pixel-wise loss for topology-aware delineation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3136–3145 (2018)
  * [14] Ng, H., Ong, S., Foong, K., Goh, P., Nowinski, W.: Medical image segmentation using k-means clustering and improved watershed algorithm. In: 2006 IEEE Southwest Symposium on Image Analysis and Interpretation. pp. 61–65. IEEE (2006)
  * [15] Nicolau, S., Soler, L., Mutter, D., Marescaux, J.: Augmented reality in laparoscopic surgical oncology. Surgical oncology 20(3), 189–201 (2011)
  * [16] Oktay, O., Ferrante, E., Kamnitsas, K., Heinrich, M., Bai, W., Caballero, J., Cook, S.A., De Marvao, A., Dawes, T., O‘Regan, D.P., et al.: Anatomically constrained neural networks (acnns): application to cardiac image enhancement and segmentation. IEEE transactions on medical imaging 37(2), 384–395 (2017)
  * [17] Petru, B., Elena, Ş., Dan, I., Klara, B., Radu, B., Constantin, D.: Morphological assessments on the arteries of the superior renal segment. Surgical and radiologic anatomy 34(2), 137–144 (2012)
  * [18] Polikar, R.: Ensemble learning. In: Ensemble machine learning, pp. 1–34. Springer (2012)
  * [19] Porpiglia, F., Fiori, C., Checcucci, E., Amparore, D., Bertolo, R.: Hyperaccuracy three-dimensional reconstruction is able to maximize the efficacy of selective clamping during robot-assisted partial nephrectomy for complex renal masses. European urology 74(5), 651–660 (2018)
  * [20] Shao, P., Qin, C., Yin, C., Meng, X., Ju, X., Li, J., Lv, Q., Zhang, W., Xu, Z.: Laparoscopic partial nephrectomy with segmental renal artery clamping: technique and clinical outcomes. European urology 59(5), 849–855 (2011)
  * [21] Shao, P., Tang, L., Li, P., Xu, Y., Qin, C., Cao, Q., Ju, X., Meng, X., Lv, Q., Li, J., et al.: Precise segmental renal artery clamping under the guidance of dual-source computed tomography angiography during laparoscopic partial nephrectomy. European urology 62(6), 1001–1008 (2012)
  * [22] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR 2015 : International Conference on Learning Representations 2015 (2015)
  * [23] Taha, A., Lo, P., Li, J., Zhao, T.: Kid-net: Convolution networks for kidney vessels segmentation from ct-volumes. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 463–471 (2018)
  * [24] Xia, Y., Xie, L., Liu, F., Zhu, Z., Fishman, E.K., Yuille, A.L.: Bridging the gap between 2d and 3d organ segmentation with volumetric fusion net. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 445–453. Springer (2018)
  * [25] Zhang, S., Yang, G., Tang, L., Lv, Q., Li, J., Xu, Y., Zhu, X., Li, P., Shao, P., Wang, Z.: Application of a functional3-dimensional perfusion model in laparoscopic partial nephrectomy with precise segmental renal artery clamping. Urology 125, 98–103 (2019)