# Detecting Changed-Hands Online Review Accounts

Geli Fei†, Shuai Wang†, Bing Liu†, Leman Akoglu‡
†Department of Computer Science, University of Illinois at Chicago, USA
‡Heinz College of Information Systems and Public Policy, Carnegie Mellon
University, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

A reputable social media or review account can be a good cover for spamming
activities. It has become prevalent that spammers buy/sell such accounts
openly on the Web. We call these sold/bought accounts the changed-hands (CH)
accounts. They are hard to detect by existing spam detection algorithms as
their spamming activities are under the disguise of clean histories. In this
paper, we first propose the problem of detecting CH accounts, and then design
an effective detection algorithm which exploits changes in content and writing
styles of individual accounts, and a proposed novel feature selection method
that works at a fine-grained level within each individual account. The
proposed method not only determines if an account has changed hands, but also
pinpoints the change point. Experimental results with online review accounts
demonstrate the high effectiveness of our approach.

##  1 Introduction

Opinion spam has become a common type of spam in review sites such as Amazon
and Yelp, as people continue to heavily rely on online reviews to make
purchase decisions. Since the early work by Jindal and Liu (2008), detecting
fake reviews and reviewers have drawn wide attention from both the research
community and the industry. The problem has been investigated through
different approaches, including those based on content or linguistic
information Ott et al. (2011); Li et al. (2014b), reviewer behaviors Feng et
al. (2012); Ye and Akoglu (2015), temporal posting patterns Xie et al. (2012);
KC and Mukherjee (2016), and relational analysis Jiang et al. (2014); Rayana
and Akoglu (2015).

As a result of the advances in spam filtering techniques, spamming has become
harder than before. For example, giving all-extreme ratings or posting many
reviews in a short time frame can be easily caught. Driven by profits, opinion
spammers resort to other strategies. One strategy is to offer to buy reputable
accounts111https://www.yelp.com/topic/boston-someone-offered-to-buy-my-yelp-
account (those with a clean history) and use them to post spam reviews.
Selling/buying accounts is also prevalent in other forms of social media.
Karma
farmers222https://www.reddit.com/r/AgainstKarmaWhores/comments/383qsp/why\\_are\\_we\\_doing\\_this/
are such an example in the community website Reddit, who try to gain high
karma (upvotes and reputation) quickly with new accounts so that their posts
can show up in the front page, and then sell these seemingly reputable
accounts to spammers.

In both of the above situations, accounts change hands at a certain time point
and they unavoidably exhibit linguistic and writing style differences in the
midst of their life span. It is hard for a spammer to align his writing style
with the original account holder’s writing style for two reasons. First, there
is no simple manual way to quantify another user’s writing style in every
aspect. Second, since spammers have different objectives than legitimate
users, e.g., promoting some products, their writing styles change naturally.
To the best of our knowledge, such changes have not been studied before. This
paper represents the first work on the topic.

In this paper, we propose this new problem of detecting changed-hands (CH)
accounts from a content and writing style perspective. An algorithm, called
CHAD (CH Accounts Detection), is proposed to identify if an account has
changed hands and to estimate the time point of change if so. In case of a
change, we assume there is only one change in an account’s life time because
once there is a change, it should be detected before a second change happens.
Existing spammer detection methods are not suitable for detecting such
accounts, and cannot identify the change point for two reasons. (1) They
assume there is a single user behind each account. (2) They examine the
overall behavior of each account. For CH accounts, their spamming activities
may not be obvious given a clean history. This work thus complements the
existing review spam detection settings and algorithms.

Problem Definition: Given an account
$\boldsymbol{A}=\\{r_{1},r_{2},\dots,r_{n}\\}$ with reviews $r_{j}$ sorted by
their posting dates, CHAD determines whether a significant linguistic and/or
writing style change has occurred starting from a particular review $r_{i}$
($1<i<n$). The algorithm returns $i$ if yes, and returns none otherwise.

The problem has two unique challenges:

  1. 1.

Inter-user differences: Different CH accounts exhibit different changes,
because not every pair of users has the same differences in their writings.
For example, in some CH accounts, the two users can be distinguished by the
average length of the words they use. In some other CH accounts, the two users
may be distinguished by the average sentence length but not by the average
word length, because one uses long sentences while the other uses short ones,
but both of them mainly use short words.

  2. 2.

Intra-user variance: Every review is unique in some way, which results in a
certain amount of difference and variance even when compared with other
reviews of the same user. However, such differences do not indicate a real
changing of hands between two users.

Given these two challenges, a desired detection method needs to perform
detection at the account level and adjust itself to different individual
accounts. In this paper, we propose an effective detection algorithm with a
novel feature selection method, called pivot-level feature selection, to
address these challenges. The key novelty of this feature selection method is
that, due to the two challenges, it works at a fine-grained level within each
individual account rather than the whole dataset as traditional feature
selection methods do.

In summary, this paper makes the following contributions:

  1. 1.

It proposes the new problem of detecting CH accounts, which have become
prevalent in many social media sites, but have not been studied so far. This
new problem complements the existing spammer detection settings.

  2. 2.

It proposes a novel algorithm, CHAD, which leverages linguistic evidences and
a novel new feature selection algorithm to identify if an account has changed
hands during its life time and estimates the change point.

  3. 3.

It evaluates CHAD on two datasets and show that the proposed approach is
highly effective.

##  2 Related Work

Our work is related to opinion spam detection, tracking linguistic evolution
and change point detection.

###  2.1 Opinion Spam Detection

Since the first work by Jindal and Liu (2008), a wide range of techniques have
been proposed for detecting spam reviews Li et al. (2011, 2014a); Hai et al.
(2016), individual spammers Lim et al. (2010); Akoglu et al. (2013) and
spammer groups Mukherjee et al. (2012). However, the use of CH accounts as a
new instrument for spamming has not been studied thus far and no techniques
are available for their detection.

Among existing techniques, detecting spammer accounts is most relevant to our
problem. Lim et al. (2010) studied users’ rating behaviors; Akoglu et al.
(2013) studied the relational collusion between reviewers and their target
products; Mukherjee et al. (2013a) used a Bayesian approach to modeling the
behavioral patterns of spammers and non-spammers. These approaches cannot
detect CH accounts and pinpoint their change locations because they examine
the overall behavioral patterns of each account. The spamming activities of CH
accounts may go undetected given a clean history.

Sockpuppet detection Hosseinia and Mukherjee (2017) refers to the detection of
a single author behind multiple accounts. These methods cannot be directly
applied as they regard reviews from one account as written by a single author.

Our work is also related to using linguistic approaches to detecting spamming
reviews Ott et al. (2011); Ren et al. (2014) and loosely related to
psycholinguistic deception detection Newman et al. (2003); Pérez-Rosas et al.
(2015), as we also use a linguistic-based approach.

###  2.2 Linguistic Evolution & Change Point Detection

On tracking linguistic evolution across time, Juola (2003) quantified the rate
of change in language across two time periods, and Lijffijt et al. (2012)
studied lexical stability in a historical corpus. Our work is different
because the above works compare language from two chosen time periods, while
our goal is to estimate the change point from a sequence of documents.
Tracking shifts in the meaning of words was studied in Mitra et al. (2014);
Kulkarni et al. (2015). Our work does not study shifts of word meaning but
“shifts in authorship.” However, authorship attribution and verification
methods Koppel and Schler (2004); Sanderson and Guenter (2006) cannot be
applied as we don’t have any training data of the users. Change point
detection is a core time series analysis problem Taylor (2000). In our work,
we adopt the single change point detection technique by Chen and Gupta (1999),
as it aligns with our goal of detecting CH accounts.

##  3 Proposed CHAD Method

This section presents the proposed CHAD algorithm.

###  3.1 The Overall Algorithm

The main idea of CHAD is based on the observation that the reviews written by
one user are similar among themselves but different from those written by a
different user. The CHAD algorithm is outlined in Alg. 1, which works on one
account at a time. Note that it needs a pre-selected feature set
$\boldsymbol{F}$ as input, which is a subset of all features
$\boldsymbol{F}^{all}$ (We will explain this shortly). We first introduce the
five main steps, and then go into details of each step.

Algorithm 1 CHAD

Input: Account := $\boldsymbol{A}=\\{r_{1},r_{2},\dots,r_{n}\\}$,
Window size := $K$, Smoothing factor := $\lambda_{S}$
Features :=
$\boldsymbol{F}(\subseteq\boldsymbol{F}^{all})=\\{f_{1},f_{2},\dots,f_{m}\\}$
Output: Res := index $i$ (1<$i$<$n$) or none

1: $\boldsymbol{C}:=\text{\O}$ // $\boldsymbol{C}$ is a multiset for voting.

2: for each $r_{i}\in\boldsymbol{A}(1\leq i\leq n-K+1)$ do

3: $\boldsymbol{S}_{i}:=\text{\O}$ // $\boldsymbol{S}_{i}$ is a set of
similarity sequences for a pivot window.

4: $\textit{pivot-window}:=\\{r_{i},\dots,r_{i+K-1}\\}$

5: $\overline{\boldsymbol{A}}:=\boldsymbol{A}\;{\backslash}\;\textit{pivot-
window}$

6: for each $f_{j}\in\boldsymbol{F}$ do

7: $ss_{ij}:=\text{compute-sim-seq}(r_{i},K,\overline{\boldsymbol{A}},f_{j})$

8: $\boldsymbol{S}_{i}:=\boldsymbol{S}_{i}\cup\\{ss_{ij}\\}$

9: end for

10: $\boldsymbol{S}_{i}:=\text{pivot-level-feature-
select}(\boldsymbol{S}_{i})$

11: $s_{i}:=\text{aggregate}(\boldsymbol{S}_{i})$

12: $c_{i}:=\text{change-point-detect}(s_{i})$ // $c_{i}$ is either a review’s
temporal index or none.

13: $\boldsymbol{C}:=\boldsymbol{C}\cup\\{c_{i}\\}$

14: end for

15: $\textit{res}:=\text{is-change-vote}(\boldsymbol{C})$

16: if ($\textit{res}\neq\textit{none}$) then

17: $\boldsymbol{C}_{-none}:=$ remove none elements from $\boldsymbol{C}$

18:
$\boldsymbol{C}_{-none}^{S}:=\text{smooth}(\boldsymbol{C}_{-none},\lambda_{S})$

19: $\textit{res}:=\text{change-point-vote}(\boldsymbol{C}_{-none}^{S})$

20: end if

21: return res

  1. 1.

Generate similarity sequences (lines 2-9): For an input account
$\boldsymbol{A}$, this step builds a set of similarity sequences
$\boldsymbol{S}_{i}$ using features in $\boldsymbol{F}$ for a pivot window of
$K$ reviews starting from a review $r_{i}$. Each sequence
$ss_{ij}\in\boldsymbol{S}_{i}$ is computed by comparing the similarity of
reviews in the pivot window and reviews within a moving window of also size
$K$ in the remaining reviews $\overline{\boldsymbol{A}}$ using one ($f_{j}$)
of the features in $\boldsymbol{F}$ (line 7). For a CH account, we expect the
similarities to be high when comparing reviews written by the same user, but
low across two different users.

  2. 2.

Eliminate noisy features (line 10)

  3. 3.

Aggregate sequences (line 11): We aggregate the remaining sequences for each
pivot window by averaging the sequences in the resulting $\boldsymbol{S}_{i}$.

  4. 4.

Change-point detection (line 12): We employ a statistical algorithm for
change-point detection on each aggregated sequence $s_{i}$ to detect the
change point.

  5. 5.

Two-round voting (line 15-20): We perform two rounds of voting on the change-
point detection results on the aggregated sequences of all pivot windows for
an account to determine if a changing of hands has occurred and also to
identify the final change point.

Global feature pre-selection: As mentioned before, CHAD requires a pre-
selected feature set $\boldsymbol{F}$ as input. $\boldsymbol{F}$ is selected
globally by running Alg. 1 without line 10 (pivot-level feature selection) on
all accounts of a development set for multiple iterations starting with all
features $\boldsymbol{F}^{all}$ as input. Each iteration removes one feature
from $\boldsymbol{F}^{all}$ that gives the biggest performance gain in F1
score under the change-point evaluation ($\textit{eval}_{\textit{cp}}$) (Sec.
4.1) . It globally removes those noisy features in $\boldsymbol{F}^{all}$.

###  3.2 Features and Similarity Metrics

Now we list the set of all features $\boldsymbol{F}^{all}$ used in the
compute-sim-seq function (line 7) in Alg. 1. Features with * produce a single
value for reviews in a given window and the rest use the Bag-of-Words (BoW)
model. Single value features include average sentence length* and average
token length*. BoW features include word unigrams, word bigrams, Part-of-
Speech unigrams, Part-of-Speech bigrams, adjectives & adverbs, nouns, function
words, and punctuations.

To measure the similarity between reviews in two review windows, we use cosine
similarity for BoW features. We tried some other measures such as Jaccard
similarity but they did not perform well. For single value features, we
compute the similarity sim using their absolute difference diff and
normalizing it to [0,1]:

$\textit{sim}=1/(1+log(1+\textit{diff}))$ (1)

Algorithm 2 Pivot-level-Feature-Select

Input: $\boldsymbol{S}_{i}$ := {$ss_{i1}$, $ss_{i2}$, …, $ss_{iT}$}
Output: $\boldsymbol{E}$ := the set of selected sequences

1: target := avg($\boldsymbol{S}_{i}$)

2: $\boldsymbol{E}=\text{\O}$

3:
$\boldsymbol{S}_{i}^{{}^{\prime}}=\text{sort}(\\{\text{Pc}(ss_{ij}\in\boldsymbol{S}_{i},target)\\})$
// computes Pearson’s correlation (Pc) of each $ss_{it}\in\boldsymbol{S}_{i}$
to target and sort in descending order.

4:
$\boldsymbol{E}:=\boldsymbol{E}\cup\\{\boldsymbol{S}_{i}^{{}^{\prime}}[1]\\}$
// adds the sequence with highest correlation to $\boldsymbol{E}$.

5: for $j\in\\{2:T\\}$ do

6: $p:=$ avg$(\boldsymbol{E})$

7: $l:=\boldsymbol{S}_{i}^{{}^{\prime}}[j]$

8: if
$\text{Pc}(\text{avg}(\boldsymbol{E}\cup\\{l\\}),\textit{target})>\text{Pc}(p,\textit{target})$
then

9: $\boldsymbol{E}:=\boldsymbol{E}\cup\\{l\\}$

10: else

11: break

12: end if

13: end for

14: return $\boldsymbol{E}$

###  3.3 Pivot-Level Feature Selection

Now we describe the pivot-level-feature-select function in line 10 of Alg. 1.
As we pointed out earlier, one of our key challenges is that the writing
differences between a pair of users in one CH account may be different from
those between other pairs in other CH accounts. Furthermore, each review is
unique in some way which can result in a certain amount of difference when
computing similarity with other reviews using some features. Such differences
however may not indicate real writing differences between two users. To solve
these two problems, we propose to perform pivot-level feature selection (Alg.
2). The corresponding similarity sequence of each removed feature is deleted
from $\boldsymbol{S}_{i}$.

The pivot-level-feature-select function selects a subset of sequences in
$\boldsymbol{S}_{i}$ through correlation analysis, which is the same as
selecting their corresponding features. It first averages all sequences in
$\boldsymbol{S}_{i}$ to construct a target sequence (line 1). It then computes
Pearson’s correlation ($Pc$) of each sequence in $\boldsymbol{S}_{i}$ with the
target and sorts the sequences based on correlation strength in descending
order (line 3). Line 4 adds the sequence with the highest correlation to the
result set $\boldsymbol{E}$. It then goes through the sorted sequence set
$\boldsymbol{S}_{i}^{{}^{\prime}}$ and tests if adding another sequence to
$\boldsymbol{E}$ would increase $\boldsymbol{E}$’s average’s correlation to
target (lines 5-13). If the correlation improves by the addition, we update
$\boldsymbol{E}$; otherwise exit and return $\boldsymbol{E}$.

The intuition is that target is a representative sequence assuming only a few
noisy sequences exist. Through correlation analysis, we identify the sequences
that align with the target and discard those outlying ones that otherwise
hinder the change point detection performance. Fig. 1 gives an example of
several similarity sequences computed for a pivot window on a CH account in
the Amazon dataset, where the actual change happens at review #119. For
clarity, we only plot a subset of sequences generated using 4 features.
Through correlation analysis, our Alg. 2 is able to effectively eliminate the
noisy sequence generated from the feature Adj&Adv.

Figure 1: Some sample sequences in an $\boldsymbol{S}_{i}$.

###  3.4 Change Point Detection

As mentioned in the introduction, we assume there is at most one change point
in each account. As such, we use the single point change detection algorithm
by Chen and Gupta (1999), which uses the Schwarz Information Criterion (SIC)
to search for the change point. Suppose $X_{1},X_{2},\dots,X_{n}$ is a
sequence of independent Gaussian random variables with means
$\mu_{1},\mu_{2},\dots,\mu_{n}$ and variances
$\sigma_{1}^{2},\sigma_{2}^{2},\dots,\sigma_{n}^{2}$, respectively. The method
tests the hypothesis of whether there is a single change in both the mean and
variance located at the unknown position $k$, $2\leq k\leq n-1$ as:

$\begin{split}H_{0}:&\mu_{1}=\mu_{2}=\dots=\mu_{n}=\mu\text{ and}\\\
&\sigma_{1}^{2}=\sigma_{2}^{2}=\dots=\sigma_{n}^{2}=\sigma^{2},\end{split}$

versus the alternative hypothesis

$\begin{split}H_{1}:&\mu_{1}=\dots=\mu_{k}\neq\mu_{k+1}=\dots=\mu_{n}\text{
and}\\\
&\sigma_{1}^{2}=\dots=\sigma_{k}^{2}\neq\sigma_{k+1}^{2}=\dots=\sigma_{n}^{2},\end{split}$

where $\mu$ and $\sigma^{2}$ are unknown common parameters when there is no
change. We thus have two models corresponding to the $H_{0}$ and $H_{1}$. The
principle of minimum SIC is used to reject $H_{0}$. In particular, $H_{0}$ is
not rejected if $\text{SIC}(n)\leq\text{min}_{k}\text{SIC}(k)$, and rejected
otherwise. $SIC(n^{\prime})$ is defined as
$-2\text{log}L(\hat{\Theta})+p\text{log}n^{\prime}$, where $L(\hat{\Theta})$
is the maximum likelihood function for each model, $p$ is the degrees of
freedom in the model ($p=2$ under $H_{0}$ and $p=4$ under $H_{1}$), and
$n^{\prime}$ is the sample size.

###  3.5 Two-Round Voting

In Alg. 1, CHAD uses a two-round voting scheme (step 5) to determine if an
account has changed hands and also to pinpoint the location of change (lines
15-20). In the first round (line 15), is-change-vote function determines if a
change has occurred. Note that each element $c_{i}\in\boldsymbol{C}$ returned
by the change point detection algorithm is either a change point (i.e., a
review) or none (indicating no change). This function simply counts the number
of votes for each change point and none. If none has the highest number of
votes, it returns none; otherwise it registers that a change has occurred and
removes all the none elements from $\boldsymbol{C}$ (line 17). It then moves
on to the second round of voting to pinpoint the actual change location.
Instead of directly voting based on elements in $\boldsymbol{C}_{-none}$, we
perform smoothing on $\boldsymbol{C}_{-none}$ first (line 18). Let us look at
an example. Given a set of votes in $\boldsymbol{C}_{-none}$ in the format of
change-point:#-of-votes 10:8, 50:5, 51:7, 52:4, the point that gets the
highest votes is $10$. However, the actual change point is more likely to be
around $51$. In order to overcome this possible noise factor, we smooth the
votes by adding some extra counts to near-by locations of every change point
in $\boldsymbol{C}_{-none}$ to construct $\boldsymbol{C}_{-none}^{S}$.
Specifically, we pick a smoothing factor $\lambda_{S}\in\mathbb{Z}_{>0}$ and
for a change point $i$ with $v$ votes, we add $v/\lambda_{S}^{dist}$ extra
votes to locations $i+\textit{dist}$ and $i-\textit{dist}$, where
$\textit{dist}=1,2,\dots$. Finally, we perform the second round of voting on
$\boldsymbol{C}_{-none}^{S}$ to determine the final change location.

##  4 Experiments

###  4.1 Datasets and Evaluation Metrics

Datasets: For experiments we constructed synthetic datasets for the following
reasons: First, no publicly available labeled data exists for our problem;
Second, identifying opinion spam manually has been shown to be very unreliable
Ott et al. (2011); Lastly, although Mechanical Turkers have been used to write
individual fake reviews Ott et al. (2011), our case is much more complicated
because of different sizes and the diversity of reviewed products for
different accounts. In fact, synthetic data was used before, e.g., in
sockpuppet detection Qian and Liu (2013). In this work, we use two public
review corpora to construct our data, one from Amazon Jindal and Liu (2008),
which contains reviews for multiple product categories such as books,
electronics, etc., and the other from Yelp Mukherjee et al. (2013b), which
contains only hotel reviews. The construction of CH accounts from each corpus
is done as follows: We first randomly select two different original accounts
with at least 10 reviews,
$\boldsymbol{A}_{1}=\\{r_{11},r_{12},\dots,r_{1n}\\}$ and
$\boldsymbol{A}_{2}=\\{r_{21},r_{22},\dots,r_{2n^{\prime}}\\}$, both sorted by
their review posting dates, and then concatenate one account to the other,
giving us
$\boldsymbol{A}_{syn}=\\{r_{11},r_{12},\dots,r_{1n},r_{21},r_{22},\dots,r_{2n^{\prime}}\\}$,
whose minimum size is 20. We in total constructed 350 CH accounts. Then we
sample 350 original accounts with at least 20 reviews as non-CH (NCH) accounts
that approximately match the mean and standard deviation of the sizes of the
constructed CH accounts by following the 68-95-99.7 rule from statistics
333https://en.wikipedia.org/wiki/68-95-99.7_rule. This way of sampling (rather
than random sampling) is important because it ensures that the NCH and CH
accounts have similar number of reviews, which eliminates the bias due to
joining two accounts in constructing CH accounts that can result in
significantly more reviews for CH accounts than for NCH accounts. Thus for
each dataset, we in total created 700 accounts. Statistics of the size of the
accounts in both datasets are given in Table 1. In Sec. 4.5 , we will show the
results when the datasets are constructed in a different way.

Although it is possible that the original corpora already contain some CH
accounts, we believe the chance of selecting existing CH accounts is very
small because the original corpora are very large. Also, we believe it is
reasonable to study accounts with more than 20 reviews because of our problem
setting, i.e., an account changes hand after it has gained enough “reputation”
or a long history.

We choose to use the Amazon and Yelp corpora for the following reason. The
Amazon corpus has reviews of all kinds of products. We use it to create the
scenario where a spammer buys an account and uses it to review products that
are potentially very different from those reviewed by the original user
(although we do not enforce this when constructing the dataset). Moreover,
products reviewed by a single user can also be quite diverse. In contrast, the
Yelp corpus has only hotel reviews, which allows us to show whether our
approach can detect CH accounts when the two users wrote reviews for the same
type of entities (i.e. when the content change is not as drastic). As we will
show, CHAD is able to perform well in both scenarios.

| Mean | Med. | Stdev | Min | Max
---|---|---|---|---|---
$Amazon_{NCH}$ | 60.0 | 40 | 37 | 20 | 170
$Amazon_{CH}$ | 53.6 | 41 | 33.2 | 20 | 231
${Yelp}_{NCH}$ | 47.7 | 43 | 14.7 | 20 | 88
${Yelp}_{CH}$ | 45.9 | 41 | 13.7 | 20 | 84
Table 1: Review Number Statistics. | $\textit{eval}_{\textit{{cha}}}$ | $\textit{eval}_{\textit{{cp}}}$
---|---|---
Feature Name | Prec. | Recall | F1 | Accu. | Prec. | Recall | F1 | Accu.
OS-AvgSentLen | 0.590 | 0.68 | 0.632 | 0.604 | 0.344 | 0.552 | 0.424 | 0.463
OS-AvgTokenLen | 0.6 | 0.530 | 0.563 | 0.588 | 0.363 | 0.406 | 0.383 | 0.484
OS-Punctuations_Ratio | 0.541 | 0.756 | 0.630 | 0.557 | 0.249 | 0.587 | 0.35 | 0.354
OF-Unigrams | 0.732 | 0.62 | 0.669 | 0.696 | 0.602 | 0.573 | 0.584 | 0.64
OF-Bigrams | 0.714 | 0.633 | 0.669 | 0.688 | 0.547 | 0.570 | 0.556 | 0.615
OF-Punctuations | 0.650 | 0.638 | 0.643 | 0.646 | 0.468 | 0.558 | 0.507 | 0.555
CHAD-PFS | 0.804 | 0.646 | 0.716 | 0.744 | 0.707 | 0.616 | 0.658 | 0.705
CHAD-F | 0.752 | 0.711 | 0.731 | 0.738 | 0.642 | 0.677 | 0.659 | 0.686
CHAD | 0.778 | 0.726 | 0.751 | 0.759 | 0.680 | 0.699 | 0.688 | 0.713
Table 2: Performance results on the Amazon dataset. | $\textit{eval}_{\textit{{cha}}}$ | $\textit{eval}_{\textit{{cp}}}$
---|---|---
Feature Name | Prec. | Recall | F1 | Accu. | Prec. | Recall | F1 | Accu.
OS-AvgSentLen | 0.614 | 0.702 | 0.655 | 0.631 | 0.364 | 0.582 | 0.448 | 0.488
OS-AvgTokenLen | 0.647 | 0.538 | 0.587 | 0.622 | 0.343 | 0.381 | 0.361 | 0.496
OS-Punctuations_Ratio | 0.547 | 0.78 | 0.643 | 0.567 | 0.248 | 0.617 | 0.354 | 0.354
OF-Unigrams | 0.822 | 0.697 | 0.754 | 0.772 | 0.698 | 0.662 | 0.678 | 0.72
OF-Bigrams | 0.790 | 0.698 | 0.741 | 0.756 | 0.648 | 0.655 | 0.651 | 0.693
OF-Punctuations | 0.711 | 0.696 | 0.702 | 0.706 | 0.512 | 0.622 | 0.561 | 0.608
CHAD-PFS | 0.851 | 0.724 | 0.782 | 0.798 | 0.709 | 0.686 | 0.695 | 0.736
CHAD-F | 0.825 | 0.781 | 0.8 | 0.805 | 0.684 | 0.748 | 0.711 | 0.738
CHAD | 0.864 | 0.769 | 0.813 | 0.824 | 0.745 | 0.742 | 0.744 | 0.771
Table 3: Performance results on the Yelp dataset.

Evaluation Schemes and Metrics: We use two evaluation schemes, CH-accounts
detection evaluation ($\textit{eval}_{\textit{{cha}}}$) and change-point
detection  evaluation ($\textit{eval}_{\textit{{cp}}}$), and report
corresponding precision, recall, F1, and accuracy on both tasks. For
$\textit{eval}_{\textit{{cha}}}$, we only identify CH accounts but not the
actual change locations. For $\textit{eval}_{\textit{{cp}}}$, we go one step
further to also evaluate the identified change locations. Since it is hard to
identify the exact change point (the review) at which a change-of-hands has
occurred, we define a window $x\pm y$ around the actual change point $x$ with
window size $y$, and consider the predicted change point as accurate if it
resides within the window. When a change point is detected for a non-CH
account, it is considered an error. We study the performance by varying $y$ in
Sec. 4.4.

All evaluation results are based on averaging the results of 5 runs on the
constructed datasets. Each time we randomly sample $200$ accounts from each
dataset, $100$ in each class, as the development set and use the rest $500$
accounts as the test set. Statistical significance tests are also performed.

###  4.2 Baselines

Since there is no previous work on detecting CH accounts, we propose the
following baselines:

  * •

One Sequence (OS). The simplest approach to detecting CH accounts is to
construct a single sequence of feature values directly from a moving window of
reviews of size $K$ (one similarity value per review window) of an account and
use it to run a change point detection algorithm. The set of features we tried
includes: average sentence length, average token length, ratio of nouns, ratio
of adjectives and adverbs, ratio of function words, and ratio of punctuations
and special characters. This baseline thus produces 6 results named with OS-
as the prefix.

  * •

One Feature (OF). This baseline is a variant of CHAD. It only uses one of the
features from $\boldsymbol{F}^{all}$ as input. Thus, lines 10-11 in Alg. 1 do
not have any effect. Since $\boldsymbol{F}^{all}$ contains 10 features, this
baseline produces 10 different results, which are named with OF- as the
prefix.

  * •

CHAD w/out Pivot-level Feature Selection (CHAD-PFS). This baseline is another
variant of CHAD. It does not perform pivot-level feature selection in Alg. 1.
It employs the same procedure to pre-select a feature set and thus shares the
same $\boldsymbol{F}$ with CHAD.

  * •

CHAD w/out Pre-selecting F (CHAD-F). This is a variant of CHAD that directly
uses $\boldsymbol{F}^{all}$ without pre-selecting feature set $\boldsymbol{F}$
as input.

Note that we do not compare with existing spam filtering or sockpuppet
detection methods as they all regard reviews from one account as written by a
single author. Thus, none of them is able to detect stylistic changes within
an account.

###  4.3 Parameter Settings

For each run, we use the respective development set of each dataset to set
parameters. For both datasets, $K=5$ was chosen for the window size in
constructing similarity sequences, and $\lambda_{S}=2$ was chosen for vote
smoothing. For change-point detection, we use the implementation in R
changepoint package Killick and Eckley (2014), which outputs an estimated
change point along with its confidence level. We set a confidence level
threshold of $\theta_{conf}=0.99$ based on the development set, and consider
any detected change point with confidence level lower than $\theta_{conf}$ as
no change (none). In parameter selection and in the main results reporting, we
use $y=5$ for $\textit{eval}_{\textit{{cp}}}$, and later show the results by
varying $y$.

We make the following remarks about these parameters. First, using a very
small $K$ (e.g., 1 or 2) leads to bad performance due to the high variance in
similarities between reviews. On the other hand, while using a large $K$
(e.g., 7 or 8) improves results for CH accounts detection
($\textit{eval}_{\textit{{cha}}}$), the performance of change-point detection
($\textit{eval}_{\textit{{cp}}}$) drops due to loss of granularity. Second, it
is important to set the confidence level threshold $\theta_{conf}$ high to
consider only the most confident detections—due to the fact that each review
is unique in some way, and the constructed similarity sequences unavoidably
fluctuate to a large extent.

###  4.4 Main Results and Analysis

We present our main results on Amazon and Yelp datasets, respectively in
Tables 2 and 3. For each dataset we only list the best-3 OS results, best-3 OF
results, CHAD-PFS, CHAD-F and our proposed CHAD.

First, CHAD significantly outperforms all the baselines ($p<0.03$) on both
datasets. CHAD-F, which only performs pivot-level feature selection, and CHAD-
PFS, which only performs global feature pre-selection are both worse.

Second, the best performing OS and OF baselines are consistent on both
datasets. The results of OS baselines are quite poor, for which there are two
possible explanations. First, they rely on computing a single value as
feature, which may not be sufficient in capturing the differences between
users in CH accounts. Second, they construct only one sequence, which is less
reliable. Although OF baselines generally perform better, they are not
reliable for the same reason.

Comparing the results on two datasets, we found that better results are
generally achieved on Yelp dataset than on Amazon dataset. We believe the
difference is mainly caused by the nature of the two datasets. Detecting CH
accounts and their change locations are generally harder on Amazon dataset
because it contains numerous categories of products. An Amazon reviewer is
likely to post reviews on a variety of products, which creates big variance
when computing similarities.

$y$ | = 1 | = 3 | = 5 | = 7
---|---|---|---|---
Amazon
CHAD-PFS | 0.129 | 0.473 | 0.658 | 0.676
CHAD-F | 0.133 | 0.469 | 0.658 | 0.675
CHAD | 0.161 | 0.5 | 0.688 | 0.703
Yelp
CHAD-PFS | 0.138 | 0.543 | 0.695 | 0.725
CHAD-F | 0.131 | 0.568 | 0.711 | 0.749
CHAD | 0.163 | 0.596 | 0.743 | 0.768
Table 4: Effect of varying $y$ on $\textit{eval}_{\textit{cp}}$.

Lastly, we investigate the effect of window size $y$ (Sec. 4.1) under change-
point evaluation ($\textit{eval}_{\textit{{cp}}}$) by varying $y=1,3,5,7$. We
report the results for CHAD-PFS, CHAD-F, and CHAD in Table 4. Only results for
$\textit{eval}_{\textit{{cp}}}$ are listed as those for
$\textit{eval}_{\textit{{cha}}}$ are not affected by $y$. Only average F1
scores are given. As we can see, CHAD significantly outperforms the rest
methods ($p<0.03$) regardless of $y$. And as expected, all results improve for
increased values of $y$.

###  4.5 Experiments in Another Setting

For our main results above, we constructed two datasets in which the CH
accounts and non-CH accounts have similar distributions in their sizes
(numbers of reviews). In reality, this may not always be the case. CH accounts
may in general contain more reviews because spammers may write a lot of fake
reviews after purchasing the accounts. In order to test the performance of our
method in such cases, we constructed two different datasets without matching
the size distributions of CH and non-CH accounts. In particular, for both
Amazon and Yelp corpora, we randomly sample 700 original accounts with at
least 20 reviews, in which 350 are directly used as non-CH accounts, and the
rest are used as the first accounts in CH accounts. Then we randomly sample
another 350 accounts with at least 10 reviews as the second accounts in CH
accounts. The reason we select at least 20 reviews for the first accounts is
because we assume accounts change hands after a sufficiently long history.
Statistics of the data is shown in Table 5. We only report F1 scores of two
strong baselines (CHAD-PFS and CHAD-F), as well as our CHAD under both
evaluation schemes (Table 6). As we can see, CHAD again performs the best in
the new datasets. We also notice the CHAD performs better here than in the
previous set of experiments (Table 2 and 3). The reason is that in the
previous experiments setting, there are fewer reviews from the first
user/reviewer in the CH accounts, which makes it harder for the algorithm to
find reliable patterns.

| Mean | Med. | Stdev | Min | Max
---|---|---|---|---|---
$Amazon_{NCH}$ | 42.1 | 30 | 30.3 | 20 | 205
$Amazon_{CH}$ | 62.8 | 52 | 34.8 | 30 | 228
${Yelp}_{NCH}$ | 44.4 | 41 | 19.1 | 20 | 85
${Yelp}_{CH}$ | 72.3 | 65 | 24.4 | 30 | 142
Table 5: Size statistics of the new data. | Amazon | Yelp
---|---|---
| $\textit{eval}_{\textit{{cha}}}$ | $\textit{eval}_{\textit{{cp}}}$ | $\textit{eval}_{\textit{{cha}}}$ | $\textit{eval}_{\textit{{cp}}}$
CHAD-PFS | 0.805 | 0.726 | 0.844 | 0.752
CHAD-F | 0.815 | 0.731 | 0.883 | 0.771
CHAD | 0.816 | 0.753 | 0.886 | 0.786
Table 6: Results on the new data ($y=5$).

##  5 Conclusion

This paper proposed the new problem setting of detecting changed-hands
accounts which complements the existing spammer detection settings and
problems. To the best of our knowledge, the problem has not been explored
before. The problem presents some unique challenges due to the differences in
intra-user and inter-user writing styles. We presented a novel detection
algorithm to determine if an account has changed hands and the possible change
point. Extensive experiments on two datasets constructed using Amazon and Yelp
review data showed that our method outperforms a list of baselines
significantly.

## References

  * Akoglu et al. [2013] Leman Akoglu, Rishi Chandy, and Christos Faloutsos.  Opinion fraud detection in online reviews by network effects.  ICWSM, pages 2–11, 2013.
  * Chen and Gupta [1999] Jie Chen and AK Gupta.  Change point analysis of a gaussian model.  Statistical Papers, 1999.
  * Feng et al. [2012] Song Feng, Longfei Xing, Anupam Gogar, and Yejin Choi.  Distributional footprints of deceptive product reviews.  ICWSM, 12:98–105, 2012.
  * Hai et al. [2016] Zhen Hai, Peilin Zhao, Peng Cheng, Peng Yang, Xiao-Li Li, Guangxia Li, and Ant Financial.  Deceptive review spam detection via exploiting task relatedness and unlabeled data.  In EMNLP, 2016.
  * Hosseinia and Mukherjee [2017] Marjan Hosseinia and Arjun Mukherjee.  Detecting sockpuppets in deceptive opinion spam.  arXiv preprint arXiv:1703.03149, 2017.
  * Jiang et al. [2014] Meng Jiang, Peng Cui, Alex Beutel, Christos Faloutsos, and Shiqiang Yang.  Catchsync: catching synchronized behavior in large directed graphs.  In KDD, 2014.
  * Jindal and Liu [2008] Nitin Jindal and Bing Liu.  Opinion spam and analysis.  In WSDM, pages 219–230, 2008.
  * Juola [2003] Patrick Juola.  The time course of language change.  Computers and the Humanities, 2003.
  * KC and Mukherjee [2016] Santosh KC and Arjun Mukherjee.  On the temporal dynamics of opinion spamming: Case studies on yelp.  In WWW, pages 369–379, 2016.
  * Killick and Eckley [2014] Rebecca Killick and Idris A. Eckley.  changepoint: An R package for changepoint analysis.  Journal of Statistical Software, 2014.
  * Koppel and Schler [2004] Moshe Koppel and Jonathan Schler.  Authorship verification as a one-class classification problem.  In ICML, page 62, 2004.
  * Kulkarni et al. [2015] Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena.  Statistically significant detection of linguistic change.  In WWW, 2015.
  * Li et al. [2011] Fangtao Li, Minlie Huang, Yi Yang, and Xiaoyan Zhu.  Learning to identify review spam.  In IJCAI, 2011.
  * Li et al. [2014a] Huayi Li, Zhiyuan Chen, Bing Liu, Xiaokai Wei, and Jidong Shao.  Spotting fake reviews via collective positive-unlabeled learning.  In ICDM, 2014.
  * Li et al. [2014b] Jiwei Li, Myle Ott, Claire Cardie, and Eduard H Hovy.  Towards a general rule for identifying deceptive opinion spam.  In ACL. Citeseer, 2014.
  * Lijffijt et al. [2012] Jefrey Lijffijt, Tanja Säily, and Terttu Nevalainen.  Ceecing the baseline: Lexical stability and significant change in a historical corpus.  In Studies in Variation, Contacts and Change in English. 2012.
  * Lim et al. [2010] Ee-Peng Lim, Viet-An Nguyen, Nitin Jindal, Bing Liu, and Hady Wirawan Lauw.  Detecting product review spammers using rating behaviors.  In CIKM, pages 939–948, 2010.
  * Mitra et al. [2014] Sunny Mitra, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal.  That’s sick dude!: Automatic identification of word sense change across different timescales.  arXiv preprint arXiv:1405.4392, 2014.
  * Mukherjee et al. [2012] Arjun Mukherjee, Bing Liu, and Natalie Glance.  Spotting fake reviewer groups in consumer reviews.  In WWW, pages 191–200, 2012.
  * Mukherjee et al. [2013a] Arjun Mukherjee, Abhinav Kumar, Bing Liu, Junhui Wang, Meichun Hsu, Malu Castellanos, and Riddhiman Ghosh.  Spotting opinion spammers using behavioral footprints.  In KDD, pages 632–640, 2013.
  * Mukherjee et al. [2013b] Arjun Mukherjee, Vivek Venkataraman, Bing Liu, and Natalie S Glance.  What yelp fake review filter might be doing?  In ICWSM, 2013.
  * Newman et al. [2003] Matthew L Newman, James W Pennebaker, Diane S Berry, and Jane M Richards.  Lying words: Predicting deception from linguistic styles.  Personality and social psychology bulletin, 2003.
  * Ott et al. [2011] Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T Hancock.  Finding deceptive opinion spam by any stretch of the imagination.  In ACL, pages 309–319, 2011.
  * Pérez-Rosas et al. [2015] Verónica Pérez-Rosas, Mohamed Abouelenien, Rada Mihalcea, Yao Xiao, CJ Linton, and Mihai Burzo.  Verbal and nonverbal clues for real-life deception detection.  In EMNLP, pages 2336–2346, 2015.
  * Qian and Liu [2013] Tieyun Qian and Bing Liu.  Identifying multiple userids of the same author.  In EMNLP, pages 1124–1135, 2013.
  * Rayana and Akoglu [2015] Shebuti Rayana and Leman Akoglu.  Collective opinion spam detection: Bridging review networks and metadata.  In KDD, pages 985–994, 2015.
  * Ren et al. [2014] Yafeng Ren, Donghong Ji, and Hongbin Zhang.  Positive unlabeled learning for deceptive reviews detection.  In EMNLP, pages 488–498, 2014.
  * Sanderson and Guenter [2006] Conrad Sanderson and Simon Guenter.  Short text authorship attribution via sequence kernels, markov chains and author unmasking: An investigation.  In EMNLP, 2006.
  * Taylor [2000] Wayne A Taylor.  Change-point analysis: a powerful new tool for detecting changes.  preprint, available as http://www. variation. com/cpa/tech/changepoint. html, 2000.
  * Xie et al. [2012] Sihong Xie, Guan Wang, Shuyang Lin, and Philip S Yu.  Review spam detection via temporal pattern discovery.  In KDD, pages 823–831, 2012.
  * Ye and Akoglu [2015] Junting Ye and Leman Akoglu.  Discovering opinion spammer groups by network footprints.  In ECML-PKDD, pages 267–282, 2015.