# A Framework for Fast Polarity Labelling of Massive Data Streams

Huilin Wu  ,  Mian Lu  ,  Zhao Zheng  4Paradigm Inc.
<EMAIL_ADDRESS>and  Shuhao Zhang  Singapore
University of Technology and Design<EMAIL_ADDRESS>
###### Abstract.

Many of the existing sentiment analysis techniques are based on supervised
learning, and they demand the availability of valuable training datasets to
train their models. When dataset freshness is critical, the annotating of high
speed unlabelled data streams becomes critical but remains an open problem. In
this paper, we propose PLStream, a novel Apache Flink-based framework for fast
polarity labelling of massive data streams, like Twitter tweets or online
product reviews. We address the associated implementation challenges and
propose a list of techniques including both algorithmic improvements and
system optimizations. A thorough empirical validation with two real-world
workloads demonstrates that PLStream is able to generate high quality labels
(almost $80\%$ accuracy) in the presence of high-speed continuous unlabelled
data streams (almost $16,000$ tuples/sec) without any manual efforts.

sentiment classification, stream processing, distributed computing

††ccs: Computing methodologies Massively parallel algorithms††ccs: Computing
methodologies Natural language processing

##  1\. Introduction

Companies have utilized sentiment analysis (Capuano and et al., 2021) in many
business services to gain useful insights, such as the opinions of new
products. Although unsupervised methods exist, many sentiment analysis tasks
still work with supervised learning models (Zimbra and et al., 2018). Their
effectiveness thus highly depends on the quantity and quality of training
data. Despite the popularity of applying unsupervised pre-trained models such
as GPT-3 and BERT (Devlin and et al., 2018) in sentiment analysis, a
supervised downstream fine-tuning task is still required to achieve
outstanding performance, such as model fine-tuning for sentiment analysis. A
large amount of labelled datasets is still required, and it is costly to label
the datasets with human efforts.

Motivated by such a demand, Iosifidis et al. (Iosifidis and Ntoutsi, 2017)
have recently introduced a semi-supervised labelling approach to annotate
polarity labels. However, their are focusing on theoretical accuracy
guarantees with self-learning and co-training techniques, but overlook the
annotating speed as well as the generality of the algorithm. In particular, it
is a well-established industry practice to use the most recent data to train
machine learning models (Ramanath and et al., 2021). New datasets are
continuously generating in the form of data streams, such as twitter tweets
and online customer reviews. When dataset freshness is critical to sentiment
analysis, frequent offline label retraining and deployment are constrained by
the cost of manual labelling, limited computation and storage resources.

Figure 1 further shows the evaluation results of the existing self-learning
approach (Iosifidis and Ntoutsi, 2017) to annotate polarity labels of tweets.
We let it first train an initial model with 5000 labelled data. Then, it uses
the high-confidence predicted results as new training data for iterative
supervised training of the model, where the rest unlabelled data (with less
confidence) and new input data are recollected for the next training
iteration. We can see that, the existing self-learning approach is incapable
in handling data streams as it involves more and more datasets for training
leading to an exponentially growing training time. It is not able to make any
further progress when number of processed input tuples exceed few hundred
thousands of tuples (i.e., tweets), but input data stream is potentially
infinite. Henceforth, we do not further evaluate such self-learning
approaches, and a new approach to continuously annotate high volumes of data
streams is required.

Figure 1. Comparing the performance of polarity labelling of increasing volume
of datasets on 48 cores: an existing approach (Self-learning), and our
framework PLStream.

In this work, we propose PLStream, a novel framework that _continuously
annotates high quality polarity labels to massive ongoing streams of
opinions_. As Figure 1 shows, PLStream is able to achieve steady high
throughput when annotating polarity labels of data streams. The key of
PLStream is a novel online sentiment learning paradigm, which updates a word-
embedding model (e.g., word2vec (Mikolov et al., 2013)) incrementally and
generates labels by referencing the learned vector representations to a small
reference table of vocabularies with assumed polarities. A list of algorithmic
improvements and system optimization are further developed to help PLStream
achieve both high performance and high labelling correctness. To ensure its
high scalability and robustness, we have implemented PLStream on top of Apache
Flink (Carbone and et al., 2015) – a production level data stream processing
system (DSPS). Nevertheless, our proposed solution is general and can be
applied to other DSPSs such as Spark (Zaharia and et al., 2012) with marginal
programming efforts.

For a comprehensive comparison, we have further implemented several
alternative methods, including statistics-based (Lin and He, 2009), lexicon-
based (Ohana and Tierney, 2009) and clustering-based (Li and Liu, 2014)
unsupervised sentiment classification algorithms in PLStream for labelling
data streams without labelled datasets. The experimental evaluation shows that
our proposed solution is able to continuously generate high quality labels
(almost $80\%$ accuracy) to high-speed continuous data streams (almost
$16,000$ tweets/s). Compared to the alternative approaches, PLStream improves
labelling accuracy by more than $20\%$, and achieves more than $50\%$
improvements in terms of both latency and throughput.

In summary, we make the following technical contributions:

  1. (1)

We design a scalable and robust framework (Section 4), PLStream for annotating
unlabelled data streams based on a production-level data stream processing
system, Apache Flink. We open-source our
system111https://github.com/HuilinWu2/PLStream.

  2. (2)

We propose and evaluate several algorithmic and system optimizations in
PLStream including _(i) fast similarity-based labelling_ (Section 4.1), _(ii)
temporal trend detection_ (Section 4.2), and _(iii) tunable model management_
((Section 4.3)) that help PLStream to achieve high performance and high
labelling accuracy.

  3. (3)

To the best of our knowledge, this is also the first empirical study that
ensures a fair comparison among alternative approaches on annotating
unlabelled data streams on modern parallel hardwares (Section 5), as we
eliminate the performance difference caused by different compilers and runtime
by implementing all alternative solutions inside the same framework.

##  2\. Related Work

We discuss related work comes from three key areas: sentiment classification,
large scale annotations and stream processing.

Sentiment classification is an application of natural language processing
(NLP) that analyzes subjective texts with emotions to determine the views,
preferences, and tendencies of a text. Most prior works (Agarwal and Mittal,
2016; Go et al., 2009; Gautam and Yadav, 2014; Haque and et al., 2018;
Ortigosa and et al., 2014) belong to _supervised sentiment learning_ , which
typically requires a _training_ phase that trains a classifier with labelled
training texts followed by an _inference_ phase that identifies the polarity
of a testing text based on the trained classifier. Their primary concern is to
obtain a high accuracy classifier from a large set of collected and labelled
datasets, assuming infinite time and resource budgets. Zimbra et al. (Zimbra
and et al., 2018) reviewed 28 existing techniques of sentiment classification
across five distinctive twitter data sets, and are mostly supervised
approaches. Unsupervised methods, such as lexicon-based approaches (Taboada et
al., 2011; Baccianella and et al., 2010) have been proposed. However, lexicon-
based approaches only deal with novel vocabulary and contextual information in
a limited way as they rely on domain-dependent opinion words and aspect-
dependent opinion words (Gao and et al., 2015). Probabilisitc models, such as
LDA (Lin and He, 2009), target learning from a large set of words from
documents, while we target processing data streams, like tweets, which are
written in a unique conversational style particular to the brevity of the
Twitter medium (i.e., 280 character limit per tweet).

Large scale annotations of unlabelled datasets is not a new demand. Early work
by Pak and Paroubek (Pak and Paroubek, 2010) utilizes emoticons to infer
labels from tweets and is limited with their application scenarios. Closely
related to our work, a recent work (Iosifidis and Ntoutsi, 2017) applies
Multinomia Naive Bayes (MNBs) as the based model to annotate sentiment
datasets. Combined the MNBs model with a confidence threshold, they use the
classification results as new training data to incrementally train the model
offline until the accuracy converges. However, we haven shown in Section 1
that, such an offline approach is not applicable to annotate high-speed data
streams due to the exponentially increasing computational complexity. To the
best of our knowledge, how to ensure high accuracy of polarity labelling of
massive unlabelled data streams such as social media opinions, so far has been
an unsolved problem.

Most existing sentiment analysis techniques have emphasized algorithmic
accuracy. They have not focused on utilizing parallel hardware resources.
Although there are general-purpose data stream processing systems (DSPSs) that
can provide fast stream computations (e.g., Apache Spark streaming (Spa, 2021)
and Apache Flink (Carbone and et al., 2015)), they have not been designed nor
optimized for the tasks of sentiment analysis. This had motivated a
distributed online sentiment analysis system as proposed by Amir et al.
(Rahnama, 2014). Liu et al. (Liu and et al., 2015) have described a standalone
system that supports real-time unsupervised machine learning tasks such as
lexicon-based sentiment analysis. We evaluate the lexicon-based sentiment
classification as an alternative approach in our experiments. Recently,
Ramanath et al. (Ramanath and et al., 2021) have proposed a library called
Lambda Learner, which trains sequential bayesian logistic regression models by
incremental updates in response to mini-batches from data streams. In
contrast, our work strictly focuses on fast polarity labelling of massive data
streams.

##  3\. Preliminary

In this section, we present the preliminary knowledge of basic concepts in
word embedding and the data stream processing systems.

Word Embedding. To analyze unstructured natural language texts, a common step
is to first convert vocabulary into a vector of numeric values. Two classical
word representation algorithms are _one-hot encoding_ and _bag of words_
(Birunda and Devi, 2021). However, one-hot encoding will lead to a dimension
explosion when the vocabulary size becomes larger, while bag of words does not
consider the context information in one sentence. Another advanced word vector
representation technique, _word embedding_ , is to find a mapping or function
to generate an expression in vector space. Representative recent word
embedding tool include _Word2Vec_ (Mikolov et al., 2013), _Glove_ (Pennington
and et al., 2014), _BERT_ (Devlin and et al., 2018) and their variants (Le and
Mikolov, 2014). _Glove_ combines local contextual information with overall
statistics of the corpus, while word embedding with pre-trained model _BERT_
provide richer embedded vectors, where _BERT_ uses deep bidirectional
transformer (Vaswani and et al., 2017) components to build the entire model
with bidirectional contextual information, i.e., each word has different
vector representations in different context. However, the training of _Glove_
and _BERT_ are batch-based, requiring the presence of the entire input
datasets and cannot be applied to learn from on-going input streams. In
contrast, Word2Vec (Mikolov et al., 2013) is able to learn incrementally, and
is thus adopted in our framework.

DSPSs. With the explosion of information, being able to process data streams
efficiently in real time is perceived as a major challenge. In the last
decade, data stream processing systems (DSPSs) have received considerable
attention. Many DSPSs have been proposed, such as Apache Flink (Carbone and et
al., 2015), Apache Storm (Apa, 2021), and Spark Streaming (Spa, 2021). We
leverage Apache Flink for stream processing due to its capability of
expressive, declarative, and efficient data analysis on data streams (Carbone
and et al., 2015), with high-level programming language API support such as
the Python API (pyf, 2021). Nevertheless, our proposed solution is general and
can be applied to other DSPSs with similar characteristics.

##  4\. Incremental Polarity Learning

We propose PLStream, a novel framework that learns and identifies the polarity
of fast ongoing data streams. We summarize the notations used throughout this
paper in Table 1. We define a _tuple_ (denoted as $T$) as a timestamp $t$
associated with a finite number of vocabularies (i.e., $v_{1}\sim v_{n}$). The
_input stream_ is a list of tuples chronologically arriving at the system,
denoted as $S=\\{T_{1},...,T_{N}\\}$, where $N$ goes to infinity. In PLStream,
every input tuple functions as both the _training_ and _testing_ datasets. The
goal of PLStream can be thus defined as follows.

######  Design Goal 1.

Given an input stream $S$, PLStream learns and identifies the polarity of
$T_{i}\in S$ as soon as $T_{i}$ arrives without relying on any offline
training phase with labelled datasets.

Figure 2. System overview. An oval denotes one instance of a stream operator.

An overview of PLStream is depicted in Figure 2. It is implemented on top of
Apache Flink as a chain-topology stream application with three stream
operators. Each operator can be carried by multiple threads to run
concurrently to improve system throughput. It is noteworthy that, PLStream is
continuously updated and can be used to label sentences with even unforeseen
vocabularies. This differs significantly from offline approaches (Agarwal and
Mittal, 2016; Haque and et al., 2018), where the pre-trained model can quickly
become outdated, as both the vocabulary and the polarity model evolve,
requiring periodically re-training with labelled datasets.

First, it continuously batches a tunable number ($b$) of tuples from the input
data stream and extracts useful vocabulary to feed to the subsequent
operators. To reduce the processing workload, information that have nothing to
do with emotions, such as numbers, stopwords, usernames, etc. is filtered out.
For example, in Figure 2, $\\#COVID$, $I$ and $!$ are removed because they are
unrelated to sentiment. It is noteworthy that we retain as much as possible of
the words that constitute the sentence to provide contextual information. This
contradicts to some prior work (Taboada et al., 2011), which only retain
adjectives and adverbs. For example, the noun “criminal” is more likely to
appear in negative texts.

Second, PLStream performs an incremental word embedding based on Word2Vec as
it captures the vector representation of each vocabulary with local
information involving low computational costs compared to alternative
embedding approaches like BERT (Devlin and et al., 2018). It continuously
trains and updates vector representations (VRs) of input tuples and help make
high-accuracy polarity predictions by extracting both _contextual and temporal
information_ from opinionated streams. For example, in Figure 2, weight matrix
$M$ is incrementally trained and each row stands for the VR for one word.

Third, with the learned vector representation of each word, PLStream
classifies the polarity of each input tuple without manual labelling efforts.
For example, in Figure 2, the polarity of four input tuples (i.e., tweets) are
labelled through similarity-based computations. To improve the system
performance and labelling accuracy, PLStream adopts a fast cosine similarity
computation method to estimate the polarity label of the sentences and enhance
it further with a temporal trend detection mechanism, to be covered in Section
4.1 and 4.2, respectively.

During the second step, the learned VRs may be stored locally by each thread
or in Redis for shared access by multiple threads. When VRs are shared among
threads, continuously updating VRs potentially leads to huge overhead,
especially the communication overhead between two systems, i.e., Flink and
Redis. To this end, we further develop several alternative model merging
approaches, detailed in Section 4.3.

Table 1. Notations used in this paper. Parameter value ranges used in our
experiments are also shown.

Name |  Notation |  Description
---|---|---
Tuple |  $T=\\{t,v_{1},v_{2},...,v_{n}\\}$ |  A timestamped ($t$) finite number $n$ of words ($v_{i}$) with a polarity to be identified
Input stream |  $S=\\{T_{1},...,T_{N}\\}$ |  A list of tuples chronologically arriving at the system to be analyzed
Weight Matrix |  $M_{m,d}$, where each element in $M$ is denoted as $m_{i,j}$ |  The vector representation of a list of words, where $m$=number of words, and $d$=number of features
Vector lookup table |  $V:word\rightarrow m_{i}$ |  A lookup table of trained word (key) with corresponding vector representation in $M$ (value)
Window |  $w=1\sim 10$ |  A tunable training window size used by Word2Vec to capture contextual information
Vector Dimension |  $d=10\sim 500$ |  A tunable feature size used by Word2vec training
Batch size |  $b=200\sim 5000$ |  A tunable size of batched tuples being collected before training
Trend Detection Window |  $TDW=200\sim 2000$ |  A tunable window size for polarity trend detection
Model Merging Periodic |  $p=20\sim 90(seconds)$ |  Model merging frequency

Table 2. Reference table used in our experiments Positive |  | dazzling brilliant phenomenal excellent fantastic
---
gripping mesmerizing riveting spectacular cool
awesome thrilling moving exciting love wonderful
best great superb still beautiful
Negative |  | sucks terrible awful unwatchable hideous bad
---
cliche boring stupid slow worst waste unexciting
rubbish tedious unbearable pointless cheesy
frustrated awkward disappointing
Figure 3. Example of similarity-based labelling

###  4.1. Fast Similarity-based Labelling

After getting the learned vector representation of each word, instead of
relying on supervised training progress for polarity identification, we
propose to rely on text similarity (Wang and Dong, 2020; Vijaymeena M.K, 2016;
Navigli and Martelli, 2019) to infer the polarity of input tuples based on a
list of reference words with assumed polarity (Lin and He, 2009) listed in
Table 2. The reference table may become outdated as well. In practice, one may
periodically update the table, which involves minor labor efforts compared to
manually labelling training datasets.

Intuitively, we can compute the cosine similarity between each word of the
input tuple and the reference words as the quantitative relative polarity of
each word. We can then compared the sum of the quantitative polarity of all
words to get the polarity score of an input tuple. However, such a simple
solution leads to significant computational overhead as it involves multiple
cosine similarity computations for every word in each input tuple.

We leverage a simple yet effective _word centroid distance (WCD)_ calculation
approach. For illustration, we show our running example in Figure 3. First, we
obtain the vector representation of the input tuple by calculating the
arithmetic mean of the learned vector representation of all words. As
illustrated in the example, for the input tuple “hate mask”, we calculate the
mean of each dimension ($d=3$) of two words ($S=2$) to obtain a new vector
$V=[v_{1},v_{2},...,v_{d}],where\ v_{i}=\frac{\sum_{j=1}^{S}{m_{i},j}}{S},and\
d=3,\ S=2$. Then, we perform cosine similarity computation (CSC) between $V$
and all reference positive or negative words to obtain the aggregated sum as
$SUM_{pos}$ and $SUM_{neg}$. Finally, we compare $SUM_{pos}$ and $SUM_{neg}$
to infer the polarity of the input tuple entirely. Our extensive experimental
results confirm the superiority of such a simple solution in terms of both
significantly reduced computational effort and higher prediction accuracy
compared to alternative solutions.

Figure 4. Example of trend detection

(a) Local model (b) Global model (c) Hybrid model

Figure 5. Three model management strategies.

###  4.2. Temporal Trend Detection

The conventional way of training word2vec assumes that an entire dataset is
trained at once and does not consider the temporal attribute. It hence
overlooks important information, such as temporally skewed opinions, which we
call _temporal semantic aggregation_. For example, there may be more positive
texts (e.g. live tweets) timely aggregated because of an unbelievable shot in
a football match. In addition, in product-level applications, continuously
capturing trend can also help one keep an eye on real-time sentiment trends
(such as latest reviews of a product). To capture such important information,
we leveraged a _temporal trend detection_ mechanism that helps our learning
algorithm achieve higher prediction accuracy.

We illustrate the whole process of trend detection with an example of two
trend detection windows ($TDWs$), shown in Figure 4. In each iteration, we
identify the polarity of the current window of recent inputs (i.e., 100 tuples
in this example). The trend detection results are used to feedback the
classification of the next window of inputs by tuning the calculation of
cosine similarity. For example, when the current window has a tendency to
accumulate higher positive polarity, the weighted coefficient ($WC$) for
calculating the positive polarity will become larger in the next window. Both
the trend detection window and the weighted coefficient adjustment step size
are tunable, and we will evaluate the effect of TTD in our experiments
(Section 5.4).

###  4.3. Periodical Model Merging

A major limitation of modern DSPSs is the lack of support for _shared mutable
state_ (Zhang and et al., 2020), i.e., a shared memory that multiple
application threads can access and update concurrently. Shared state is
crucial for PLStream as parallel training requires multiple threads to access
and update the trained embedding model. This is not supported by Flink by
default. Commonly, one will leverage a key value store (KVS) to maintain such
shared mutable states for stream processing. Typical examples of such KVS
systems include Redis (red, 2021) and RocksDB (roc, 2021). As our experimental
results will demonstrate, naively relying on a KVS to manage the access and
update of a trained classifier leads to a huge run-time overhead.

We propose three alternative model management strategies in PLStream, as
depicted in Figure 5. 1) _local model:_ we can maintain the model locally at
each thread. This promises the best system performance as it does not involve
costly Redis access. However, it raises the problem of divergence among
learned models at each thread, and it can lead to lower prediction accuracy.
Deploying the system in a distributed environment with multiple nodes would
further exacerbate the problem. 2) _global model:_ we can share the learned
model of word2vec among threads. This ensures a global model in the system
that promises higher prediction accuracy. However, when the input stream is
processed in a parallel fashion, different processing threads may subsequently
access the same model concurrently, which can lead to significant access
contention overheads. And 3) _hybrid model:_ we can periodically merge models
among threads using a customizable frequency that allows users to trade-off
accuracy for performance on demand. In particular, we compute the $mean$ of
two vectors as the new vector representation of the corresponding word. Other
pooling mechanisms (Bommasani and et al., 2020) may be also applied such as
$min$ and $max$.

##  5\. Evaluation

In this section, we show that PLStream manages to achieve much better
prediction accuracy, higher throughput, and lower processing latency compared
to the state-of-the-art through a detailed experimental evaluation.

###  5.1. Experimental Methodology

In this section, we first introduce the benchmark workloads and evaluation
metrics, followed by the discussion of several alternative approaches that we
implemented in the same system to validate the efficiency of our proposal.

####  5.1.1. Benchmark Workloads

We evaluate PLStream on two well-known large-scale datasets (Go et al., 2009;
Zhang and et al., 2015) that are labelled with specific classifying rules.
They cover different use cases and have distinct workload features.
Sentiment140 contains mostly short texts, while Yelp Review contains
relatively long texts. Note that, the original labels of the two datasets are
only used to evaluate the prediction accuracy, and are not used for training
in PLStream.

  1. (1)

The Sentiment140 dataset (Go et al., 2009) is extracted through the Twitter
API from Twitter. It is labelled as positive if the tweet contains emoticons
like ”:)” or negative if tweet contains emoticons like ”:(”. The dataset
contains 1.6 million random-topic tweets, and half are labelled as positive.

  2. (2)

The Yelp Review Polarity dataset (Zhang and et al., 2015) was extracted from
the Yelp Dataset Challenge in 2015. Yelp review polarity consists of 560,000
training samples and 38,000 testing samples. Half are labelled as positive by
considering reviews if they have stars 4 and 3; negative if they have stars 1
or 2. We combine training data (with labels removed) with test data as the
input data stream in our experiment. Unless otherwise stated, we evaluate the
PLStream on Yelp Review Polarity dataset in Section 5.2, 5.4 and 5.5.

####  5.1.2. Evaluation Metrics

We evaluate the system with three performance metrics.

  * •

Throughput. Achieving high throughput is a must to keep up with large volume
data streams. For example, when some major events happen, the opinions on
social media may suddenly explode. We measure the maximum number of input
tuples per second that the system can sustain as the throughput.

  * •

Latency. We measure the 95% latency as the elapsed time from when the input
tuple arrives to when the corresponding classification result is produced. It
is an important indicator to denote the system’s responsiveness.

  * •

Accuracy. We define prediction accuracy as the proportion of correct
predictions (summation of _true positives_ and _true negatives_) over the
total number of tuples processed. To evaluate the prediction accuracy for a
workload with class imbalance, we also use the _F1-score_ , which is a
harmonic mean of the precision and recall.

####  5.1.3. Alternative Approaches

To validate the superiority of our proposal, we implement and evaluate
alternative unsupervised solutions including lexicon-based, probabilistic
model-based (i.e., LDA), and clustering-based ones inside the same system.

Lexicon. Lexicon-based sentiment classification relies heavily on the quality
of lexicons (Taboada et al., 2011). The core idea of this approach is to
classify the polarities through scoring, by looking up the words in the
lexicon and obtaining the corresponding scores. However, there is no lexicon
that can cover all words because a lexicon is costly to build and requires
intensive manual maintenance if it needs to be updated. Text-Blob (Loria,
2018), VADER (Hutto and Gilbert, 2014) and SentiWornet (Baccianella and et
al., 2010) are three well-known lexicons for unsupervised sentiment
classification. Among them, the Vader lexicon contains only 7,500 features,
while Sentiwordnet consists of more than 100,000 words and has a better
ability to cover large-scale data than Vader and other lexicons. Therefore, we
have implemented and evaluated Sentiwordnet in our experiments.

LDA. Latent dirichlet allocation (LDA) (Blei et al., 2003) is widely used for
processing large-scale corpora. LDA determines the topic of each document in
the document set in the form of a probability distribution. Based on LDA, Lin
et al. (Lin and He, 2009) proposed a Joint Sentiment Topic (JST) model that
can infer sentiment by adding an additional sentiment layer between the
document and topic layers. In conjunction with prior information from
lexicons, JST has shown great performance on the IMDB movie reviews dataset
(document-level) with an overall accuracy of 82%, close to the performance of
supervised approaches. In our experiments, we have implemented and evaluated
the performance of JST (JST, 2021).

Clustering. Data clustering aims at grouping data such that objects in the
same group are similar to each other and those in different groups are
dissimilar according to the properties of the instances. In the literature,
some have used clustering for unsupervised sentiment classification (Li and
Liu, 2014; Zimmermann and et al., 2015). We have implemented the clustering-
based approach based on streaming K-means, and set the $K$ parameter to 2
(i.e., either positive or negative) as another baseline method for sentiment
classification in our evaluation.

####  5.1.4. Experimental Setup

We have deployed our experiments on a prototyping cluster with four nodes.
Each node is equipped with 3.60GHz Intel(R) Xeon(R) CPU E5-1650 with 12 cores
and 125GB RAM. The network bandwidth is 1Gb/s. We have evaluated the system
performance up to 48 working threads. We have used Flink v1.12.0 with Java 8
(Oracle JVM), Python 3.7 and Redis v6.0.6. Finally, Table 2 summarizes the
hyperparameters of PLStream and their default values.

###  5.2. Overall Performance Comparison

In this section, we compare algorithms on processing benchmark workloads in
terms of throughput, latency, and accuracy. Each algorithm is experimentally
tuned to its best-performing configuration.

Throughput and Latency. Figure 6a shows the throughput of alternative
approaches on processing two datasets when the system is stable. The LDA-based
method has the highest throughput because it only involves a simple inference
stage in this case. Its performance may drop significantly when the prior
information (i.e., the probability distribution) it requires needs to be re-
calculated. On the contrary, PLStream continuously updates its learned vector
representation from new data, which introduces unavoidable overhead.
Nevertheless, PLStream’s throughtput is still promising, reaching 9877
tuples/second (Sentiment140) and 7182 tuples/second (Yelp Review)
respectively. Lexicon-based and clustering-based approaches achieve poor
performance because the former involves costly lookups from the lexicon, while
the latter involves a costly clustering computation process for every batch of
input stream. Figure 6b shows that PLStream achieves the lowest processing
latency confirming the effectiveness of its fast incremental word embedding
and low overhead similarity-based classification mechanisms. We will
demonstrate the impact of each optimization techniques in detail later in
Section 5.3.

(a) Throughput

(b) $95^{th}$ Latency

Figure 6. Overall performance comparison.

Accuracy. Figure 7 shows the evaluation of labelling accuracy. There are three
key observations. First, we can see that the accuracy of PLStream increases
with more input tuples, since it can incrementally learn from new data, i.e.,
the quality of the vector representation improves gradually. Starting from
$360,000$ and $480,000$ input tuples, the accuracy begins to converge,
reaching an average accuracy of 79.2% and 68.8% for Yelp Review and
Sentiment140, respectively. Second, the LDA-based method, which is sometimes
treated as the representative unsupervised sentiment classification algorithm
(Liu and Zhang, 2012), achieves an accuracy of 68.5% on average for handling
the Yelp Review dataset. Its accuracy is even poorer when processing
Sentiment140 (61.3%), because it relies on a larger corpus for distribution
computations of vocabulary and topics, while Sentiment140 contains mostly
short texts. Third, the clustering-based approach achieves the worst accuracy
on both datasets. It achieves an average accuracy of 52.3% and 51.9% for
Sentiment140 and Yelp Review, respectively. This is because the topics
involved in the input stream are relatively random, and the text lacks
important clustering features required for it to be effective.

(a) Sentiment140

(b) Yelp Review

Figure 7. Overall accuracy comparison.

###  5.3. Impact of Optimizations

A series of system optimization strategies further improve PLStream in terms
of both accuracy and performance. In this section, we study the effectiveness
of the key optimization techniques. Unless otherwise stated, we evaluate the
labelling accuracy of PLStream on Yelp Review Polarity. We tune each
optimization technique to its optimal configuration experimentally.

Impact of Temporal Trend Detection (TTD). We have generated datasets with more
or less positive texts based on the Yelp Review dataset. The generated
datasets have proportions of positive texts ranging from 0-100%. We have
evaluated the performance of various algorithms on these regenerated datasets
in terms of accuracy and $F1-score$. It can be observed from Figure 8 that the
accuracy of PLStream becomes higher with a more skewed polarity distribution.
This is due to the _temporal trend detection_ technique employed by PLStream,
which is able to react well to temporal semantic aggregation and improves the
classification accuracy by up to $5.2\%$. In contrast, the accuracy of
lexicon- and LDA-based algorithms, is clearly lower for negative samples than
for positive samples. This indicates that they misclassified a large number of
samples, resulting in many _false positives_. This is also reaffirmed by the
$F1-score$ shown in Table 3. Specifically, the Lexicon- and LDA-based
algorithms have an $F1-score$ of less than 0.7 for the dataset with low
positive samples, whereas PLStream outperforms all of them with an $F1-score$
greater than 0.8.

Our further investigation found that the trend detection window ($TDW$) has a
significant impact on the effectiveness of the temporal trend detection
technique. Specifically, the system performance with a smaller size $TDW$
(such as 10 or 20) becomes unstable. This is because prediction accuracy may
be low in a small window, leading to low trend detection accuracy, which
continuously affects the classification accuracy of subsequent windows. On the
contrary, an oversized $TDW$ may miss the semantic aggregation. Automatically
choosing the right size of $TDW$ is important and possible (Yao and et al.,
2018), and we leave its tuning as a future work due to its considerable
complexity.

(a)  Figure 8. Impact of temporal trend detection. Table 3. $F_{1}-score$ on different polarity distributions algo. pos. dist. | 0% | 12.5% | 25% | 50% | 75% | 87.5% | 100%
---|---|---|---|---|---|---|---
Lexicon-based | 0.568 | 0.545 | 0.557 | 0.656 | 0.747 | 0.833 | 0.936
LDA-based | 0.639 | 0.661 | 0.6667 | 0.731 | 0.734 | 0.793 | 0.876
Clustering | 0.674 | 0.581 | 0.541 | 0.613 | 0.578 | 0.591 | 0.648
| PLStream
---
without TTD
0.922 | 0.853 | 0.798 | 0.803 | 0.812 | 0.837 | 0.975
PLStream | 0.927 | 0.865 | 0.827 | 0.832 | 0.845 | 0.850 | 0.99

Impact of Model Merging. Figure 9 clearly shows the trade-offs among three
model management strategies in terms of accuracy and throughput. There are
three observations. First, the global model strategy achieves the highest
prediction accuracy, as it generates the most comprehensive vector
representations in the system. However, its throughput is the worst due to the
contended global model updates to Redis. Second, unlike the global model
strategy, there is no communication between threads in the local model
strategy, which leads to significantly higher throughput. However, it achieves
the lowest classification accuracy and slowest convergence speed since the
model on each thread can only learn from the data that is distributed to it.
Third, as expected, the hybrid model achieves reliable accuracy and relatively
good system performance at the same time. Our further evaluation has confirmed
the clear trade-off of varying model merging frequency from 20 to 90 seconds.
Higher frequency leads to better accuracy, but lower performance as frequent
model merging can bring down the system performance. In the following
experiments, we apply a hybrid model merging strategy with a merging frequency
of 30 seconds.

(a) Accuracy

(b) Throughput

Figure 9. Impact of model merging.

###  5.4. System Sensitivity Study

In this section, we present the effectiveness of tuning various
hyperparameters of PLStream in detail.

Impact of Adaptive Batching ($b$). A larger batch size allows PLStream to
learn from more input tuples at a time. Hence, it achieves a faster
convergence speed, lower latency, and higher throughput with amortized
function call cost, as shown in Figure 10. However, there is a clear decrease
in system performance when $b$ exceeds 2000. The reason is that each thread in
PLStream needs to wait for more tuples to start training, resulting in more
system idle time.

(a) Accuracy

(b) $95^{th}$ Latency (c) Throughput

Figure 10. Impact of adaptive batching ($b$).

The Effect of Varying Vector Dimension ($d$). We may tune the number of
features (i.e., vector dimension $d$) used in word2vec training. The common
wisdom is that a large vector dimension ($\sim$500 (Patel and Bhattacharyya,
2017; Mikolov et al., 2013)) helps to obtain a better quality vector
representation. Figure 11 (a) illustrates how the classification accuracy
changes by varying $d$ from 10 to 500. We can see that the converged accuracy
(when the number of processed input tuples=30k) increases from $d=10$ to
$d=20$. This matches the previous findings (Patel and Bhattacharyya, 2017).
However, the classification accuracy does not increase and sometimes even
drops once $d$ exceeds 20. This is due to the well-known _curse of
dimensionality_ (Verleysen and François, 2005) problem raised during our
similarity-based classification. In particular, when the dimensionality is
high, data points (i.e., vector representations in our case) are mostly sparse
and disimilar, which prevents reliable detection of data points of similar
properties. Figure 11 (b) and (c) show that as the vector dimension becomes
higher, PLStream achieves lower throughput and higher processing latency. This
is due to the increasing cost spend in matrix multiplications during
incremental word vectorization, as well as the costly similarity-based
classification.

(a) Accuracy

(b) $95^{th}$ Latency (c) Throughput

Figure 11. Effect of varying vector dimension ($d$).

###  5.5. Solution Scalability

In this section, we evaluate the scalability of PLStream with respect to
hardware resources and workload complexity.

Hardware Resource Scalability. Our next experiment shows that PLStream scales
effectively , when we increase the number of CPU cores. In this experiment,
PLStream is configured with hybrid model strategy and LRU-based model pruning
technique. Figure 12 (a) shows that as the number of threads increases,
PLStream needs to process more tuples to achieve accuracy convergence. This is
expected, as each thread learns less with increasing number of threads. In
contrast, PLStream’s processing and incremental learning at different threads
are mostly independent with little contention among others. In particular, we
can see from Figure 12(b) that the processing latency of PLStream does not
vary with the different number of cores in one node. The processing latency
increases marginally when PLStream is deployed on multiple nodes due to the
additional network communication overhead. Figure 12 (c) further shows that
the throughput of PLStream improves significantly when increasing the number
of cores from 2 to 12 cores (one node), up to 48 cores (four nodes),
reaffirming its good scalability. Nevertheless, further exploration is
required to improve the system towards linearly scale-up.

(a) Accuracy (b) $95^{th}$ Latency (c) Throughput

Figure 12. Effect of varying number of CPU cores.

Workload Complexity Scalability. Data streams such as reviews have longer
text, whereas tweets, for example, are dominated by short text. We now examine
our annontating effect in terms of text length. We selectively pick texts of
varying length from the two benchmark workloads to regenerate the input stream
for this experiment. Figure 13 shows the converged accuracy of each algorithm
on the data set of different text lengths. We have repeated each experiment
three times and report the average result. We have four major observations.
First, it can be observed that the accuracy of PLStream always outperforms
other algorithms. Interestingly, in the dataset with a text length of less
than 30, the accuracy of all algorithms is poorer. This is because, unlike
paragraphs or documents, short texts are more ambiguous: they do not contain
enough contextual information, which poses a great challenge for
classification (Chen and et al., 2019). Second, the accuracy of the LDA-based
algorithm improves with increased text length.. This is expected, as the LDA-
based approach is designed for document-level sentiment analysis (Lin and He,
2009). Third, the accuracy of the lexicon-based algorithm also increases
initially due to the increased contextual information in the input workloads.
However, its accuracy decreases when the text length exceeds 300. This is due
to the increasing noise interfering with its scoring process. For example, one
input tuple may contain multiple opinions. Fourth, in contrast, varying text
length shows no impact on the clustering-based approach. This is again due to
the lack of important clustering features in our benchmark workloads.

Figure 13. Accuracy on different text length.

##  6\. Conclusion

We have presented PLStream, a new framework designed for annotating fast
ongoing _unlabelled_ data streams at _scale_ on modern parallel machines.
PLStream encompasses several algorithmic and system architectural innovations,
that reduce workloads, enhance learned embedding models incrementally, and
control model management overhead. Based on extensive experiments with two
real-world workloads, we demonstrate PLStream’s high labelling accuracy,
system throughput, and low processing latency. In practice, users can further
use a confidence threshold to control the labelling of the vast majority of
the data with high confident by PLStream and leave the rest to human experts.
Labelled datasets can be immediately consumed by subsequent supervised
learning tasks and reduce the development cost of sentiment analysis services.
In the future, we plan to further exploiting “auto-ML” techniques to identify
the optimal value of the various tuning knobs of PLStream.

## References

  * (1)
  * JST (2021) Last accessed by 11 November 2021.  _Joint sentiment topic code,https://github.com/blhoy/JST_model-on-MR_.
  * Apa (2021) Last accessed by 14 August 2021.  _Apache Storm,https://storm.apache.org/_.
  * Spa (2021) Last accessed by 14 August 2021.  _Spark Streaming,https://spark.apache.org/streaming/_.
  * pyf (2021) Last accessed by 16 September 2021.  _Pyflink overview,https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/python/overview/_.
  * roc (2021) Last accessed by 24 August 2021.  _RocksDB key value store,https://rocksdb.org_.
  * red (2021) Last accessed by 24 June 2021.  _Redis key value store,https://redis.io/topics/introduction_.
  * Agarwal and Mittal (2016) B. Agarwal and N. Mittal. 2016.  _Machine Learning Approach for Sentiment Analysis_.  Springer International Publishing. 21–45 pages.
  * Baccianella and et al. (2010) Stefano Baccianella and et al. 2010.  Sentiwordnet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining.. In _Lrec_ , Vol. 10. 2200–2204.
  * Birunda and Devi (2021) S Selva Birunda and R Kanniga Devi. 2021.  A Review on Word Embedding Techniques for Text Classification.  In _Innovative Data Communication Technologies and Application_. Springer, 267–281.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003\.  Latent dirichlet allocation.  _the Journal of machine Learning research_ 3 (2003), 993–1022.
  * Bommasani and et al. (2020) Rishi Bommasani and et al. 2020.  Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings. In _ACL’20_.
  * Capuano and et al. (2021) Nicola Capuano and et al. 2021.  Sentiment analysis for customer relationship management: an incremental learning approach.  _Applied Intelligence_ (01 Jun 2021).
  * Carbone and et al. (2015) Paris Carbone and et al. 2015.  Apache flink: Stream and batch processing in a single engine.  _Bulletin of the IEEE Computer Society Technical Committee on Data Engineering_ 36, 4 (2015).
  * Chen and et al. (2019) Jindong Chen and et al. 2019.  Deep Short Text Classification with Knowledge Powered Attention.  _AAAI’19_ (Jul. 2019).
  * Devlin and et al. (2018) Jacob Devlin and et al. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Gao and et al. (2015) Yang Gao and et al. 2015\.  Pattern-based Topics for Document Modelling in Information Filtering.  _TKDE’15_ (2015).
  * Gautam and Yadav (2014) Geetika Gautam and Divakar Yadav. 2014.  Sentiment analysis of twitter data using machine learning approaches and semantic analysis. In _IC3’14_.
  * Go et al. (2009) Alec Go, Richa Bhayani, and Lei Huang. 2009.  Twitter sentiment classification using distant supervision.  _CS224N project report, Stanford_ 1, 12 (2009), 2009\.
  * Haque and et al. (2018) Tanjim Ul Haque and et al. 2018.  Sentiment analysis on large scale Amazon product reviews. In _ICIRD_. IEEE, 1–6.
  * Hutto and Gilbert (2014) Clayton Hutto and Eric Gilbert. 2014.  Vader: A parsimonious rule-based model for sentiment analysis of social media text. In _AAAI’14_ , Vol. 8.
  * Iosifidis and Ntoutsi (2017) Vasileios Iosifidis and Eirini Ntoutsi. 2017.  Large scale sentiment learning with limited labels. In _KDD’17_. 1823–1832.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed Representations of Sentences and Documents. In _ICML’14_.
  * Li and Liu (2014) Gang Li and Fei Liu. 2014\.  Sentiment analysis based on clustering: a framework in improving accuracy and recognizing neutral opinions.  _Applied intelligence_ 40, 3 (2014), 441–452.
  * Lin and He (2009) Chenghua Lin and Yulan He. 2009.  Joint sentiment/topic model for sentiment analysis. In _CIKM’09_. 375–384.
  * Liu and Zhang (2012) Bing Liu and Lei Zhang. 2012.  A survey of opinion mining and sentiment analysis.  In _Mining text data_. Springer, 415–463.
  * Liu and et al. (2015) Qingyuan Liu and et al. 2015.  FLORIN - A System to Support (Near) Real-Time Applications on User Generated Content on Daily News.  _Proc. VLDB Endow._ 8 (2015), 1944–1947.
  * Loria (2018) Steven Loria. 2018\.  textblob Documentation.  _Release 0.15_ 2 (2018), 269.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013\.  Efficient Estimation of Word Representations in Vector Space. In _ICLR’13_.
  * Navigli and Martelli (2019) Roberto Navigli and Federico Martelli. 2019.  An overview of word and sense similarity.  _Natural Language Engineering_ 25, 6 (2019), 693–714.  https://doi.org/10.1017/S1351324919000305
  * Ohana and Tierney (2009) Bruno Ohana and Brendan Tierney. 2009.  Sentiment classification of reviews using SentiWordNet.  _Proceedings of IT &T_ (2009).
  * Ortigosa and et al. (2014) Alvaro Ortigosa and et al. 2014.  Sentiment analysis in Facebook and its application to e-learning.  _Computers in human behavior_ 31 (2014), 527–541.
  * Pak and Paroubek (2010) Alexander Pak and Patrick Paroubek. 2010.  Twitter as a Corpus for Sentiment Analysis and Opinion Mining. In _LREC’10_.
  * Patel and Bhattacharyya (2017) Kevin Patel and Pushpak Bhattacharyya. 2017.  Towards Lower Bounds on Number of Dimensions for Word Embeddings. In _IJCNLP’17_.
  * Pennington and et al. (2014) J. Pennington and et al. 2014.  Glove: Global Vectors for Word Representation. In _EMNLP’14_.
  * Rahnama (2014) Amir Hossein Akhavan Rahnama. 2014\.  Distributed real-time sentiment analysis for big data social streams. In _CoDIT’14_. IEEE, 789–794.
  * Ramanath and et al. (2021) Rohan Ramanath and et al. 2021.  Lambda Learner: Fast Incremental Learning on Data Streams. In _KDD’21_.
  * Taboada et al. (2011) M. Taboada, J. Brooke, M. Tofiloski, K. D. Voll, and M. Stede. 2011.  Lexicon-Based Methods for Sentiment Analysis.  _Computational Linguistics_ 37, 2 (2011), 267–307.
  * Vaswani and et al. (2017) Ashish Vaswani and et al. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Verleysen and François (2005) Michel Verleysen and Damien François. 2005.  The Curse of Dimensionality in Data Mining and Time Series Prediction. In _Computational Intelligence and Bioinspired Systems_ , Joan Cabestany, Alberto Prieto, and Francisco Sandoval (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 758–770.
  * Vijaymeena M.K (2016) Kavitha K Vijaymeena M.K. 2016\.  A Survey on Similarity Measures in Text Mining.  _Machine Learning and Applications: An International Journal_ 3, 1 (2016), 19–28.  https://doi.org/10.5121/mlaij.2016.3103
  * Wang and Dong (2020) Jiapeng Wang and Yihong Dong. 2020.  Measurement of Text Similarity: A Survey.  _Information_ 11, 9 (2020).  https://doi.org/10.3390/info11090421
  * Yao and et al. (2018) Q. Yao and et al. 2018\.  Taking Human out of Learning Applications: A Survey on Automated Machine Learning.  (2018).
  * Zaharia and et al. (2012) Zaharia and et al. 2012\.  Resilient Distributed Datasets: A $\\{$Fault-Tolerant$\\}$ Abstraction for $\\{$In-Memory$\\}$ Cluster Computing. In _9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12)_. 15–28.
  * Zhang and et al. (2020) Shuhao Zhang and et al. 2020.  Towards concurrent stateful stream processing on multicore processors. In _ICDE’20_.
  * Zhang and et al. (2015) Xiang Zhang and et al. 2015\.  Character-Level Convolutional Networks for Text Classification .  _arXiv:1509.01626 [cs]_ (Sept. 2015).  arXiv:1509.01626 [cs]
  * Zimbra and et al. (2018) David Zimbra and et al. 2018.  The state-of-the-art in twitter sentiment analysis: A review and benchmark evaluation.  _TMIS’18_ (April 2018).
  * Zimmermann and et al. (2015) Max Zimmermann and et al. 2015.  Discovering and Monitoring Product Features and the Opinions on them with OPINSTREAM.  _Neurocomput._ Volume 150, Part A (2015), Pages 1–346.  accepted 4/2014.