Topic-based sentiment analysis for Chinese microblog aims to identify the user attitude on specified topics. In this paper, we propose a joint model by incorporating Support Vector Machines (SVM) and deep neural network to improve the performance of sentiment analysis. Firstly, a SVM Classifier is constructed using N-gram, N-POS and sentiment lexicons features. Meanwhile, a convolutional neural network is applied to learn paragraph representation features as the input of another SVM classifier. The classification results outputted by these two classifiers are merged as the final classification results. The evaluations on the SIGHAN-8 Topic-based Chinese microblog sentiment analysis task show that our proposed approach achieves the second rank on micro average F1 and the fourth rank on macro average F1 among a total of 13 submitted systems.
Introduction With the development of the Internet, microblog has become a popular user-generated content platform where users share the newest events or their personal feelings with each other. Topic-based microblogs are the most common interactive way for users to share their opinions towards a specified topic. To identify the opinions of users, sentiment analysis techniques are investigated to classify texts into different categorizations according to their sentiment polarities. Most existing sentiment classification techniques are based on machine learning algorithms, such as Support Vector Machine, Naïve Bayes and Maximum Entropy. The machine learning based approach uses feature vectors as the input of classification to predict the classification results. Thus, feature engineering, a method for extracting effective features from texts, plays an important role. Some commonly used features in sentiment classification are unigram, bigram and sentiment words. However, these features cannot work well for cross-domain sentiment classification because of the lack of domain knowledge. Danushka Bollegala et al. (2011) used multiple sources to construct a sentiment sensitive thesaurus to overcome the lack of domain knowledge. New sentiment words expansion is another kind of approach to improve the performance of sentiment analysis. Strfano Baccianella et al. (2010) constructed SentiWord-Net by extending WordNet with sentiment information. It is now widely used in sentiment classification for English. As for Chinese sentiment analysis, Minlie Huang et al. (2014) proposed a new word detection method by mining the frequent sentiment word patterns. This method may discover new sentiment words from a large scale of unlabeled texts. With the rapid development of pre-trained word embedding and deep neural networks, a new way to represent texts and features is devloped. Mikolov et al. (2013) showed that word embedding represents words with meaningful syntactic and semantic information effectively. Recursive neural network proposed by Socher et al. (2011a; 2011b; is shown efficient to construct sentence representations based on the word embedding. Convolutional neural networks (CNN), another deep learn model which achieved success in image recognition field, was applied to nature language processing with word embed-dings. Yoon Kim (2014) used CNN with pretrained word embedding to achieve state-ofthe-art performances on some sentence classification tasks, including sentiment classification. Siwei Lai et al. (2015) incorporated global information in a recurrent convolutional neural network. It obtained further improvements comparing to other deep learning models. In this paper, we propose a joint model which incorporates traditional machine learning based method (SVM) and deep learning model. Two different classifiers are developed. One is a word feature based SVM classifier which uses word unigram, bigram and sentiment words as features. Another one is a CNN-based SVM classifier which takes paragraph representations features learned by CNN as input features. The classification results of these two classifiers are integrated to generate the final classification results. The evaluations on the SIGHAN-8 Topic-based Chinese microblog sentiment analysis task show that our proposed approach achieves the second rank on micro average F1 and the fourth rank on macro average F1 among a total of 13 submitted systems. Furthermore, the joint classifier strategy brings further performance improvement on individual classifiers. The rest of this paper is organized as follows. Section 2 presents the design and implementation of our proposed joint model. Section 3 gives the evaluation results and discussions. Finally, Section 4 gives the conclusion and future research directions. Our Approach The SIGHAN8 topic-based Chinese polarity classification task aims to is to classify Chinese microblog into three topic-related sentiment classes, namely neutral, positive and negative. This task may be generally regarded as a three-category classification problem. The SVM classifier which has been shown effective to document classification is adopted as the core classifier. Here, two different feature representation models, namely word-based vector space model and CNN-based composition representation, are adopted to generate the classification features for two classifiers, respectively. The classification outputs of two clas-sifiers are integrated to generate the final output. Data preprocessing Chinese microblog text is obviously different from formal text. Many microblogs have noises, including nickname, hashtag, repost or reply symbols, and URL. Therefore, before the feature representation and extraction, preprocessing is performed to filter out noise text in the microblogs. Meanwhile, the advertising text and topic-irrelevant microblog are identified as neutral text. Especially, this task is designed to identify the topic-relevant sentiments. Therefore, the information coming from the reply, repost and sharing parts should be filtered out to avoid their influences to the sentiment analysis of the microblog author. Generally speaking, such filtering is based on rules. The table 1 shows the example data preprocessing rules with illustrations. Table 2 shows the rules for identifying the advertisement and topic-irrelevant microblogs. The identified microblogs are labeled as neutral for topic-based sentiment classification. Word feature based classifier The word feature based classifier is designed based on the vector model. Firstly, the new sentiment words from unlabeled sentences data are recognized to expand the sentiment lexicon. The classification features are extracted from the labeled training data and sentiment lexicon resources. In order to alleviate the influences of unbalanced training data, SMOTE, which is an oversampling algorithm, is applied to training data before classifier training. Finally, a SVM classifier is trained on the balanced data. The framework of word feature based classifier is shown in Figure 1. Feature selection Unigram, Bigram, Uni-Part-of-Speech and Bi-Part-of-Speech features are selected as the basic features. CHI-test based feature selection is applied to obtain the top 20000 features. To improve the performance of sentiment classification, additional features based on lexicons including sentiment word lexicons, negation word lexicons, and adverb word lexicons, are incorporated. Rules Raw Text Processed Text Sharing news with 好看？吗？//【Galaxy S6：三星证明自 好看？吗？ personal comments 己能做出好看的手机】http: //t.cn/ RwHRsIb(分享自 @ 今日头条) Removing HashTag # 三星 Galaxy S6# 三星 GALAXY S6 三星 GALAXY S6， ，挺中意 [酷][酷] [位置] 芒砀路 挺中意 [酷][酷] Removing URL 699 欧元起传三星 Galaxy S6/S6 Edge 售 699 欧元起传三星 Galaxy 价获证实(分享自 @ 新浪科技) S6/S6 Edge 售价获证实 http://t.cn/RwTo3on (分享自 @ 新浪科技) Removing nickname 玻璃取代塑料，更美 Galaxy S6 的 5 大 http://t.cn/RwHY6Az 妥协 http://t.cn/RwHY6Az 罗永浩我去 罗永浩我去小米和三星这 小米和三星这是要闹哪样， ， ，老罗。 。不 是要闹哪样， ， ，老罗。 。 能忍啊， ， ， ， ，@ 锤子科技营销帐号 @ 罗 不能忍啊， ， ， ， ， 永浩 Removing 【视频：三星 S6 对比苹果 iPhone6 【视频：三星 S6 对比苹果 information sources MWC2015 @youtube 科技 】 iPhone6 MWC2015 http://t.cn/RwHQzJ8(来自于优酷安 @youtube 科技 】 卓客户端) http://t.cn/RwHQzJ8  Rules Type Including many different Advertisement topic ("#...#") tag. Including many words Advertisement like "微商", "商机", "想赚钱","面膜". No actual content Topic-irrelevant Table 2 : Microblog text matching rules. By analyzing the expressions of the microblog text in training data, some special expression features in microblog text are identified. For example, the continuous punctuations are always used to express a strong feeling and thus, the microblog with continuous punctuations tends to be subjective. Another adopted feature for microblog text is the use of emoticons. Sentiment lexicon expansion In microblogs, abundant new or informal sentiment words are widely used. Normally, these new sentiment words are short but meaningful for expressing a strong feeling. These new sentiment words play an important role in Chinese microblog sentiment classification. Therefore, sentiment word identification is performed to recognize new sentiment words as the supplement of sentiment lexicon. Twenty million microblog text collected from Sina Weibo Platform are used in new sentiment word detection. Considering that new words normally cannot be correctly segmented by the existing segmentor, identifying new words from preliminary segmentation results together with their POS tags is a feasible method. Here, potential components for new words are limited to the segmentation tokens shorter than three. Using word frequency, mutual information and context entropy as the evaluation indicators for words, the most possible new word candidates are obtained. With the help of word embedding construction model, each word in the corpus can be represented as a low dimension vector together with its context information. Hence, the distances between the new words and the existed sentiment words corresponding to difference sentiment polarity are estimated. The new words are then classified into one of the three polarity classes by following voting mechanism. Classification Two steps are performed to determine the topic-relevant sentiment for input microblogs. The first step is to distinguish topic relevant messages from topic irrelevant messages. Sentiment classification is then applied to topic relevant messages in the second step. Topic relevant words generated by clustering analysis are employed as distinguishable features to filter out topic irrelevant microblogs because normally the topic irrelevant microblogs have few intersections with topic relevant words. Some advertisement posts consisting of several hot topic hash tags are also filtered out by considering the number of hash tag types in the microblog. The provided labeled dataset is used to train the SVM classifier with linear kernel. A new challenge is that the provided training set is imbalanced. There are about 3973 neutral microblogs, while the numbers of positive and negative microblogs are 394 and 538, respectively. In order to reduce the influences of imbalanced training dataset, the SMOTE algorithm (Chawla et al., 2002) is applied to oversampling the samples on minority class. Oversampling ratio is set to 10 and 7.4 for positive class and negative class, respectively. In this way, the training dataset becomes balanced. Another classifier is CNN-based SVM classifier. The classifier framework is shown in Figure 2 . Firstly, continuous bog of word (CBOW) model (Mikolov et al., 2013 ) is used to learn word embeddings from Chinese microblog text. A deep convolutional neural networks (CNN) model is applied to learn distributed paragraph representation features for Chinese microblog training and testing data. Finally, the distributed paragraph representation features are used in SVM classifier to learn the probability distribution over sentiment labels. CNN-based SVM classifier Word embedding construction Word embedding, wherein words are projected from a sparse, 1-of-V encoding (here V is the vocabulary size) onto a lower dimensional vector space via a hidden layer, are essentially feature extractors that encode semantic features of words in their dimensions. Mikolov et al. (2013) introduced CBOW model to learn vector representations which captures a large number of syntactic and semantic word relationships from unstructured text data. The main idea of this model is to find word representations which use the surrounding words in a sentence or a document to predict current word. In this study, we train the CBOW model by using 16GB Chinese microblog text. Finally, we obtain 200-dimension word embeddings for Chinese microblog text. CNN-based SVM classifier In the CNN-based SVM classifier, the input is a matrix which is composed of the word embeddings of microblogs. There are windows with the lengths of three, four and five words, respectively. A convolution operation involves three filters which are applied to these windows to produce new features. After convolution operation, a max-over-time pooling operation is applied over these features. The maximum value is taken as the feature corresponding to this particular filter. The idea is to capture the most important feature which has the largest value. Since one feature is extracted from one filter, the model uses multiple filters (with varying window sizes) to obtain multiple features. These features constitute the distributed paragraph feature representation. In the last step, a SVM classifier is applied on these distributed paragraph representation features to obtain the probability distributions over labels (positive, negative, and neutral). A set of merging rules is designed to incorporate the individual classification results of the two classifiers for generating the final result. If the two classification outputs are the same, naturally, the final output is the same. If the two classification outputs are different, the final result is determined from the merge rules shown in Table 3 . Simply speaking, if any of two classifiers output neutral category, the final output is neutral. If two classifiers outputs positive and negative, respectively, the final output is the result of CNN-based clas-sifier. Such a classification outputs merging strategy is based on the statistical analysis on the individual classifier performances on training dataset. Outputs Merging Experimental results and analysis Data set In the SIGHAN-8 Chinese sentiment analysis bakeoff dataset, 4905 topic-based Chinese microblog are provided as training data which consists of 394 positive, 538 negative and 3973 neutral microblogs corresponding to 5 topics, namely "央行降息", "油价", "日本马桶", "三星 S6"and "雾霾". In the testing data, there are 19,469 microblogs corresponding to 20 topic, such as "12306 验证码", "中国政 府也门撤侨", "何以笙箫默", "刘翔退役". Metrics Precision, recall and F1-value are used as the evaluation metrics, as shown below: P recision = SystemCorrect SystemOutput (1) Recall = SystemCorrect HumanLabeled (2) F 1 = 2 × P recision × Recall P recision + Recall (3) Where System.Output refers to the total number of the submitted results, System.Correct refers to the number of correctly classified results in the submitted results, Human.Labeled refers to the total number of manually labeled results in the Gold Standard. The evaluation metrics corresponding to positive, negative and overall are estimated, respectively. The corresponding microaverage and macro-average performances are then estimated. The micro-average estimates the average performance of the three evaluation metrics over the entire dataset. The macro-average estimates the average performances of the evaluation metrics on positive, negative and neutral, respectively. Experimental results and analysis There are two subtasks in SIGHAN-8 topicbased Chinese microblog polarity classification Table 6 : Performances by different classifiers in unrestricted resource subtask. task: restricted resource and unrestricted resource subtasks. Table 4 gives the performances in restricted resource subtask. The first column lists the name of participants who achieves higher macro average F1 values while out system is named as HLT_HITSZ. It is observed that our proposed approach achieves better performance on negative and positive categories, but obviously lower performance on neutral category. The good performance on the recall of minority classes showed the effectiveness of our consideration on imbalanced dataset training. The achieved performances in the unrestricted resource subtask are listed in Table 5 . Our system achieves about 3% of performance improvement on each category, respectively. It shows the contributions of extra training corpus and merging rules. In order to validate the effectiveness of merging rules, the performances of Classifier 1 and Classifier 2 are evaluated, individually. The achieved performances are given in Table 6. It is observed that generally speaking, Classifier 1 achieves a higher classification precision because many features are coming from manually compiled sentiment-related lexicons. However, these features are limited to training data so that Classifier 1 achieved a lower recall. On the contrary, Classifier 2 may learn the representation features automatically from training data which is better for generalization. Thus, a good recall is achieved. Meanwhile, the achieved performances show that our joint model obtains better performances compared to two individual classifiers which indicate the effectiveness of our proposed joint classification strategy. Conclusion In this work, we propose a joint model for sentiment topic analysis on Chinese microblog messages. A word feature based SVM classifier and a SVM classifier using CNN-based paragraph representation features are developed, respectively. To overcome the limitation of each classifier, their classification outputs are merged to generate the final output while the merging rules are based on statistical analy-sis on the performances on training dataset. Experimental results show that our proposed joint method achieves better sentiment classification performance over individual classifiers which show the effectiveness of the joint classifier strategy. In future, we intend to study the way to distinguish the subjective messages from objective messages for further improving the sentiment classification performance.
1 2 2.1 2.2 2.2.1 2.2.2 2.2.3 2.3.1 2.3.2 3.1 3.2 3.3 4
Introduction Our Approach Data preprocessing Word feature based classifier Feature selection Sentiment lexicon expansion Classification Word embedding construction CNN-based SVM classifier Data set Metrics Experimental results and analysis Conclusion
Experiments
Training data: 4905 microblogs (394 positive, 538 negative and
Testing data: 19469 microblogs, 20 topics
P r ecision SystemCorrect
F Pr ecision Re call
Performances in unrestricted resource subtask
Team Name Precision Recall F1 Precision Recall F1 Precision Recall F1
Performances by different classifiers in unrestricted resource subtask
Approach Precision Recall F1 Precision Recall F1 Precision Recall F1
Training data: 4905 microblogs (394 positive, 538 negative and
Testing data: 19469 microblogs, 20 topics
P r ecision SystemCorrect
F Pr ecision Re call
Performances in unrestricted resource subtask
Team Name Precision Recall F1 Precision Recall F1 Precision Recall F1
Performances by different classifiers in unrestricted resource subtask
Approach Precision Recall F1 Precision Recall F1 Precision Recall F1