Aspect extraction in online product reviews is a key task in sentiment analysis and opinion mining. Training supervised neural networks for aspect extraction is not possible when ground truth aspect labels are not available, while the unsupervised neural topic models fail to capture the particular aspects of interest. In this work, we propose a weakly supervised approach for training neural networks for aspect extraction in cases where only a small set of seed words, i.e., keywords that describe an aspect, are available. Our main contributions are as follows. First, we show that current weakly supervised networks fail to leverage the predictive power of the available seed words by comparing them to a simple bag-of-words classifier.   Second, we propose a distillation approach for aspect extraction where the seed words are considered by the bag-of-words classifier (teacher) and distilled to the parameters of a neural network (student). Third, we show that regularization encourages the student to consider non-seed words for classification and, as a result, the student outperforms the teacher, which only considers the seed words. Finally, we empirically show that our proposed distillation approach outperforms (by up to 34.4% in F1 score) previous weakly supervised approaches for aspect extraction in six domains of Amazon product reviews. Aspect extraction is a key task in sentiment analysis, opinion mining, and summarization BID11 BID8 Pontiki et al., 2016; BID0 . Here, we focus on aspect extraction in online product reviews, where the goal is to identify which features (e.g., price, quality, look) of a product of interest are discussed in individual segments (e.g., sentences) of the product's reviews.Recently, rule-based or traditional supervised learning approaches for aspect extraction have been outperformed by deep neural networks BID16 BID22 , while unsupervised probabilistic topic models such as Latent Dirichlet Allocation (LDA) BID2 have been shown to produce less coherent topics than neural topic models BID9 BID5 BID17 : when a large amount of training data is available, deep neural networks learn better representations of text than previous approaches.In this work, we consider the problem of classifying individual segments of online product reviews to predefined aspect classes when ground truth aspect labels are not available. Indeed, both sellers and customers are interested in particular aspects (e.g., price) of a product while online product reviews do not usually come with aspect labels. Also, big retail stores like Amazon sell millions of different products and thus it is infeasible to obtain manual aspect annotations for each product domain. Unfortunately, fully supervised neural approaches cannot be applied under this setting, where no labels are available during training. Moreover, the unsupervised neural topic models do not explicitly model the aspects of interest, so substantial human effort is required for mapping the learned topics to the aspects of interest.Here, we investigate whether neural networks can be effectively trained under this challenging setting using only weak supervision in the form of a small set of seed words, i.e., descriptive keywords for each aspect. For example, words like "price," "expensive," "cheap," and "money" are represen-tative of the "Price" aspect. While a traditional aspect label is only associated with a single review, a small number of seed words can implicitly provide (noisy) aspect supervision for many reviews.Training neural networks using seed words only is a challenging task. Indeed, we show that current weakly supervised networks fail to leverage the predictive power of the seed words. To address the shortcomings of previous approaches, we propose a more effective approach to "distill" the seed words in the neural network parameters. First, we present necessary background for our work.2 BACKGROUND: NEURAL NETWORKS FOR ASPECT EXTRACTION Consider a segment s = (x 1 , x 2 , . . . , x N ) composed of N words. Our goal is to classify s to K aspects of interest {α 1 , . . . , α K }, including the "General" aspect α GEN . In particular, we focus on learning a fixed-size vector representation h = EMB(s) ∈ R l and using h to predict a probability distribution p = p 1 , . . . , p K over the K aspect classes of interest: p = CLF(h).The state-of-the-art approaches for segment embedding use word embeddings: each word x j of s indexes a row of a word embedding matrix W b ∈ R V ×d to get a vector representation w xj ∈ R d , where V is the size of a predefined vocabulary and d is the dimensionality of the word embeddings. The set of word embeddings {w x1 , ..., w x N } is then transformed to a vector h using a vector composition function such as the unweighted/weighted average of word embeddings BID20 BID1 , Recurrent Neural Networks (RNNs) BID19 BID21 , and Convolutional Neural Networks (CNNs) BID10 BID7 . During classification (CLF), h is fed to a neural network followed by the softmax function to get p 1 , . . . , p K .Supervised approaches use ground-truth aspect labels at the segment level to jointly learn the EMB and CLF function parameters. However, aspect labels are not available in our case. Unsupervised neural topic models avoid the requirement of aspect labels via autoencoding BID9 BID5 . In their Aspect Based Autoencoder (ABAE), BID5 reconstruct an embedding h for s as a convex combination of K aspect embeddings: DISPLAYFORM0 is the k-th row of the aspect embedding matrix A ∈ R K×d . The aspect embeddings A (as well as the EMB and CLF function parameters) are learned by minimizing the segment reconstruction error. 1 Unfortunately, unsupervised approaches like ABAE do not utilize information about the K aspects of interest and thus the probabilities p 1 , . . . , p K cannot be used directly 2 for our downstream application. To address this issue, BID0 proposed a weakly supervised extension of ABAE. Their model, named Multi-seed Aspect Extractor, or MATE, learns more informative aspect representations by also considering a distinct set of seed words G k = {g k1 , . . . , g kL } for each aspect. In particular, MATE initializes the k-th row of the aspect embedding matrix A to the weighted 3 average of the corresponding seed word embeddings: DISPLAYFORM1 As initializing the aspect embeddings to particular values does not guarantee that the aspect embeddings after training will still correspond to the aspects of interest, BID0 fix (but do not fine tune) the aspect embeddings A and the word embeddings W b throughout training. However, as we will show next, MATE fails to effectively leverage the predictive power of seed words.
We effectively leverage a few keywords as weak supervision for training neural networks for aspect extraction. Discusses a variant of knowledge distillation which uses a "teacher" based on a bag-of-words classifier with seed words and a "student" which is an embedding-based neural network.