Cost and Benefit of Using WordNet Senses for Sentiment Analysis
Typically, accuracy used to represent the performance of an NLP system. However, accuracy attainment a function of investment in annotation. Typically, the more the amount and sophistication of annotation, higher the accuracy. However, a moot question is the accuracy improvement commensurate with the cost incurred in We present an economic model to assess the marginal benefit accruing from increase in cost of annotation. In particular, as a case in point we have chosen the sentiment analysis (SA) problem. In SA, documents normally are polarity classified by running them through classifiers trained on document vectors constructed from lexeme features, i.e., words. If, however, instead of words, one uses word senses (synset ids in wordnets) as features, the accuracy improves dramatically. But this improvement significant enough to justify the cost of annotation? This question, to the best of our knowledge, has not been investigated with the seriousness it deserves. We perform a cost benefit study based on a vendor-machine model. By setting up a cost price, selling price and profit scenario, we show that although extra cost incurred in sense annotation, the profit margin high, justifying the cost.