Take the 2-minute tour ×

I have a lucene index with a lot of text data, each item has a description, I want to extract the more common words from the description and generate tags to classify each item based on the description, is there a lucene.net library for doing this or any other library for text classification?

share|improve this question
Not sure if this will help but have a look at this MSDN Link msdn.microsoft.com/en-us/library/dd885640.aspx –  DJ KRAZE Jan 24 '12 at 17:00
This article might send you in the right direction. –  Sean U Jan 24 '12 at 17:20
thanks, but I'm using .NET mahout is only available in java –  ryudice Jan 24 '12 at 20:04
add comment

2 Answers

up vote 2 down vote accepted

No, lucene.net can make search, index, text normalization, "find more like this" funtionalty, but not a text classification.

What to suggest to you depends from your requirements. So, maybe more description needed. But, generally, easiest way try to use external services. All external services have REST API, and it's very easy to interact with it using C#.

From external services:

Also there good Java SDK like Mahout. As I remember interactions with Mahout could be also done like with service, so integration with it is not a problem at all.

I had similar "auto tagging" task using c#, and I've used for that Open Calais. It's free to make 50,000 transactions per day. It was enough for me. Also uClassify has good pricing, as example "Indie" license 99$ per year.

But maybe external services and Mahout is not your way. Than take a look at DBpedia project and RDF. And the last, you can use some implementations of Naive Bayes algorithm, at least. It's easy, and all will be under your control.

share|improve this answer
add comment

This is a very hard problem but if you don't want to spend time on it you can take all words which have between 5% and 10% frequency in the whole document. Or, you simply take the most common 5 words.

Doing tag extraction well is very very hard. It is so hard that whole companies live from webservices exposing such an API.

You can also do stopword removal (using a fixed stopword list obtained from the internet).

And you can find common N-grams (for example pairs) which you can use to find multi-word tags.

share|improve this answer
add comment

Your Answer


