Take the 2-minute tour ×

I am attempting to build a model that will attempt to identify the interest category / topic of supplied text. For example:

"Enjoyed playing a game of football earlier."

would resolve to a top level category like:


I'm not sure what the correct terminology is for what I am trying to achieve here so Google hasn't turned up any libraries that may be able to help. With that in mind, my approach would be something like:

  1. Extract features from text. Use tagging to classify each feature / identify names / places. Would probably use NTLK for this, or Topia.
  2. Run a Naive Bayes classifier for each interest category ("Sport", "Video Games", "Politics" etc.) and get a relevancy % for each category.
  3. Identify which category has the highest % accuracy and categorise the text.

My approach would likely involve having individual corpora for each interest category and I'm sure the accuracy would be fairly miserable - I understand it will never be that accurate.

Generally looking for some advice on the viability of what I am trying to accomplish, but the crux of my question: a) is my approach is correct? b) are there any libraries / resources that may be of assistance?

share|improve this question
Perhaps a good idea in your case would be to parse the Wikipedia articles (dumps of their databases are provided on a regular basis), and to use the categories they define as a model for your own data. It may be sufficient to only parse "Category:" pages (like this one: en.wikipedia.org/wiki/Category:Sports_by_type), and to save only the articles titles. If you need more precision, you can also check the most specific words which define each category. –  doukremt Jul 6 '13 at 14:16
Wikipedia categories are pretty noisy because Wikipedians will include things not in the category, but just related or of interest. If you wanted to use Wikipedia, I'd suggest using Freebase types/properties and Wikipedia text. –  Tom Morris Jul 7 '13 at 12:46
add comment

2 Answers

up vote 4 down vote accepted

You seem to know a lot of the right terminology. Try searching for "document classification." That is the general problem you are trying to solve. A classifier trained on a representative corpus will be more accurate than you think.

  • (a) There is no one correct approach. The approach you outline will work, however.
  • (b) Scikit Learn is a wonderful library for this sort of work.

There is plenty of other information, including tutorials, online about this topic:

  • This Naive Bayesian Classifier on github probably already does most of what you want to accomplish.
  • This NLTK tutorial explains the topic in depth.
  • If you really want to get into it, I am sure a Google Scholar search will turn up thousands of academic articles in computer science and linguistics about exactly this topic.
share|improve this answer
add comment

You should check out Latent Dirichlet Allocation it will give you categories without labels , as always ed chens bolg is a good start.

share|improve this answer
add comment

Your Answer


