title: Data Structure for Text Classification Task
body: <p>I'm doing a text classification / tagging task and I would like to ask what kind of data structure would serve me best.  The training data set I have is about 4 gigs (after some cleaning, but should be even smaller if I discard the rare words) with 6 million documents.  Each document has 4 fields:</p>
  
  <ol>
  <li>Document ID</li>
  <li>Title</li>
  <li>Body</li>
  <li>Tags (as a string, e.g. "apple sql-server linux". This represents three tags, separated by a space.  Documents can have 1-5 tags)</li>
  </ol>
  
  <p>I've just finished the cleaning phase (stemming, stop words etc etc) and I'm about to convert them into a TF-IDF word vector with scikit so the output is a scipy sparse matrix.  I would like to keep the Title and Body as two vectors and combine them at a later stage when I decide on what weighting to give the Title.  The Title and Body are sparse vectors, but they are built with the same dictionary so have the same no. of columns.</p>
  
  <p>What is the best way to represent this information?  I come from R so I'm just used to storing things in data.tables / data frames but that doesn't seem very applicable for text classification and sparse matrices.  One thing I thought about doing is creating my own "Document" class and just have a list of these objects to represent the corpus.  I don't think this is very efficient, since I would probably want to do something like <code>return all docs with the Tag apple</code>.</p>
  
  <p>ML algorithms I plan to run are k-means clustering, kNN, Naive Bayes and possibly SVM.  There will probably others that I haven't thought about yet.</p>
  
  <p>I'm new to Python and text classification - any help is greatly appreciated and I am especially interested in ppl who have done it before.</p>
  
  <p>Thank you!</p>